From Prompt to Token: The Life of an LLM Request
One chat request followed through a real inference server, from chat template and tokens to prefill, decode, sampling, and streaming, each stage timed.
prefill-decode
2 posts
Prefill vs. Decode: Why the Second Token Is a Different Workload
Why a prompt token costs hundreds of times less than an answer token on the same weights: prefill and decode, seen in a kernel trace.