From Prompt to Token: The Life of an LLM Request
One chat request followed through a real inference server, from chat template and tokens to prefill, decode, sampling, and streaming, each stage timed.
inference
4 posts
Prefill vs. Decode: Why the Second Token Is a Different Workload
Why a prompt token costs hundreds of times less than an answer token on the same weights: prefill and decode, seen in a kernel trace.
The KV Cache, Sized: What the Formula Says and What Your Engine Gives You
Compute what each token of KV cache costs from a model's config, then predict the capacity vLLM reports at startup, and see where the two differ.
A GPU in One Picture: SMs, Warps, and Memory
The parts of a GPU that decide inference speed, from streaming multiprocessors to DRAM, with one real GPU's numbers and three measurements.