From Prompt to Token: The Life of an LLM Request
One chat request followed through a real inference server, from chat template and tokens to prefill, decode, sampling, and streaming, each stage timed.
How an LLM Request Becomes Tokens
The core mental model of LLM inference, one mechanism at a time, shown with figures and widgets and checked on real hardware.
Prefill vs. Decode: Why the Second Token Is a Different Workload
Why a prompt token costs hundreds of times less than an answer token on the same weights: prefill and decode, seen in a kernel trace.
The KV Cache, Sized: What the Formula Says and What Your Engine Gives You
Compute what each token of KV cache costs from a model's config, then predict the capacity vLLM reports at startup, and see where the two differ.
When Most Layers Stop Caching: Hybrid, Sliding, and Sparse Attention, Sized
Why a 27B model holds 262k tokens of context on a 24 GB GPU: what hybrid, sliding-window, and sparse attention store and read per token, on real models.
Continuous Batching, Animated: How a Server Keeps the GPU Busy
Why serving engines re-form the batch at every decode step, what it buys in throughput and latency, and the settings that control it.
Paged Attention: Virtual Memory for the KV Cache
How serving engines hand out KV cache memory in 16-token blocks instead of reserving the maximum, how the same blocks let requests share a prompt, and what that saves in memory and time to first token.
Chunked Prefill: Trading First-Token Time for Smooth Streaming
How engines split long prompts across steps, what the per-step token budget trades between time to first token and time per token, and how to set it.
Structured Output: How Engines Force Valid JSON, Token by Token
How vLLM and SGLang guarantee valid JSON or a grammar: compiling a schema into a token mask, applying it at every step, and what it costs in latency and quality.
Speculative Decoding from First Principles
How a cheap guess and one verification pass produce several tokens per step, when it speeds things up and when it does not, measured.
Prefix Caching: When It Pays and How to Design Prompts for It
Why a repeated prompt can cost almost nothing, how vLLM and SGLang find shared prefixes, what breaks a hit, and how to structure prompts for agents and chat.