5 posts
How serving engines hand out KV cache memory in 16-token blocks instead of reserving the maximum, how the same blocks let requests share a prompt, and what that saves in memory and time to first token.
Compute what each token of KV cache costs from a model's config, then predict the capacity vLLM reports at startup, and see where the two differ.
Why a 27B model holds 262k tokens of context on a 24 GB GPU: what hybrid, sliding-window, and sparse attention store and read per token, on real models.
Why a repeated prompt can cost almost nothing, how vLLM and SGLang find shared prefixes, what breaks a hit, and how to structure prompts for agents and chat.
Seven KV cache formats in vLLM, from FP8 to 3-bit, on one model: how many more tokens fit, what long-context recall loses, and why speed follows the kernel.