Prefix Caching: When It Pays and How to Design Prompts for It

An agent resends the same system prompt, the same tool definitions, and the whole conversation so far with every request. Most of those tokens were read a few seconds earlier. Prefix caching lets the engine skip them, and in our measurement it was the difference between a quarter of a second and two seconds before the first token. One detail decides whether it works: whether the start of your prompt stays byte for byte the same. This piece shows how vLLM and SGLang find a shared prefix, what breaks it, and how to lay out prompts so it holds.

Paying once for a repeated prompt

Reading a prompt, the prefill, is the expensive part of a request's start: on our test machine, Qwen3-8B reads about 4,000 tokens per second (measured in the chunked prefill piece). An agent turn with a 5,000-token prompt (system prompt plus history) pays a second or more just to read what it already read on the previous turn.

The Paged Attention piece showed the mechanism that makes reuse possible: the KV cache lives in blocks, and two requests can point at the same block. Prefix caching keeps the blocks of finished requests and looks them up when a new request starts the same way. Every block found is a block not computed: prefill skips straight to the first new token.

How the engines find a shared prefix

vLLM hashes blocks in a chain. Each full block of 16 tokens gets a hash computed from three things: the hash of the block before it, its own tokens, and a few extra keys (hash_block_tokens). A new request is hashed block by block and each hash is looked up; the first miss ends the match. Because each hash includes the previous one, a block matches only if everything before it matches. The extra keys keep things apart that must not be shared (source): the LoRA adapter, images, and a per-request cache_salt that you can set to isolate tenants from each other. Prefix caching is on by default.

SGLang walks a radix tree. Its RadixAttention cache stores token sequences in a tree, each edge a run of tokens; a new request walks down from the root as long as its tokens match (RadixCache.match_prefix). Matching is token by token by default, or page by page when the attention backend uses pages. Old branches are evicted least-recently-used first (--radix-eviction-policy lru by default, with LFU and others available).

Both lead to the same rule: a prefix cache matches from the first token up to the first difference, and nothing after it, even if the rest is identical.

Try it on two requests:

1. Same system prompt, new question: every block before the question is reused.
2. Timestamp at the top: the clock moved five seconds, so the very first block differs, and nothing is reused, not even the identical tool definitions after it.
3. Timestamp at the end: the same timestamp, placed after the instructions, costs almost nothing.
4. Tools reordered: two tool definitions swapped places, and everything from that point on is a miss.

(The widget counts words as tokens; real tokenizers cut text differently, but the chaining is the same.)

Measured: three layouts, same agent

We ran a small agent workload against vLLM 0.31.0 and SGLang 0.5.21 with Qwen3-8B (BF16) on an NVIDIA GB10, 2026-10-06: 8 conversations of 8 turns each, a system prompt of 4,805 tokens (instructions and 24 tool definitions), and the whole history resent on every turn, as agent frameworks do. Same conversations, three layouts of the system prompt:

• Stable: identical on every request.
• Timestamp at the top: "Current time: ..." to the second, the first line of the system prompt.
• Tools shuffled: the same 24 tools, in a random order on every request (a dictionary iterated in an unstable order, a set serialized to JSON).

The hit rates come from each engine's counters (vLLM's prefix_cache_hits and prefix_cache_queries, SGLang's cached tokens over prompt tokens), a fresh server per layout, every request completed.

What the numbers say:

1. A stable prefix makes agent turns cheap. 94 to 97% of all prompt tokens came from the cache; after the first turn, the first token came back in about a quarter of a second, although each request carried about 5,000 tokens.
2. A timestamp at the top costs the history. The hit rate still looks high, 72 to 81%, for an instructive reason: our eight conversations ran in step, so requests sent in the same second carried the same timestamp and shared the system prompt with each other. What the timestamp destroyed is each conversation's own history: every turn, the clock had moved, the first block differed, and the whole history was read again. The first token came 4.7 to 7 times later than with a stable prompt.
3. Reordering is the worst, because it is invisible. Shuffled tools leave only the instructions before them in common: 6.5% of tokens from the cache, and 8 to 18 times slower first tokens. Nothing in the prompt looks different to a human; to the cache, everything after the first moved tool is new.
4. The engines behave alike. vLLM's block-based and SGLang's token-based caches reached similar hit rates on the same traffic; SGLang was faster on the stable layout here, vLLM on the others.

Designing prompts for the cache

• Put what never changes first, what changes last. Instructions, then tool definitions, then documents shared across requests, then the conversation, then the new message. Dates, user names, request IDs, and retrieved snippets that differ per request go at the end, or in the user message.
• Serialize deterministically. Sort tool lists and JSON keys, pin the order of few-shot examples, and do not rebuild the system prompt from an unordered set. A cache hit depends on identical tokens, not identical meaning.
• Append, don't rewrite. Agents that summarize or truncate old turns change the middle of the history and lose the cache from there on; when you must compact, do it rarely and in big steps.
• Watch the hit rate. vLLM exposes vllm:prefix_cache_queries and vllm:prefix_cache_hits and prints the rate in its log; SGLang exposes cached and prompt token counters. A sudden drop usually means a template change.
• Separate tenants on purpose. If users must not share cached prefixes (timing can reveal whether a prefix was cached), give each tenant a cache_salt in vLLM, rather than relying on chance.
• Route for the cache at scale. With several replicas, a request only hits the cache of the replica that served its prefix. Cache-aware routing (llm-d, NVIDIA Dynamo, covered in disaggregated serving) sends it there.

Next

Prefix caching avoids work the GPU already did. The last piece of this run steps back to ask why every technique in the series helps where it does: the roofline, which places prefill and decode against a GPU's compute and memory limits.