Disaggregated Serving: Dynamo, llm-d, and When Splitting Prefill from Decode Pays

Reading a prompt and generating an answer are two different jobs. The first is limited by computation, the second by memory bandwidth, and when both share a GPU they get in each other's way: the chunked prefill piece measured answers stalling for seconds while long prompts were read. Production stacks increasingly run the two jobs on separate GPUs and move the KV cache between them. This piece shows how that works, what it costs, what it buys, and how NVIDIA Dynamo and llm-d organize it.

Two jobs, one GPU

Every request has two phases:

• Prefill reads the whole prompt at once and writes its KV cache. Thousands of tokens are processed together, so the GPU's arithmetic is the limit. Its user-visible cost is the time to first token.
• Decode produces the answer one token per step. Each step reads all the weights and the KV cache for a handful of new tokens, so memory bandwidth is the limit. Its user-visible cost is the time per output token.

On a shared GPU, the scheduler mixes both in every step, and each hurts the other: a prompt chunk makes every answer's next token wait, and a large batch of answers slows prompt reading. Chunked prefill limits the damage with a token budget. Disaggregation removes it: one pool of GPUs only reads prompts, another only generates, and each request's KV cache crosses from the first to the second.

The idea was formalized in 2023-2024 by Splitwise (separating "a compute-intensive prompt computation, and a memory-intensive token generation"), DistServe ("eliminating prefill-decoding interference"), and Mooncake, the "KVCache-centric disaggregated architecture" that serves Moonshot AI's Kimi.

What the split buys, and what it does not

• Steady time per token. The decode pool never reads prompts, so its steps have the same length whatever arrives. The vLLM documentation calls disaggregation "a much more reliable way to control tail ITL" than tuning a chunk size (docs).
• Independent sizing. Each pool gets the number of GPUs, the parallelism, and even the GPU model that suits its job: more prefill GPUs for long prompts, more decode GPUs for long answers.
• Not, by itself, more throughput. The same vLLM page is blunt: "Disaggregated prefill DOES NOT improve throughput." The GPUs do the same total work; some deployments report throughput gains anyway (below), by tuning each pool separately.
• A new cost: moving the KV cache. Every prompt's cache must cross from one pool to the other. For Qwen3-8B that is 144 KiB per token, as the KV cache piece computed: about 2.2 GiB for a 16,000-token prompt.

Size a split for your traffic:

Three things to try:

1. Default traffic, then 4 requests per second. The shared GPUs spend about half their time reading prompts, and every answer slows down with them. The split keeps answers at the decode pool's pace, until its 2 prefill GPUs saturate: then the first token suffers instead. Move GPUs to the prefill side.
2. 32,000-token prompts. The KV transfer grows with the prompt: about 100 ms over a 400 Gb/s RDMA network, almost 0.4 s over 100 Gb/s Ethernet.
3. Short prompts and long answers (500 tokens in, 2,000 out). Prompt reading is a small share of the work; the split costs an extra hop and buys little. Chunked prefill is enough.

Moving the KV cache

Engines do not move the cache themselves; they hand it to a transfer library through a KV connector. In vLLM 0.31.0 the main ones are NixlConnector (NVIDIA's NIXL library over UCX, RDMA, or other backends), MooncakeConnector, and LMCache, configured with --kv-transfer-config (docs). The feature is still marked experimental. SGLang 0.5.21 has its own: --disaggregation-mode prefill or decode, with Mooncake as the default transfer backend.

A disaggregated vLLM setup is two servers and a router in front:

$ VLLM_NIXL_SIDE_CHANNEL_PORT=5600 vllm serve Qwen/Qwen3-8B --port 8100 \
    --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'
$ VLLM_NIXL_SIDE_CHANNEL_PORT=5601 vllm serve Qwen/Qwen3-8B --port 8200 \
    --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'
$ python toy_proxy_server.py --port 8192 --prefiller-ports 8100 --decoder-ports 8200

The router sends the request to the prefill server with an instruction to keep its cache for remote decoding, then forwards it to the decode server, which pulls the blocks it needs over NIXL and continues from there. The proxy here is vLLM's test proxy; production stacks do this routing themselves.

Measured: the price of the split, on one GPU

We ran exactly that setup on one NVIDIA GB10 with vLLM 0.31.0, Qwen3-8B (BF16), and NIXL 1.4.1, 2026-10-06: the prefill and decode servers shared the GPU, with 0.38 of its memory each, and an aggregated server with the same memory served as the baseline. One request at a time, 8 per prompt size, every request completed. This cannot show the benefit of disaggregation: both servers share one GPU's compute, so nothing is isolated. It shows the mechanics and their cost.

What it shows:

1. The cache is exactly as large as the formula says. A 1,000-token prompt moved 141.75 MiB: 1,008 tokens (rounded up to whole 16-token blocks) × 144 KiB. For the longer prompts, the decode server already held part of the cache from earlier, similar prompts (a 24% prefix cache hit rate), and only the missing blocks crossed.
2. The transfer is the whole difference. The split's extra time is the transfer plus 0.1 to 0.26 s for the second hop and the proxy. Decoding speed is unchanged, as expected with one request at a time.
3. The link decides. On this machine NIXL moved the cache at about 530 MB/s, a slow path we did not tune. The same 1.77 GB (1,687.5 MiB) would take about 35 ms over a 400 Gb/s RDMA link and a few milliseconds over NVLink. Disaggregation is built for clusters with fast interconnects; over a slow link, it costs more than it saves.

The stacks that run it

Splitting is only half the job: something must route each request, size the pools, and keep the KV caches findable. That is what the serving stacks do.

• NVIDIA Dynamo (v1.5.0) describes itself as "the orchestration layer above inference engines": it runs SGLang, TensorRT-LLM, or vLLM workers and adds disaggregated serving, a KV-aware router ("routes requests based on worker load and KV cache overlap"), a KV block manager that offloads caches from GPU to CPU, SSD, and remote storage, and an SLA-driven planner that resizes the pools.
• llm-d (v0.10.0), a CNCF sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA, does the same on Kubernetes around vLLM: prefix-cache- and load-aware routing, tiered KV offloading, prefill/decode disaggregation, and wide expert parallelism.

Both publish results from partners, with their setups. llm-d's README cites "up to 70% higher tokens/sec with prefill/decode disaggregation vs standard vLLM" for GPT-OSS on NVIDIA B200 instances (AWS), and "10–30% throughput improvement" for GPT-OSS-120B and Llama 3.3 70B on AMD MI300X (Oracle). Those gains come from sizing and parallelizing each pool separately at scale, which a single GPU cannot reproduce; we have not measured them.

When to split

• Split when long prompts share GPUs with latency-sensitive answers at scale, you have more than a handful of GPUs, and a fast link between them (NVLink domain, InfiniBand, RoCE). Use a stack (Dynamo, llm-d) rather than hand-wiring connectors.
• Don't split for short prompts, a few GPUs, or a slow network: tune chunked prefill instead, and add prefix caching so repeated prompts are not read twice.
• Measure per pool: the prefill pool's queue sets the time to first token, the decode pool's batch sets the time per token, and the link adds the transfer to every request.

Next

Routing decides which GPU gets a request; prefix caching decides how much of it that GPU has already read. The next piece looks at prefix caching in depth.