Disaggregated Serving: Dynamo, llm-d, and When Splitting Prefill from Decode Pays
Why production stacks run prefill and decode on separate GPUs, how NVIDIA Dynamo and llm-d move the KV cache between them, and when the split pays.
Serving in Production
From one engine to a service: metrics, routing, disaggregation, and the stacks that run them.
TTFT, ITL, Goodput: Measuring What Users Feel
What TTFT, TPOT, ITL, end-to-end latency, and goodput each measure, how the tools compute them, and why peak throughput is the wrong target.
Benchmark an Inference Server Without Lying to Yourself
Common load-test mistakes made on purpose next to the correct run, on one vLLM server, so each rule comes with how far it moves the numbers.
Expert Parallelism, and What DeepEP Does for It
Why serving a big MoE model means spreading its experts over many GPUs, what dispatch and combine cost, and how DeepEP and its V2 redesign make it fast.