Expert Parallelism, and What DeepEP Does for It
Why serving a big MoE model means spreading its experts over many GPUs, what dispatch and combine cost, and how DeepEP and its V2 redesign make it fast.
serving-in-production
3 posts
TTFT, ITL, Goodput: Measuring What Users Feel
What TTFT, TPOT, ITL, end-to-end latency, and goodput each measure, how the tools compute them, and why peak throughput is the wrong target.
Benchmark an Inference Server Without Lying to Yourself
Common load-test mistakes made on purpose next to the correct run, on one vLLM server, so each rule comes with how far it moves the numbers.