Subscribe
Once a week: what changed in serving AI models, and what it means if you build, run or pay for them, in about five minutes.
Inference Notes
Once a week: what changed in serving AI models, and what it means if you build, run or pay for them, in about five minutes.
Latest issue: No. 1, vLLM matches DeepInfer on GB300 NVL72, plus GLM-5.3 and MiniMax day-0
Hi, and welcome to the first Inference Notes, from The Serving Stack. Once a week: what changed in serving AI models, and what it means if you build, run or pay for them, in about five minutes.
This week, a general-purpose engine caught up with a model maker's own: vLLM now serves DeepSeek's latest model within a few percent of DeepSeek's in-house engine on decode, and scales it to a full GB300 rack. It lands the same week we published our own look at that engine, DeepInfer, and four more articles on what's inside DeepSeek and Qwen.
The big one
vLLM catches DeepSeek's own engine
vLLM now serves DeepSeek-V4.1-Flash across a full GB300 NVL72 rack at 954K tokens/s per GPU (counting cached and new input plus output), while keeping 196 tokens/s per user at P90. It gets there by running prefill and decode on separate GPUs and spreading the experts wide. On decode, measured on GB200, it comes within 6% of the numbers DeepSeek published for its own engine, DeepInfer, from batch 8 to 256.
Why so close: the model keeps only 890 bytes of KV cache per token, so decode is limited by reading expert weights, and the levers for that, wide expert parallelism with load balancing, are already in vLLM. If you serve this model, a general-purpose engine now gets you near the model maker's own numbers.
What to watch: at rack scale the CPU becomes the bottleneck (an unoptimized 32-way setup ran slower per step than a 16-way one), and a separate low-latency path reaches 826 tokens/s for a single user on four B300s. vllm.ai
Our take on the engine itself: DeepInfer, what DeepSeek open-sourced and what it kept.
The week in inference
Engines
- GLM-5.3 at 1M tokens on two H200 nodes. vLLM splits prefill across 8 GPUs (PCP8 × DCP8) with Hybrid HiSparse on the decode side, the only setup the team tested that serves 1M-token contexts. At up to 32 concurrent users on 142K contexts it gives 1.7 to 3.2× the per-user speed of plain data-parallel prefill at the same throughput. vllm.ai
- MiniMax-M3.1-Flash runs in vLLM from day one. On NVIDIA Blackwell and Rubin and AMD MI350, with an NVFP4 KV cache and a DSpark draft model for speculative decoding; on Blackwell a new CUDA 13.4 instruction speeds up one attention-indexer kernel by up to 1.7×. vllm.ai
- DFlash 2 drafts a whole block of tokens in parallel. Inco AI's new draft model for speculative decoding, with a Qwen3.8-27B drafter, which Inco says lets SGLang serve at 2.7 to 3.4× plain decoding at batch size 1 (hardware not stated). inco.ai
Models
- Kimi K3.1 Pro. Moonshot's new 3T-class model with a 1M-token context is available now through Kimi's API and apps; open weights are promised by 4 November. kimi.ai
- Mercury 3. Inception's diffusion LLM, more than 40× larger than Mercury 1, claims 1,068 tokens/s on NVIDIA GPUs (unnamed); half price at launch, $0.10 / $0.35 per 1M input / output tokens. inceptionlabs.ai
Agents
- A harness changes the bill more than the score. Five models through Claude Code, OpenCode and mini-SWE-agent on SWE-bench Verified; pass rates mostly within rerun noise (except OpenCode, up to 9 points behind), but cost per task differed by up to 3×, driven by the system prompt and tool schemas resent at every step. arxiv.org
Ecosystem
- PyTorch goes native on TPUs and Trainium. At PyTorch Conference, Google and Meta showed TorchTPU (a native PyTorch backend for TPUs, working with vLLM and SGLang) and AWS opened the TorchNeuron public beta for Trainium; Nebius, Nscale and Shopify joined the foundation as Platinum members, among nine new members. pytorch.org
- Which Hub models run on Trainium. A Hugging Face Space shows, for popular text and image-text models, whether they run with torch-neuronx, from a public dataset. huggingface.co
New on The Serving Stack
Inside DeepSeek and Qwen
Five articles this week, best read in order:
- Engram: Memory You Look Up Instead of Compute14 min DeepSeek's hashed n-gram memory, which a model looks up instead of computing, and how DeepSeek-V4.1 and Qwen serve it.
- Qwen3.8-Flash-Next, Layer by Layer16 min Qwen's next architecture piece by piece: linear and sparse attention, 512 experts, an n-gram memory, and what each costs to serve.
- DeepInfer: What DeepSeek Open-Sourced, and What It Kept22 min The engine behind this week's big story: what the code shows, what was kept back, and what it means for vLLM users.
- Expert Parallelism, and What DeepEP Does for It13 min Why a big MoE model spreads its experts over many GPUs, what dispatch and combine cost, and how DeepEP makes it fast.
- DeepGEMM: The Matrix Multiplications Behind DeepSeek's Models14 min Block-scaled FP8 and FP4 matrix multiplications, grouped GEMMs for experts, and the fused Mega MoE kernel.
That's the week.
Reply to this email if something here is wrong, missing, or worth a deeper look: every reply reaches us, and the best questions become articles.
Next week: the first full week of news, and more from the site.
If a colleague runs inference, forward this or send them to the sign-up page.