DeepInfer: What DeepSeek Open-Sourced, and What It Kept
DeepSeek's open-source engine runs one model on Blackwell and Ascend: why it came out, what the code shows, what was kept back, and what it means for vLLM users.
sglang
9 posts
Expert Parallelism, and What DeepEP Does for It
Why serving a big MoE model means spreading its experts over many GPUs, what dispatch and combine cost, and how DeepEP and its V2 redesign make it fast.
Choosing an Inference Server: The Landscape, from Data Centers to Laptops
Inference servers in three groups: production engines and their stacks, local apps built on llama.cpp, and enthusiasts' specialized engines, with how to pick.
Chunked Prefill: Trading First-Token Time for Smooth Streaming
How engines split long prompts across steps, what the per-step token budget trades between time to first token and time per token, and how to set it.
Structured Output: How Engines Force Valid JSON, Token by Token
How vLLM and SGLang guarantee valid JSON or a grammar: compiling a schema into a token mask, applying it at every step, and what it costs in latency and quality.
Speculative Decoding from First Principles
How a cheap guess and one verification pass produce several tokens per step, when it speeds things up and when it does not, measured.
Prefix Caching: When It Pays and How to Design Prompts for It
Why a repeated prompt can cost almost nothing, how vLLM and SGLang find shared prefixes, what breaks a hit, and how to structure prompts for agents and chat.
Disaggregated Serving: Dynamo, llm-d, and When Splitting Prefill from Decode Pays
Why production stacks run prefill and decode on separate GPUs, how NVIDIA Dynamo and llm-d move the KV cache between them, and when the split pays.
A 27B Model as Your Local Agent: What It Can and Can't Do
Real coding-agent turns replayed on a 27B model on one desk-side GPU: what the prefix cache saves, how many agents fit, how well it calls tools.