vLLM, SGLang and DeepInfer: What "Matches the Vendor's Engine" Means

Within days of DeepSeek open-sourcing DeepInfer, the engine behind its own service, vLLM said it was "within 6%" of it and SGLang said it was "on par or ahead". Both are close to DeepSeek's engine on DeepSeek-V4.1-Flash, but the two claims measure different things: vLLM compared its own runs with numbers read off DeepInfer's published charts, and SGLang reran DeepInfer's benchmark scripts on the same machines. Neither measured the configuration DeepSeek runs in production, and most of the code behind both is not in a release yet.

The claims, dated

DeepInfer's first commit landed on 20 October 2026 at 04:09 UTC (our piece on the engine itself). Twenty hours later, vLLM posted on X: "within 6%" of DeepInfer, up to a million tokens per second per GPU, 826 tokens per second for a single user. Its blog post carries the date of 23 October and went live on 24 October; LMSYS's post for SGLang followed 80 minutes later.

The headlines, verbatim:

• vLLM: "vLLM's decode throughput is within 6% across batch sizes 8–256, and its prefill tracks DeepInfer's published curve." Plus a full-system number: 954K tokens per second per GPU on one GB300 NVL72 rack, on SemiAnalysis's AgentX workload.
• SGLang: "Up to 1.04x prefill throughput on 4x GB300 and 4x B300, up to 7.4% shorter decode steps on 64x GB300, and within 1% at the remaining points."

What DeepInfer's own benchmark measures

The reference both engines measured against is DeepInfer's benchmark, and it is a benchmark, not a production run (our DeepInfer piece explains its token accounting). From examples/benchmark.py and the README at commit 774ce24:

• Expert routing is forced to be balanced (DEEPINFER_FORCE_LOAD_BALANCE=1): every expert gets the same load, which real traffic never does.
• One prompt, forked: the benchmark builds a single random prompt, prefills its cached prefix once, and copies it across the whole batch.
• Decode verifies a fixed three draft tokens every step, and the chart converts step time into tokens at an assumed 3.25 accepted per step. The confidence scheduling that lets DSpark vary its draft length is off by default and not set by the benchmark.
• Timing comes from rank 0: prefill over about ten steady-state steps, decode over 60 intervals.
• Hardware: "4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain". Sixty-four GPUs in one NVLink domain means an NVL72 rack, so GB200 or GB300 for decode; the prefill part is not named.

So the published curves are a clean measurement of the engine's step time under ideal routing. That is useful: it isolates kernels and scheduling. It is not how fast DeepSeek-V4.1-Flash serves real traffic, on DeepInfer or anything else. (The commit SGLang reran, which is also the one the charts were published from, 774ce24, differs from the one our DeepInfer piece read, 9bd9cf2, only by a build script and a DeepEP submodule bump: same engine.)

Two ways to compare

vLLM compared its own measurements with points read off DeepInfer's two published charts. SGLang ran both engines, with DeepInfer's own benchmark settings, on the same machines.

vLLM compared its setup with DeepSeek's chart; SGLang compared two engines on one setup.

Two differences matter more than the posts let on. vLLM says its prefill numbers are GPU forward time for one request per rank; it doesn't point out that DeepInfer's are wall time per engine step over many 4K-token requests. And SGLang's DeepInfer baseline is the best of four configurations at each point, two of which apply a DeepEP setting through a patch because "DeepInfer has no flag for it"; those two gave the best DeepInfer result at 8 of the 10 prefill points. That is generous to DeepInfer, but it isn't stock DeepInfer.

What the numbers say

Decode. vLLM's "within 6%" is its worst point. Per-user speed against DeepInfer's chart, at 8 to 256 requests per GPU: +4.5%, +2.3%, −0.9%, −3.9%, −6.0% at 128, +3.6% at 256 (our arithmetic from vLLM's figure data). It got there at half DeepInfer's expert-parallel width, with real routing and expert rebalancing, which is harder than the chart's forced balance, and on GB200 against a chart made on an NVL72 that SGLang's GB300 rerun matches within −6% to +2%, so most likely GB300. SGLang's same-machine rerun, with forced balance on both sides, has step times 0.4% slower at 8 requests and 1.1% to 7.4% shorter from 16 to 256. Both results say the same thing: on this model, decode is now about kernels all three engines share, and the open engines schedule them as well as DeepSeek does.

Prefill. SGLang's rerun is the clean comparison: from 0.6% behind to 3.9% ahead at every point, on both machines. vLLM's is not, by its own account: "we do not read the gap between the two curves as an engine-to-engine speedup". It also has a problem it doesn't mention: its reference values don't match the chart at both ends (61.5 ms at 4K where the chart reads about 68.2 ms; 755.6 ms at 64K where it reads about 731.7 ms, by our reading of the PNG, calibrated on its gridlines; our readings agree with throughput = tokens / latency to 0.2%, and with vLLM's own reading of the decode chart to 0.3%). SGLang's data shows something larger:

DeepInfer rerun on GB300 beats its own published curve by 13 to 29%, so "tracking the published curve" is a low bar on that hardware.

DeepInfer rerun on GB300 is 13 to 29% faster than its published curve; on B300 it is within about 1% of it at 32K and 64K tokens. The published chart was most likely made on B300-class hardware, which the README's "SM100" leaves open. vLLM's prefill, run on GB300, "tracks" a curve that DeepInfer itself beats on GB300. Its forward times land between 1% faster and 8% slower than SGLang's DeepInfer rerun on the same GPU type, but at a different parallelism, with a different timer and different prompts: indicative, not a comparison either post makes.

Why they land close

The first reason is lineage. DeepInfer was originally built on vLLM (23 of its files still say "Derived from vLLM; modified by DeepSeek"), and both open engines now run DeepSeek's own kernels: FlashMLA, DeepGEMM, DeepEP, DeepSelect, and fused "Mega" kernels for attention, the MoE (dispatch, expert GEMMs and combine in one kernel) and the hyper-connections; SGLang also runs the fused gate. When the kernels are the same, the engine's job is to keep them fed.

The second is the model. DeepSeek-V4.1-Flash keeps only 890 bytes of global KV cache per token, decode is dominated by the MoE at narrow expert parallelism (62% of the target model's step at DEP4, 128K context and 64 requests per GPU in vLLM's breakdown, against 13% for attention and the indexer), and a decode step barely grows with context (target forward 8.9 ms at 128K and 9.5 ms at 855K, at DEP32 with 16 requests per GPU on GB300). At DEP32 the MoE drops to 6.0 ms of a step of about 16 ms: wide expert parallelism moves the bottleneck to everything else, which is why the CPU fixes below matter. A model built this way leaves little room for engine-specific tricks on decode; what's left is expert parallelism and CPU overhead.

The engineering that transfers

The useful part of both posts is not the headline but the work behind it, most of which applies to any large MoE model. vLLM documents it change by change:

Of the 16 pull requests vLLM's post cites, 6 are merged and 9 open (one more was closed); one is in a final release (v0.32.0) and three more in the v0.32.1rc0 candidate. SGLang's measured code sits on main from 24 October, about 810 commits after v0.5.22, its latest release, and its post gives no change-by-change effects.

What carries over to other large MoE models:

• Prefill at small expert-parallel width. In an AgentX prefill-only run at concurrency 256, vLLM's DEP4 gave 83% of DEP2's throughput per GPU. Replaying identical requests on every rank, at once and then staggered, showed why: when ranks fall out of step, DEP4 loses 13.6% to imbalance, DEP2 5.9%.
• Decode at wide expert-parallel width, with rebalancing. Spreading experts over more GPUs cuts the MoE's weight reads (13.4 → 6.0 ms per step from DEP4 to DEP32), and expert rebalancing recovers almost all of the gap to uniform routing. The expert parallelism piece covers the mechanics.
• Bounded replay and Engram placement are where the prefill gains come from. vLLM's replay to layer 20 gave +19.9% at 32K new tokens. SGLang can split the Engram tables across GPUs and read them over NVLink (opt-in), while vLLM keeps them in host memory with 2 MB pages (an open PR) (Engram explains the tables).
• At rack scale, the CPU is the bottleneck. Unoptimized, vLLM's DEP32 ran slower than DEP16 (17.6 against 13.8 ms per step) although its GPU work was 1.1 ms faster. The causes it lists: synchronous KV reads, an all-reduce that grew with the rank count, admission bursts and per-prompt bookkeeping.

What DeepInfer leaves out

DeepInfer's README lists what a complete service needs and the release doesn't include: a KV cache system, prefill/decode disaggregation, and load balancing across replicas. That is the strongest argument both open engines have (disaggregated serving explains why the second matters), and SGLang makes it explicitly: its radix cache with HiCache and Linker tiers down to shared storage, disaggregation over Mooncake, NIXL or MORI, a gateway with prefix-aware routing, and DeepSeek's own chat format through its deepseek-recipe package. vLLM's 954K tokens per second per GPU on an NVL72 rack is the one number from a full serving system on a real workload; it counts cached input, uncached input and output tokens, and its $38.96 per GPU-hour is that throughput priced at DeepSeek's off-peak API rates, revenue rather than cost.

Gotchas

A benchmark mode is not a deployment. Forced balanced routing, one forked prompt and a fixed 3.25 accepted tokens per step make the engines comparable; they also make every number here optimistic. Measure on your traffic, with real routing and real acceptance.

Read the date of the code, not of the post. The NVL72 results ran on a vLLM build no release contains, and SGLang's on main from 24 October. If you run v0.32.0 or v0.5.22, you have only part of this work.

A curve read off a chart is not a measurement. vLLM's prefill reference values differ from DeepInfer's PNG by up to 10% at the short end. When a comparison depends on another team's chart, ask for their data.

GPU-only time flatters prefill. vLLM's prefill numbers exclude scheduling and admission, which its own decode section shows can dominate at scale.

The tweet is not the post. vLLM's "within 6%" circulated with a one-line method three days before the post's details; secondary coverage repeated it as "the same model and hardware", which it wasn't, and misquoted the single-user gain as 26% instead of 44%.

Verdict

On DeepSeek-V4.1-Flash, both open engines are as fast as DeepSeek's own engine on the parts DeepSeek published, and they add the serving system DeepSeek kept back. SGLang's same-machine rerun is the stronger evidence for parity; vLLM's post is the better guide to the engineering, and its 6% is a fair worst case for a harder setup than the chart's. Neither tells you how the three compare on your traffic.

What would change that call: DeepSeek publishing production numbers or the hardware behind its charts, a third party rerunning all three with real routing and acceptance, or the open pull requests landing in releases with the same effects.

If you need parity evidence on your own hardware class, SGLang's rerun is the one to reproduce: its gist has every command, and one 4× B300 node covers a prefill point. If you need a full disaggregated deployment on an NVL72, vLLM's post is the only published recipe, on an unreleased build. On v0.32.0 or v0.5.22, expect part of these gains, not all. If you read engine claims, ask what was measured on whose hardware, in which benchmark mode, on which build. For DeepSeek, the open engines were already at parity on the parts it published, with part of the work merged in vLLM before DeepInfer came out; what it kept, the serving layer, is where they still differ.

Next: benchmarking inference servers covers how to run a comparison like this on your own workload.