Expert Parallelism, and What DeepEP Does for It

DeepSeek serves decode for its V4.1-Flash model on 64 to 128 GPUs per instance, not 8, and gets more tokens per GPU for it. The reason is not compute: with experts spread over more GPUs, each GPU reads fewer weight bytes per step and each expert sees more tokens. Expert parallelism (EP) is what spreads them, and DeepSeek's DeepEP library is what makes the traffic it creates cheap enough to pay off.

Why experts get their own kind of parallelism

A mixture-of-experts (MoE) layer holds many experts and sends each token to a few of them. DeepSeek-V4.1-Flash has 384 routed experts per layer, each about 35 million parameters, and every token uses 6 of them plus a shared expert. Across its 40 layers, the routed experts are almost all of its 552 billion parameters.

There are two ways to split such a layer over GPUs:

• Tensor parallelism cuts every expert into slices, one per GPU. Every GPU works on every token, and an all-reduce sums the slices at the end of each layer. It keeps GPUs equally busy, but the slices get thin as you add GPUs (2,304 columns over 64 GPUs is 36 each, too narrow for efficient matrix multiplications), and every token's activations reach every GPU through an all-reduce, not only the 6 GPUs that hold its experts.
• Expert parallelism gives each GPU whole experts. A token travels to the GPUs that hold its 6 experts and its results travel back. Each GPU reads only its own experts.

The attention part of the model has no experts, so engines pair EP with data-parallel attention: each GPU runs attention for its own requests, one replica per GPU, and only the MoE layers are shared. DeepSeek runs it that way, with as many attention replicas as expert ranks.

Why wider is better

With data-parallel attention, each GPU you add brings its own requests, so the global batch grows while each GPU holds fewer experts. Take V4.1-Flash, 64 requests per GPU, one token per request:

Eight times more GPUs: each reads an eighth of the weight bytes per step, and each weight byte serves eight times more tokens.

The numbers come from the config. An expert is three matrices of 5,120 × 2,304, about 18.8 MB in FP4 with its scales (see quantizing MoE models). At EP8, a GPU holds 48 experts per layer; the 8 GPUs together route 8 × 64 × 6 = 3,072 token-expert pairs per layer over 384 experts, 8 per expert on average, so practically every local expert is used and the GPU reads about 36 GB of expert weights per step. At EP64, it holds 6 experts, about 4.5 GB, and each expert sees about 64 tokens. (DeepSeek adds redundant copies of popular experts on top, 7 per GPU at EP64; see below.)

Decode is memory-bound: an expert multiplication with 8 tokens spends its time reading weights, not computing (see the roofline for LLMs). With 64 tokens, the same bytes do eight times the work. DeepSeek's own deployment Q&A, which we walk through in DeepInfer, puts it as "Wide-EP substantially reduces per-step memory traffic in the MoE expert part", and its benchmark shows EP64 ahead of EP8 at every batch size: at 64 requests per GPU, about 15,700 output tokens per second per GPU at 246 per user, against 12,600 at 198 (V4.1-Flash, 131,072-token context, DeepSeek's measurement with an assumed draft acceptance).

The price is communication: every MoE layer now moves tokens between GPUs, twice.

Dispatch and combine

Each MoE layer makes two all-to-all exchanges: one to reach the experts, one to come back.

Dispatch sends each token's hidden state to the GPUs holding its experts. A token going to two experts on the same GPU is sent once. Combine sends each expert's output back to the token's GPU, where the outputs are weighted by the router scores and summed. Every GPU sends to and receives from many others at once: an all-to-all, with sizes known only after routing.

The bytes add up. A V4.1-Flash hidden state is 5,120 values: about 5 KB in FP8 for dispatch, 10 KB in BF16 for combine, the precisions DeepSeek described for V3. With 6 experts on 6 different GPUs, that is up to about 91 KB per token per MoE layer. For 64 tokens per GPU over 40 layers, about 230 MB leave and return to each GPU per decode step. Inside an NVLink domain that is affordable; across nodes, RDMA gives each GPU about an eighth of the bandwidth (DeepEP's own numbers below), which is why DeepSeek runs EP "within a (super)node": 64- to 128-GPU NVLink supernodes for decode.

What DeepEP does

DeepEP is the communication library DeepSeek released in February 2025 to do this exchange fast: GPU kernels that pack, send and unpack tokens for dispatch and combine, over NVLink inside a node and RDMA between nodes.

It started with two families of kernels, because prefill and decode want different things:

• High-throughput kernels for prefill and training: thousands of tokens per step, so bandwidth is what counts. They first exchange counts, then move the tokens into compact buffers, and forward traffic from the RDMA network into NVLink in two hops.
• Low-latency kernels for decode: a few tokens per GPU, so every microsecond of latency is a share of the step. They write into pre-reserved slots without a counting round trip, and work inside CUDA graphs, which decode needs.

Engines still expose that split: SGLang's default --deepep-mode auto uses low-latency kernels for decode batches and normal ones for prefill.

DeepEP V2, released on 30 April 2026, rewrote it:

• One interface for both regimes (ElasticBuffer, renamed EPBuffer in V2.5); for decode, you can skip the CPU synchronization and let the expert kernels read the token counts on the GPU.
• The NCCL GIN backend instead of NVSHMEM, so DeepEP reuses communicators the engine already has.
• Bigger domains: larger NVLink systems and larger scale-out.
• SM and queue counts computed analytically instead of auto-tuned. This matters because DeepEP's kernels run on the GPU's streaming multiprocessors, which are then unavailable to compute.

DeepSeek's release numbers on SM100: 726 GB/s for dispatch and 740 GB/s for combine over NVLink at EP8, using 64 SMs, or 643 and 675 GB/s with only 24 SMs; about 90 GB/s over RDMA across 2 nodes of 8 GPUs, using 12 SMs. Against V1, "up to 1.3x peak performance, while saving up to 4x SM count".

V2.5 (29 September 2026) removed V1 and NVSHMEM entirely and split ElasticBuffer into EPBuffer, EngramBuffer (the RDMA lookups for Engram tables), PPBuffer and BucketBuffer. Ports exist for Huawei's Ascend 950 (same API) and AMD (an experimental branch).

Hybrid or direct

Across nodes, DeepEP has two topologies. Direct sends every token over RDMA straight to its destination GPU. Hybrid is hierarchical: tokens cross the network over RDMA, then reach their final GPU over NVLink inside the destination node. DeepEP's code describes it as reaching "higher bandwidth" and being "more friendly to multi-plane/multi-rail networks", the clusters with one NIC per GPU on separate network rails.

Hybrid crosses the network, then finishes over NVLink inside the node.

vLLM v0.32.0 turned DeepEP V2's hybrid NVLink + RDMA mode on by default; SGLang exposes it as --deepep-v2-mode hybrid (the default, direct, is for a single NVLink domain).

Load balance: redundant experts

Experts are not equally popular, and in EP the slowest GPU sets the pace of every layer. The fix is expert-parallel load balancing (EPLB): count how many tokens each expert receives over a window of steps, give hot experts extra copies on other GPUs, and reshuffle periodically.

DeepSeek's production settings, from the Q&A:

An imbalance of 1.09 means the slowest GPU carries 9% more than average: the expert part of every MoE layer waits that much longer than a perfectly balanced one would.

In vLLM and SGLang

Both engines can run DeepEP for EP, and both default to something else. The flags at their latest releases:

Two-batch overlap splits each batch in two so that one half's dispatch and combine run while the other half computes; at about 230 MB per GPU per decode step, that is what keeps communication off the critical path.

DeepEP is not the only option any more: vLLM also lists MoRI (AMD), MoonEP, NIXL-EP and FlashInfer's all-to-all kernels, and SGLang Mooncake, NIXL and Mori. The newcomer to watch comes from DeepSeek itself: Mega MoE in DeepGEMM fuses dispatch, the expert matrix multiplications and combine into a single kernel inside an NVLink domain, and DeepSeek calls it "significantly faster than unfused MoE" at any EP size there (DeepGEMM covers it). SGLang exposes it as --moe-a2a-backend megamoe.

When wide EP pays

Wide EP is a decode strategy for big MoE models under heavy traffic:

• It needs requests. The gain comes from filling every expert with many tokens; with a few concurrent users per GPU, a 64-GPU instance mostly moves bytes around. DeepSeek's smallest efficient unit is 64 GPUs of decode (one EP64 instance) plus 64 GPUs of prefill, split into many small EP2 to EP8 instances.
• It needs one fast domain. Keep decode EP inside NVLink (an NVL72 rack, or a node); across RDMA, communication starts to dominate.
• It needs balance. Turn EPLB on, with redundant experts, and watch the imbalance metric.
• For one node, start simpler. On 8 GPUs, a model that fits runs well with tensor parallelism or small EP; switch when per-GPU throughput at your target latency stops improving.

Gotchas

Communication uses SMs. DeepEP's kernels run on the GPU's streaming multiprocessors; zero-SM RDMA EP is no longer supported. More SMs for communication means fewer for the experts: let DeepEP compute the count, and measure before overriding it.

The default backend is not DeepEP. vLLM defaults to allgather_reducescatter and SGLang to none; turning on EP alone does not get you DeepEP's kernels.

Overlap needs two batches' worth of requests. Two-batch overlap only hides communication when each half still fills the experts; at low concurrency it adds a split without hiding much, so measure with it on and off at your real load.

SGLang renamed DP attention. In v0.5.22 it is --attn-dp-size N; the old --dp-size N --enable-dp-attention still resolves but is deprecated, with removal planned after 2026-12-31. Update launch scripts copied from older recipes.

EP turns one failure into many. All GPUs of a wide-EP instance form one communication group: one bad GPU stops the instance. DeepSeek restarts with however many healthy GPUs remain and takes that as the new EP size.

The kernel that does the expert multiplications on the other side of dispatch is DeepGEMM; the layer above, routing requests across instances, is covered in disaggregated serving.