DeepGEMM: The Matrix Multiplications Behind DeepSeek's Models

An FP8 matrix multiplication on Hopper keeps about 14 bits of precision while it accumulates, and at an inner dimension of 4,096 that is worth up to 2% of relative error. DeepSeek trains and serves its models in FP8 anyway. DeepGEMM is the library that makes that work, and it has grown from one 300-line kernel into most of the arithmetic its models run on, including on Huawei's chips.

Why not just call cuBLAS

A matrix multiplication is the one operation GPU vendors optimize hardest, and cuBLAS and CUTLASS are very good at it. DeepGEMM does not replace them from the bottom up: it borrows CUTLASS and CuTe concepts and still requires CUTLASS 4.0 to build. What it owns is the part above the multiply primitives, the scheduling, the accumulation path and a kernel compiled per shape, because DeepSeek's models ask for three things a general library doesn't give them.

The scales move with the data. Quantizing to FP8 means dividing numbers by a scale so they fit in a narrow range. One scale for a whole tensor is at the mercy of a single outlier (see post-training quantization and outliers). DeepSeek-V3 instead scales activations per 1×128 tile, one scale per token per 128 channels, and weights per 128×128 block. A GEMM kernel then has to apply a different scale every 128 elements along the inner dimension, in the middle of its inner loop.

The accumulation needs help. The V3 report measured Hopper's FP8 accumulation at about 14 bits of precision, with a maximum relative error near 2% at an inner dimension of 4,096. DeepGEMM's answer, from its first release, is two-level accumulation: the tensor cores accumulate in their own limited precision, and every 128 elements the partial result is copied into FP32 registers on the CUDA cores, where the scales are applied and the sum continues in full precision. Because two tensor-core instructions run concurrently, one warpgroup can promote while the other multiplies.

The scale format is part of the hardware. Hopper takes these scales as FP32 numbers; Blackwell wants them as UE8M0, an exponent-only format, four of them packed into a single 32-bit integer. The same checkpoint therefore needs a different scale tensor on each architecture.

FP4 keeps the idea and shrinks the group. On Blackwell, DeepSeek's routed-expert weights are MXFP4: one UE8M0 scale per 32 values, against 128 for the FP8 activations. The activations stay FP8, so the kernel multiplies FP8 by FP4, which is why DeepGEMM added an fp8xfp4 path in April 2026 and why Mega MoE takes an mma_type of fp8xfp4 or fp8xfp8. The kernel work is the same work: read a scale, apply it at the right boundary, keep accumulating in FP32.

Fine-grained scales keep outliers local; promoting to FP32 every 128 elements keeps the sum honest.

The shapes are known late, and there are many of them. A mixture-of-experts layer is not one matrix multiplication but hundreds, one per expert, each with a different number of tokens that is only known after routing. That is the second half of the library.

DeepGEMM compiles its kernels at runtime, with its own JIT, so each shape gets a kernel specialized to it without a compilation step at install time. The first release was deliberately small, "only one core kernel function comprising around ~300 lines of code", and the project still describes itself as a readable resource for learning GPU kernel optimization. Small did not mean slow: by April 2025 DeepSeek reported up to 1,550 TFLOPS on an H800.

One layer, many experts

Experts in an MoE layer share a shape: the same input width, the same output width, a different number of tokens each. DeepGEMM exploits that by grouping only the M axis, the token axis, and requiring N and K to be fixed. The result is two layouts, one per serving phase.

Prefill knows the token counts and packs them; decode doesn't, so it reserves capacity and masks.

Contiguous layout, for training forward passes and prefill: all the tokens are concatenated into one tall matrix, with each expert's segment padded up to the kernel's M block size (get_mk_alignment_for_contiguous_layout()). One kernel launch does the whole layer.

Masked layout, for decode: when CUDA graphs are capturing the step, the CPU does not know how many tokens each expert received, and a graph cannot branch on a number that lives on the GPU. So each expert gets a fixed-capacity slot and a mask tensor says how many rows are real; the kernel computes only those. The README names the natural input: the output of DeepEP's low-latency kernels. The two libraries are designed to meet here.

A third variant groups the K axis instead, for the backward pass that computes expert weight gradients.

Mega MoE: one kernel for the whole layer

The newest idea removes the boundary between communication and computation. In an expert-parallel MoE layer, the usual sequence is: dispatch the tokens to the GPUs holding their experts, multiply by the first expert matrix, apply SwiGLU, multiply by the second, then combine the results back. Five steps, two of them network transfers, each waiting for the one before.

Mega MoE fuses all five into a single kernel that overlaps NVLink traffic with tensor-core work: while some tokens are still arriving, the experts are already multiplying the ones that landed.

The communication doesn't get faster; it stops being a phase of its own.

DeepSeek's benchmarks, their measurement and not ours: 8-way expert parallelism, averaged over 8 ranks, against the legacy unfused path (DeepEP dispatch and combine plus separate grouped GEMMs). The pull request does not name the GPU, so read the speedups and treat the absolute throughput as unanchored. V4-Flash here is 256 experts with top-6 routing, hidden 4,096, intermediate 2,048; V4-Pro is 384 experts, hidden 7,168, intermediate 3,072.

The gain is largest at a single token per rank, where the unfused path is almost entirely waiting, and stays between 1.5x and 1.6x from 512 tokens up. DeepSeek's own deployment notes call Mega MoE "the general solution within a scale-up domain": inside one NVLink system, at any expert-parallel size and any batch size, it beats the unfused path.

The constraint is in that phrase. Mega MoE needs symmetric memory across every participating process and PyTorch 2.9 or later, and what it overlaps is NVLink traffic. DeepSeek scopes its claim to a scale-up domain and the API says nothing about node counts, but symmetric memory across nodes is not how anyone deploys this: treat Mega MoE as an inside-the-NVLink-domain optimization.

What else is in there

The library has outgrown its name. Besides the GEMMs it now holds:

• MQA logits kernels for the "lightning indexer" that picks which tokens sparse attention reads, in a prefill version and a paged decode version. They compute a weighted ReLU of query-key products, summed over heads: scoring, not attention.
• HyperConnection kernels for the multi-stream residual used by V4 and V4.1.
• Mega Gate and Mega mHC, fused versions of the routing gate and the residual mixing.

Hardware support is narrower than the feature list suggests: SM90 or SM100 only upstream, with CUDA 12.9 or later and CUTLASS 4.0 or later. And in September 2026 DeepSeek released DeepGEMM-Ascend, a port to Huawei's Ascend 950 that installs under the same package name and exposes the same API, down to Mega MoE.

The differences leak into the API: the same checkpoint needs a different scale tensor per architecture, and "API-compatible" on Ascend is an engineering achievement rather than a label.

Where your engine uses it

If you serve a block-quantized FP8 model on Hopper or Blackwell, you are probably already running DeepGEMM:

• vLLM v0.32.0 builds its own fork of DeepGEMM, pinned to a commit, and VLLM_USE_DEEP_GEMM defaults to on. VLLM_MOE_USE_DEEP_GEMM controls the MoE path alone, and VLLM_USE_DEEP_GEMM_E8M0 (also on by default) selects the Blackwell scale format. Notably, vLLM's fork is built for SM120 and SM121 as well when the CUDA toolkit is new enough, the RTX Blackwell and GB10 chips that upstream does not list.
• SGLang v0.5.22 exposes it as --fp8-gemm-backend deep_gemm, which its help text describes as "enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) when DeepGEMM is installed". The alternatives are FlashInfer's kernels, CUTLASS (the pick for SM120), Triton as a fallback, and AITER on ROCm. Its paged MQA logits backend for sparse attention (--dsa-paged-mqa-logits-backend) also defaults to DeepGEMM on CUDA, and --enable-w4a4-mxfp4-megamoe turns on the fused path with --moe-a2a-backend megamoe.
• DeepInfer, DeepSeek's own engine, calls the public deep_gemm. Its Q&A says the internal indexer and mega-series kernels differ somewhat but land "at most 5% apart" end to end.

To check it is really running: DG_PRINT_CONFIGS=1 makes the library print the configuration it picks for each shape, and the JIT cache directory (DG_JIT_CACHE_DIR, $HOME/.dj by default) fills up as kernels compile for your shapes. In a profile, DeepGEMM's kernels appear under their own generated names; if the expert multiplications still show up as cublas kernels, the backend fell back and you are not running it.

Gotchas

The first request pays for compilation. DeepGEMM compiles per shape at runtime, and a cold JIT cache turns the first request of each new shape into a stall. vLLM has VLLM_DEEP_GEMM_WARMUP for this; set its full mode to compile every shape the model can produce at startup, and keep DG_JIT_CACHE_DIR on a warm, persistent volume in your container image.

The toolchain is strict. CUDA 12.9 or later, CUTLASS 4.0 or later, and a C++ standard library with C++20 <format>, which is what bites first on an older base image.

Hopper and Blackwell want different scale tensors. FP32 on SM90, packed UE8M0 on SM100, and the left-hand scales must be transposed and TMA-aligned. The engine converts when it loads the weights (VLLM_USE_DEEP_GEMM_E8M0 picks the Blackwell format), so the cost lands at startup, not in your p99. What does hurt is a mismatch you don't notice: the engine quietly falls back to a slower backend.

Deterministic mode is a separate switch. use_deterministic_algorithms exists because the default kernels are not run-to-run identical. If you are chasing an eval discrepancy, turn it on before blaming the model.

Contiguous grouping pads. Each expert's segment rounds up to the kernel's M block size, so every expert pays for a partial block of rows whose output is discarded. It is the price of one launch per layer, and one more reason to keep experts busy: a GPU holding many lightly used experts wastes work in the kernel as well as in the weights it reads.

What this means for you

If you serve DeepSeek models on Hopper or Blackwell, you are running DeepGEMM whether you picked it or not. Your job is the JIT: warm it at startup and keep the cache, or let a user pay for it on the first request of each new shape.

If you serve other MoE models, the idea worth stealing is the grouped layout: contiguous for prefill, masked for decode under CUDA graphs. An engine that launches one kernel per expert is leaving most of this on the table.

If you write kernels, this is an unusually readable example of a production GEMM: a block-scaled multiply with a promotion loop that still reaches vendor-class throughput, in a few hundred lines you can follow.

Next: DeepInfer is the engine built on these kernels, and your first CUDA kernel is where to start if you want to write one.