DeepSeek: From V1 to V4, the Models and the Breakthroughs

DeepSeek is a research lab funded by the Chinese quantitative trading firm High-Flyer. In under three years it went from a dense 67-billion-parameter model to million-token mixture-of-experts models, and along the way it published almost every idea in papers and released the weights. For people who serve open models, DeepSeek matters twice: its models are widely deployed, and its architectures set what engines like vLLM and SGLang had to learn to run. This piece goes through the models in order, the one idea behind each step, and what it changed.

2023-2024: experts and GRPO

DeepSeek's first models, DeepSeek-Coder and DeepSeek LLM 67B (late 2023), were conventional dense transformers in the Llama mold. The first idea of its own came in January 2024 with DeepSeekMoE. A mixture-of-experts (MoE) model replaces each feed-forward layer with many smaller "experts" and routes each token to a few of them, so only a fraction of the parameters runs per token. DeepSeekMoE cut experts finer (more, smaller experts, more of them active) and added shared experts that every token uses, so common knowledge is not copied into every expert. Its 16-billion-parameter model matched Llama 2 7B "with only about 40% of computations", by the paper's count.

A month later, DeepSeekMath introduced GRPO (Group Relative Policy Optimization), a variant of the PPO reinforcement-learning algorithm that drops the separate value model: it samples a group of answers to the same question and scores each against the group's average. Less memory, simpler training. GRPO became the reinforcement-learning method behind R1 a year later, and one of the most copied training recipes of 2025.

V2: latent attention

DeepSeek-V2 (May 2024) combined DeepSeekMoE (236 billion parameters, 21 billion active per token) with a new attention design: multi-head latent attention (MLA).

Standard attention caches a key and a value for every head, every layer, every token: that KV cache is what limits how many requests fit on a GPU. Grouped-query attention (GQA) shares keys and values among groups of heads. MLA goes further: it caches one compressed vector per layer and token, 512 numbers, plus a 64-number key for positions, and reconstructs every head's keys and values from it during attention. The compression is learned, so the model is trained to work with it.

With the same 128 heads, standard attention would make V3 cache 4.8 MiB per token; MLA caches 68.6 KiB, 71 times less, and less than models a twentieth of V3's size. The V2 paper reports a KV cache "reduced by 93.3%" against DeepSeek's 67B model; that figure also includes compressing the cache to 6 bits per number, and MLA alone accounts for 82%.

For engines, MLA was a new kernel to write: attention over a latent cache, with the up-projections absorbed into the query side during decoding. DeepSeek later released its own, FlashMLA, and vLLM and SGLang both carry dedicated MLA backends. Kimi K2, from Moonshot AI, reuses the V3 architecture outright: its config declares DeepseekV3ForCausalLM.

V3: FP8 training and a cost figure

DeepSeek-V3 (December 2024) scaled the recipe to 671 billion parameters, 37 billion active, trained on 14.8 trillion tokens. Three ideas:

• FP8 mixed-precision training, validated, in the paper's words, "for the first time" on an extremely large model. The weights ship in FP8, which halves their memory against BF16 and runs on Hopper's FP8 tensor cores.
• Auxiliary-loss-free load balancing: MoE routers tend to overload some experts; instead of adding a balancing term to the loss, which hurts quality, V3 adjusts a per-expert bias during training.
• Multi-token prediction (MTP): an extra module trained to predict the token after next. At inference, it doubles as a built-in draft model for speculative decoding; vLLM and SGLang can use it that way.

The report also put a number on training: 2.788 million H800 GPU-hours, "$5.576M" at $2 per GPU-hour, "excluding the costs associated with prior research and ablation experiments". The headline number traveled far without its qualification. It is the cost of the final training run only, by the lab's own accounting, but it still showed that a frontier-class model no longer required a frontier-class budget for that run.

R1: reasoning from reinforcement learning

DeepSeek-R1 (January 2025) took V3 and trained it to reason. Its sibling R1-Zero was "trained via large-scale reinforcement learning (RL) without supervised fine-tuning (SFT) as a preliminary step": GRPO, with rewards for correct answers and the right format, and the long chains of thought emerged on their own. R1 added a small set of curated examples first, for readability. Both were released under the MIT license, along with six smaller dense models (Qwen and Llama) distilled from R1's outputs.

R1 changed two things for everyone else. Open reasoning models became a category, and inference workloads changed shape: thousands of thinking tokens per answer, so decode speed, long KV caches, and the cost of output tokens started to dominate serving.

Open infrastructure week

In February 2025, DeepSeek released the systems code behind its models, one repository a day: DeepGEMM (FP8 matrix multiplication), DeepEP (expert-parallel communication), FlashMLA (MLA decoding kernels), smallpond and 3FS (data processing and a distributed file system), DualPipe and EPLB (pipeline scheduling and expert load balancing). Engines picked them up: vLLM's release notes track DeepGEMM and DeepEP versions, and its newest DeepSeek support runs on FlashMLA.

V3.1 and V3.2: hybrid thinking, sparse attention

• V3.1 (August 2025) merged chat and reasoning into one model: thinking or not is chosen per request through the chat template. It also moved to the UE8M0 FP8 scale format, a microscaling format that newer GPUs support natively.
• V3.2-Exp (September 2025) and V3.2 (December 2025) introduced DeepSeek Sparse Attention (DSA): a small "indexer" scores all previous tokens cheaply, and full attention runs only over the top 2,048 of them (index_topk in the config). Past a few thousand tokens, the cost of each new token stops growing with the length of the context. A variant trained for hard reasoning, V3.2-Speciale, is reported on its card to reach gold-medal level at the 2025 International Mathematical Olympiad and Olympiad in Informatics, by the lab's own evaluation.

V4 and V4.1: a million tokens

V4 (April 2026) came in two sizes, V4-Flash (284 billion parameters, 13 billion active) and V4-Pro (1.6 trillion, 49 billion active), both with a one-million-token context. Its attention mixes compressed and sparse layers; DeepSeek reports that at a million tokens, V4-Pro needs 27% of V3.2's computation per token and 10% of its KV cache. Our hybrid attention piece sizes that cache in detail.

V4.1-Flash (September 2026) added images and reorganized the network as a "Causal Encoder-Decoder": a 20-layer causal encoder followed by a 20-layer decoder, 552 billion parameters, with 8 billion active while reading a prompt and 16 billion while generating.

The models at a glance

From the papers, model cards, configs, and Hugging Face license tags, 2026-10-06. V3 was relicensed under MIT with V3-0324 (March 2025).

What it means if you serve them

• They are big, but each token is cheap. 37 billion active parameters out of 671 means the compute per token of a mid-size model with the memory footprint of a huge one. Serving them is a memory and communication problem: expert parallelism across GPUs, FP8 weights, and engines with MoE kernels.
• The KV cache is unusually small. MLA lets many more requests, or much longer contexts, fit in memory than a GQA model of similar quality would allow. That is why long-context serving of DeepSeek models is cheaper than their size suggests.
• Each generation needs new kernels. MLA, DSA, and V4's compressed attention each required engine work before they could be served efficiently; check that your engine release supports the exact model version before you plan on it.
• The license is no longer a question. Since 2025 the weights are MIT, which is why so many products and fine-tunes build on them.

Next

Z.ai's GLM models took a different path from the same starting point: the next piece follows them from ChatGLM to GLM-5.