Distillation: How Small Models Learn from Big Ones

If you run small models locally, you have met distillation in their names: DeepSeek-R1-Distill-Qwen-7B, or Llama 3.2 3B, which Meta says learned from its larger siblings. Distillation is how most small models worth running are made today. It is also, since 2025, the subject of public accusations between AI labs. This piece explains what it is, shows it on real numbers, and goes through how labs use it, from their own papers and statements.

Two models, one next token

A language model's output, before it picks a word, is a probability for every token in its vocabulary. Here is what two models of the same family predict after the same words: Qwen3-8B, which will play the teacher, and Qwen3-0.6B, a model about fourteen times smaller, playing the student. We computed both distributions on 2026-10-09.

Three things to see:

1. "I think this is the beginning of a beautiful" (the example from the DistilBERT paper). The teacher puts 83% on " friendship", the end of the line from Casablanca, and 8% on " relationship". The student has not learned the quote: it spreads its bets over " journey", " and", " story". Training the student to match the teacher's distribution, not just the word "friendship", tells it that " relationship" was a reasonable second choice and " journey" was not.
2. Switch to "Hard label". Ordinary training, with the right answer only, says one thing per token: "friendship". The teacher's distribution says much more: an order and a weight for every alternative. Hinton and colleagues called this extra information the reason distillation works.
3. "The capital of Australia is". As plain text completion, the teacher gives Sydney and Canberra equal weight (about 11% each). Soft targets pass on the teacher's knowledge and its mistakes: a student is only as good as what it imitates.

Across the vocabulary, the two distributions differ by 6.3 bits (KL divergence) on the first prompt and 0.8 bits on the tea prompt: distillation is the training that pushes that number toward zero, on billions of tokens.

Soft targets and temperature

The idea has a precise origin: Distilling the Knowledge in a Neural Network, by Hinton, Vinyals, and Dean (2015). Their proposal was to "raise the temperature of the final softmax until the cumbersome model produces a suitably soft set of targets", and train the small model to match those targets. Temperature flattens a distribution: at 1 it is the model's own; above 1, unlikely tokens gain weight and the ranking among them becomes visible. Move the widget's slider to 3 and the teacher's second, third, and tenth choices appear. The paper's argument: soft targets "provide much more information per training case than hard targets and much less variance in the gradient between training cases, so the small model can often be trained on much less data".

The best-known early result for language models was DistilBERT (Hugging Face, 2019), which used distillation during pre-training to "reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster".

Three forms of distillation

Today the word covers three practices that differ in what the teacher hands over and in who writes the text the student learns from.

• Soft targets (logit distillation). The student sees the teacher's full distribution at every position and is trained to match it. It needs the teacher's weights, or at least its probabilities, so labs use it on their own models.
• The teacher's text. The teacher generates answers, often with its full reasoning, and the student is fine-tuned on them like on any training data. It needs only text. This is the cheapest form, the one that works through an API, and the one at the center of the disputes.
• On-policy distillation. The student writes its own answers and the teacher scores each token of them. The student learns where it goes wrong, in the situations it actually reaches, instead of imitating texts written by a model that never makes its mistakes. Thinking Machines described it in 2025 as combining "the on-policy relevance of RL with the dense reward signal of distillation".

How labs distill their own models

Every major open-model lab now documents distillation in how it builds its small models:

• Google, Gemma 2 (2024): "We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction" (report). In Gemma 3 (2025), "all Gemma 3 models are trained with knowledge distillation".
• Meta, Llama 3.2 1B and 3B (September 2024): made by pruning Llama 3.1 8B and distilling into the smaller network, with the logits of Llama 3.1 8B and 70B used as token-level targets during pre-training (Meta).
• DeepSeek, R1 (January 2025): six smaller dense models, built on Qwen and Llama, fine-tuned for two to three epochs on about 800,000 samples generated by R1 (paper). That is the teacher's-text form, and the reason those models carry "Distill" in their names.
• Alibaba, Qwen3 (May 2025): "strong-to-weak distillation" for the small models, first on teacher-written text, then on-policy, with Qwen3-32B or Qwen3-235B-A22B as the teacher (report). Its Table 21 makes the case in numbers: from the same starting point, on-policy distillation took Qwen3-8B to 74.4% on AIME'24 in 1,800 GPU hours, where reinforcement learning reached 67.6% in 17,920.
• DeepSeek, V4 (April 2026): distillation inside one model: domain experts trained separately, then merged into one model "via on-policy distillation" (model card).

The pattern is consistent: the expensive work (pre-training at scale, reinforcement learning) is done once on a large model, and distillation passes the result down to smaller ones at a fraction of the cost.

A product, and a clause

Distillation is also sold. In October 2024, OpenAI launched Model Distillation in its API: store the outputs of a large model, then use them to fine-tune a smaller one. In OpenAI's cookbook example, a wine-classification task, gpt-4o-mini went from 64.67% to 79.33% after being fine-tuned on gpt-4o's answers, close to gpt-4o's 79.67%.

The same companies forbid distillation into competitors. OpenAI's terms of use prohibit using "Output to develop models that compete with OpenAI". Anthropic's commercial terms forbid accessing its services "to build a competing product or service, including to train competing AI models". The technique is the same; what the terms govern is who may use it, on whose outputs, and for what.

The disputes, as documented

In February 2026, two American labs made public accusations, each in its own document:

• OpenAI, 12 February 2026. In a memo to the U.S. House Select Committee on China, as reported by Bloomberg and CNBC, OpenAI described "ongoing efforts to free-ride on the capabilities developed by OpenAI and other US frontier labs" and said it had observed accounts associated with DeepSeek employees accessing its models "through obfuscated third-party routers".
• Anthropic, 23 February 2026. In Detecting and preventing distillation attacks, Anthropic named DeepSeek (more than 150,000 exchanges), Moonshot (3.4 million), and MiniMax (more than 13 million), through about 24,000 accounts it calls fraudulent. It also wrote that distillation is "a widely used and legitimate training method", and objected to its use against its terms and regional access restrictions. Its responses: classifiers and behavioral fingerprinting to detect such traffic, stronger account verification, and safeguards that make outputs less useful for training.

The accused labs had not responded to requests for comment when CNBC reported the story. CNBC also quoted researchers who questioned the framing: "the boundary between legitimate use and adversarial exploitation is often blurry" (Erik Cambria, Nanyang Technological University); the controversy is less about the technique than about "alleged fraudulent access and possible violation of contractual terms" (Lia Raquel Neves, EITIC); and the narrative "strengthens the case for stricter chip restrictions", which both companies support (Rui Ma, Tech Buzz China).

What can be said from the documents: the claims are the accusers' own, with evidence that has not been published in full; the accused have not answered them publicly in the sources we found; and both accusing companies sell, or describe as legitimate, the same technique when it is used within their terms. Disclosure: this site's articles are produced with Claude, an Anthropic model.

What "distilled" means when you run the model

• A distilled model is a normal model. "Distill" in a name describes how it was trained, not how it runs: DeepSeek-R1-Distill-Qwen-7B is a Qwen2.5-7B architecture, served like any other model of that size.
• It inherits its teacher's habits. R1's distilled models write long reasoning before answering, because that is what they were trained on. Expect long outputs, and budget for them.
• It can be better than its size suggests, because its training signal was richer, and worse than its teacher where the teacher's knowledge did not fit. Test it on your task, not on the teacher's benchmarks.
• Check the license chain. A distilled model can carry obligations from both its base model's license and its teacher's.

Next

Distillation makes models smaller by training a smaller network. The other way is to keep the network and store it in fewer bits: quantization, as in our NVFP4 piece, and the subject of a coming series. At inference, a small model can also work alongside a big one: see speculative decoding.