Kimi: Moonshot AI's Models, from Long Context to Trillion-Parameter Agents

Moonshot AI is a Beijing company whose Kimi chat app was known, from 2023, for reading very long documents. Its open-weight story starts in 2025 and is short, fast, and unusually instructive: first a new optimizer, then a trillion-parameter model on someone else's architecture, then an architecture of its own. This piece follows the models in order and what each means for people who serve them.

k1.5: reasoning through reinforcement learning

Kimi k1.5 was published on the same day as DeepSeek-R1, in January 2025, and made a similar case: reinforcement learning on long chains of thought, with long-context training as a key ingredient, and "a simplistic, effective RL framework without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models". Unlike R1, k1.5 stayed a report and a product; its weights were not released.

Moonlight: Muon at scale

Moonshot's first open release, in February 2025, was really about an optimizer. Muon updates each weight matrix with an orthogonalized version of its gradient instead of AdamW's per-number scaling, and had worked on small models. The paper Muon is Scalable for LLM Training found two changes that made it work at scale (weight decay, and per-parameter update scaling) and measured "~2×" computational efficiency against AdamW. To prove it, Moonshot released Moonlight, a 16-billion-parameter MoE with 3 billion active, trained on 5.7 trillion tokens, and its distributed Muon implementation.

For people who serve models, an optimizer changes nothing directly. What it changed was the cost of training the next model, and Moonshot's ability to afford a trillion parameters.

K2: a trillion parameters on DeepSeek's architecture

Kimi K2 (July 2025) has 1 trillion parameters, 32 billion active per token. Its config declares DeepseekV3ForCausalLM: it is DeepSeek-V3's architecture, multi-head latent attention and DeepSeek-style MoE, with different proportions.

Two choices stand out. More experts (384 against 256) with the same number active makes the model sparser: more knowledge for the same compute per token. Half the attention heads (64 against 128) makes attention cheaper, which matters for long agent sessions; the cache per token is the same, since MLA caches one latent vector regardless of the number of heads.

The training contribution was MuonClip: Muon with "a novel QK-clip technique to address training instability", rescaling the query and key weights when attention scores grow too large. K2 was pre-trained on 15.5 trillion tokens "with zero loss spike", by the report's account, then post-trained for agents with large-scale synthesis of tool-use data and reinforcement learning in real and simulated environments.

For engines, K2 was nearly free to support: anything that served DeepSeek-V3 could serve K2. That is the quiet advantage of reusing an architecture. K2-Instruct-0905 (September 2025) doubled the context to 256K.

K2 Thinking: four bits, by training

K2 Thinking (November 2025) turned K2 into a reasoning agent that interleaves thinking with tool calls over hundreds of steps. Its serving-relevant detail is in the weights: it ships in INT4, and the precision was not applied after training but trained in. Quantization-aware training "in post-training stage" let Moonshot report a "lossless 2x speed-up in low-latency mode". A trillion parameters in 4 bits is about 500 GB of weights instead of a terabyte in FP8, the difference between one 8-GPU node and two. Our NVFP4 piece covers what 4-bit weights do to decoding speed.

K2.5 to K2.7: vision, swarms, coding

• K2.5 (January 2026) continued K2's pretraining on about 15 trillion mixed text and image tokens, with a 400-million-parameter vision encoder, MoonViT, and introduced an "agent swarm" mode in which the model splits a task among sub-agents it creates.
• K2.6 (April 2026) scaled that to "300 sub-agents executing 4,000 coordinated steps" and focused on long-horizon coding and design.
• K2.7-Code (June 2026) is a coding-focused version of K2.6.

The architecture stayed K2's throughout: the same DeepSeek-V3 text model, now with vision.

Kimi Linear and K3: Kimi Delta Attention

In October 2025, Moonshot published Kimi Linear, its own attention design, with a small model to prove it: 48 billion parameters, 3 billion active. Its core is Kimi Delta Attention (KDA), a linear attention that extends Gated DeltaNet with finer-grained gating. Like all linear attention, it keeps a fixed-size state instead of a cache that grows with each token; the delta rule lets the state overwrite what it stores rather than only accumulate, which makes the limited memory more useful. Kimi Linear interleaves three KDA layers with one latent-attention (MLA) layer, and the paper reports that it outperformed full MLA "under fair comparisons", while cutting the KV cache by up to 75% and decoding up to 6 times faster at a million tokens. The KDA kernel was released in the open flash-linear-attention library.

K3 (June 2026) is that design at full scale: 2.8 trillion parameters with 104 billion active, 69 KDA layers and 24 latent-attention layers, 16 of 896 experts per token, a one-million-token context, and native vision. Its weights ship as MXFP4 with MXFP8 activations, again from quantization-aware training. Per token of context, only the 24 MLA layers keep a cache, 27 KiB in BF16, less than half of K2's 68.6 KiB even though the model is almost three times larger.

MiniMax tried linear attention at scale and stepped back from it (see the MiniMax piece); Moonshot spent a year proving its version on a small model first, then bet its flagship on it.

The models at a glance

From the papers, model cards, and configs, 2026-10-06. Kimi Linear's context is from its card.

The licenses

The K2 family uses a "Modified MIT" license: MIT, plus one condition: a commercial product with more than 100 million monthly active users or more than 20 million US dollars of monthly revenue must display "Kimi K2" in its interface. K3's license adds a second condition: a company running a model-as-a-service business (third parties controlling inference or fine-tuning, typically through an API) with more than 20 million US dollars of revenue over 12 months needs a separate agreement with Moonshot before commercial use. Internal use, and products that only embed the model in their own features, are exempt.

What it means if you serve them

• K2 to K2.7 run wherever DeepSeek-V3 runs, with the same MLA kernels and expert parallelism; plan for a trillion parameters (about 1 TB in FP8, about 500 GB in INT4).
• Use the shipped precision. K2 Thinking and later were trained for INT4, and K3 for MXFP4: those are the reference weights, not an approximation of them.
• K3 needs hybrid-attention support: an engine that manages KDA's fixed-size state alongside an MLA cache, as for the models in our hybrid attention piece, and MXFP4 kernels, native on Blackwell.
• Mind the license thresholds if you sell API access: K3's model-as-a-service clause applies above 20 million US dollars a year.

Next

This closes the series on the open-weight labs. Each has a different answer to the same questions of attention, precision, and licensing, and each answer shows up directly in what it costs to serve the model.