Engram: Memory You Look Up Instead of Compute
DeepSeek's Engram adds a hashed n-gram memory that a model looks up instead of computing: how it works, why it helps, and how DeepSeek-V4.1 and Qwen serve it.
moe
8 posts
Qwen3.8-Flash-Next, Layer by Layer
Qwen's preview of its next architecture, piece by piece: linear and sparse attention, a gated four-branch residual, 512 experts and an n-gram memory, and what each costs to serve.
DeepGEMM: The Matrix Multiplications Behind DeepSeek's Models
What DeepGEMM computes for DeepSeek's models: block-scaled FP8 and FP4 matrix multiplications, grouped GEMMs for MoE experts, and the fused Mega MoE kernel.
Expert Parallelism, and What DeepEP Does for It
Why serving a big MoE model means spreading its experts over many GPUs, what dispatch and combine cost, and how DeepEP and its V2 redesign make it fast.
DeepSeek: From V1 to V4, the Models and the Breakthroughs
DeepSeek's models in order, the idea behind each one (MoE, latent attention, FP8 training, RL reasoning, sparse attention), and what each changed for everyone else.
Z.ai's GLM Models: From ChatGLM to GLM-5
Z.ai's (Zhipu's) GLM models in order, from the GLM pretraining objective to agentic MoE models, the idea behind each step, and what each changed for the ecosystem.
MiniMax: Lightning Attention, Agents, and What They Tried
MiniMax's models in order: lightning (linear) attention at a 4-million-token context, reasoning, the return to full attention for agents, and what each choice taught the ecosystem.
Kimi: Moonshot AI's Models, from Long Context to Trillion-Parameter Agents
Moonshot AI's Kimi models in order, from long-context chat to trillion-parameter agentic MoE models, the idea behind each step, and what each changed for the ecosystem.