Engram: Memory You Look Up Instead of Compute
DeepSeek's Engram adds a hashed n-gram memory that a model looks up instead of computing: how it works, why it helps, and how DeepSeek-V4.1 and Qwen serve it.
qwen
3 posts
Qwen3.8-Flash-Next, Layer by Layer
Qwen's preview of its next architecture, piece by piece: linear and sparse attention, a gated four-branch residual, 512 experts and an n-gram memory, and what each costs to serve.
When Most Layers Stop Caching: Hybrid, Sliding, and Sparse Attention, Sized
Why a 27B model holds 262k tokens of context on a 24 GB GPU: what hybrid, sliding-window, and sparse attention store and read per token, on real models.