Engram: Memory You Look Up Instead of Compute

A transformer spends its first layers rebuilding things it could look up: that "Alexander the Great" names one person, that "New York" is a city, that for i in is followed by range. Engram, from DeepSeek's January 2026 paper, gives the model a table for those patterns, indexed by the last few tokens, so it looks them up in one step instead of computing them. Two production models now ship one, DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, and both are designed to keep that table in host memory, off the GPU.

Memory you don't compute

A mixture-of-experts (MoE) model already avoids running most of its parameters: a router sends each token to a few experts out of hundreds. That is conditional computation: capacity grows, the cost per token doesn't. But every active parameter still does arithmetic, and the paper argues that much of what early layers compute is retrieval in disguise: recognizing a multi-token name, an idiom, a code pattern. Most of the time the context doesn't change the answer: the same tokens call for the same pattern. Step 4 below handles the exceptions.

Engram adds a second kind of sparsity, conditional memory: parameters that are never multiplied, only read. Each position reads a handful of rows from a huge table, chosen by the tokens just before it. The table can hold billions of parameters, and the cost per token stays a few rows.

One lookup, step by step

The lookup needs nothing but the token ids: only the last step, the gate, looks at the hidden state.

For each position, Engram does five things:

1. Normalize the tokens. A fixed many-to-one map collapses token ids that differ only in case, accents or a leading space, so Apple, apple and APPLE count as one. On a 128k vocabulary, the paper's map removes 23% of the ids. DeepSeek keeps this step; Qwen dropped it.
2. Form suffix n-grams. The last 2 tokens and the last 3 tokens (DeepSeek-V4.1 also uses the last 4) are the keys. An n-gram is just that: n consecutive tokens.
3. Hash them. There are far too many possible n-grams to give each a row, so each n-gram goes through several hash functions (hash heads, 8 per n-gram order), each mapping it to one row of its own table, sized to a prime number of rows to spread the keys evenly. A hash function turns a key into a table index; two different n-grams sometimes land on the same row, and using several heads makes it unlikely that they collide in all of them. The retrieved rows are concatenated into one memory vector.
4. Gate it. The memory vector is a prior: it knows nothing about the sentence beyond the last few tokens, and a hash collision can bring back a row that belongs to another n-gram. So the model compares it with its hidden state, which has already seen the context through attention: a sigmoid of their scaled dot product gives a gate between 0 and 1. When the memory contradicts the context, the gate closes.
5. Add it. The gated vector is added to the residual stream (in the paper, after a short causal convolution; Qwen keeps one, with kernel size 4). DeepSeek's hyper-connections and Qwen's gated residual both keep 4 parallel copies of the residual stream, mixed with learned weights; each copy gets its own gate on a shared value.

Where to put it is a trade-off. Early is better for quality: in the paper's ablation, layer 2 is the best single placement (it beats layer 1 and every deeper layer), because the point is to spare the first layers. Later gives more time to fetch the rows, which matters once the table leaves the GPU. DeepSeek-V4.1-Flash uses layers 1 and 14; Qwen3.8-Flash-Next uses layer 2.

What memory buys

The paper's main experiment holds everything fixed except where the sparse parameters go. Both models have 26.7 billion parameters, 3.8 billion active per token, and were trained on the same 262 billion tokens. The MoE baseline puts all its spare capacity into 72 experts; Engram-27B keeps 55 experts and moves 5.7 billion parameters into memory.

Same parameters, same compute per token: trading 17 of 72 experts for memory wins on knowledge, reasoning and long context.

Knowledge benchmarks improve, as you would expect from a memory, but so do reasoning and code: BBH by 5.0 points, MATH by 2.4, HumanEval by 3.0. The paper's explanation is that the early layers, no longer busy reassembling static patterns, behave like extra depth for the rest. Long context improves most: after extension to 32k tokens, and compared at matched pretraining loss, multi-query needle-in-a-haystack goes from 84.2 to 97.0, as local patterns no longer compete with long-range retrieval for attention.

More memory is not always better. Sweeping the share of sparse parameters between experts and memory gives a U-shaped loss curve: all-MoE is worse than a mix, and so is too much memory. The best split stays at roughly 75 to 80% experts and 20 to 25% memory across the compute budgets tested. Engram-27B sits at 74% experts.

Switching the memory off at inference shows what it holds: TriviaQA keeps only about 29% of its score, while reading comprehension keeps 81 to 93%. Facts live in the table; reasoning over the text in front of the model mostly doesn't.

Serving tables that don't fit

Here is the property that matters for serving: the rows a token needs depend only on the token ids. No hidden state, no routing decision. The engine can compute every hash the moment a batch is scheduled, before the first layer runs, and start fetching the rows while the GPU computes the layers before the Engram layer. DeepInfer, DeepSeek's own engine, does exactly that: it hashes the batch before the layer loop and hands each Engram layer a callback that returns its rows.

So the tables don't need to be in GPU memory. The paper measured the extreme case: a 100-billion-parameter table entirely in host memory, read over PCIe on an H800, cost 1.9% of throughput on a 4B dense model and 2.8% on an 8B one.

The production numbers are large. DeepSeek-V4.1-Flash has, per Engram layer, 24 hash heads (2-, 3- and 4-grams, 8 heads each) of about 16 million rows, each row 256 FP8 values plus 8 bytes of scales: 94.4 GiB per layer, about 189 GiB for the two. A token touches 48 of those rows.

A token reads 12.4 KiB out of 189 GiB, and the engine knows which 12.4 KiB before the forward pass starts. The early layer is the tight one.

All three engines that serve V4.1-Flash can keep the tables in host memory; only vLLM does it by default:

• vLLM v0.32.0 does it by default: pinned host memory (locked in RAM so the GPU can read it directly), read by a Triton kernel through unified virtual addressing (one address space for CPU and GPU memory), on a side CUDA stream. --engram-config.cpu_offload false keeps them on the GPU. When several data-parallel replicas (full copies of the model serving different requests) share a node, dp_shared_memory keeps one copy of the tables per node in /dev/shm instead of one per replica.
• SGLang v0.5.22 does it on request (SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1). The SGLang team measured 36% more KV cache capacity on 4 GB300 GPUs (TP4, EP4), with comparable decode throughput and time to first token, and all 28 greedy probe completions matched the baseline.
• DeepInfer keeps them in host memory on NVIDIA, read over RDMA, and in device memory on Ascend.

The freed GPU memory goes to the KV cache, which is usually what limits how many requests run at once.

The cost moved from bandwidth to latency. A lookup reads many small rows scattered over 189 GiB. vLLM's engineers found that on GB200, lookups are bound by address translation, not bandwidth: each row misses the GPU's page-translation cache when the table sits on small memory pages. Sorting the rows by address made large lookups up to 3.7 times faster (32K tokens per rank, kernel benchmark on one GB200). Keeping the rows in FP8 end to end instead of expanding them to BF16 halved the bytes moved between data-parallel ranks and made the Engram step about twice as fast on 4 GB300 GPUs (#59923).

Two production designs

DeepSeek built Engram into V4.1. Qwen's preview of its next architecture has the same mechanism under another name, "N-gram embedding" in the paper, "PLE" in its config. The Qwen3.8-Next paper cites the Engram paper, borrows its context gate, and reports that Engram's token normalization gave no consistent gain in its own recipe, so Qwen's table hashes raw token ids; it does not benchmark against Engram. vLLM serves both through one code path and calls Qwen's version "same mechanism, called PLE".

Two things stand out. Both teams land on the same recipe: early placement, short n-grams, 8 heads per order, a context gate, several residual branches. And both land near the paper's split. Counted the paper's way, as a share of the parameters a token doesn't use, memory is about 27% for DeepSeek (196 billion against roughly 540 billion idle backbone parameters) and about 30% for Qwen (51 billion against roughly 119 billion), just past the 20 to 25% optimum. Qwen's paper sweeps the table from 20 to 200 times its 250K vocabulary: the loss keeps falling, downstream scores flatten, and the shipped model sits in the middle of the sweep, at about 80 times.

What it changes for you

If you serve one of these models, the memory plan has a third line next to weights and KV cache: the tables, which want host RAM, not HBM.

• Size host memory per node, not per GPU. About 189 GiB for V4.1-Flash and about 95 GiB for Qwen3.8-Flash-Next in BF16. For V4.1-Flash in vLLM, that is once per node with shared tables and once per replica with neither shared nor DP-sharded tables: on a node running 8 data-parallel replicas, 189 GiB against about 1.5 TiB. vLLM shards Qwen's table across GPUs instead (expert-tensor parallelism), so size it once per model instance.
• The CPU-GPU path is now in your critical path. PCIe, NVLink-C2C on Grace systems, or RDMA. On GB200, vLLM found page size mattered more than raw bandwidth: check huge pages before you blame the link.
• The freed HBM is KV cache. On V4.1-Flash that was 36% more in SGLang's test: room for more concurrent requests or longer contexts before the engine has to queue.

On a unified-memory machine, where CPU and GPU share one pool, "offloading" moves nothing: the tables still take the same bytes out of the pool, and what you plan for is total memory, not where it sits.

Gotchas

No huge pages, slow lookups. Host tables on 4 KiB pages make every row a translation miss. SGLang warns "expect ~10x slower lookups" when it finds no huge pages; check /sys/kernel/mm/transparent_hugepage and the startup log.

/dev/shm too small in containers. vLLM's shared tables live in /dev/shm; Docker's default is 64 MB. Without --shm-size or --ipc=host, vLLM falls back to per-replica (or DP-sharded) tables, with a warning, and your host memory plan can be off by the number of replicas.

Engram and micro-batching don't mix in vLLM. For V4.1-Flash, vLLM requires dual-batch overlap (--enable-dbo, which splits each batch in two to hide communication behind compute) off and --ubatch-size 0.

The hash depends on the tokenizer map. The multipliers derive from the compressed vocabulary size. An engine whose tokenizer normalization differs from the checkpoint's reads the wrong rows, and nothing crashes: vLLM checks the size at startup for that reason.

The same table appears in Qwen3.8-Flash-Next, layer by layer, next to the other four ideas in that model; DeepInfer shows how DeepSeek serves it in production.