When Most Layers Stop Caching: Hybrid, Sliding, and Sparse Attention, Sized

Every full-attention layer keeps a key and a value for every token a request has seen, so the cache grows with every token, in every layer. Many open models released this year stop doing that. This piece shows the four ways they do it, on five real models, and what each way costs you when you serve them.

Two costs to watch

The previous piece sized the cache: for Qwen3-8B, 144 KiB per token, so 4.5 GiB for one 32,768-token request. That is the first cost, memory held: it decides how many requests fit on your GPU at once.

The second cost is easier to miss. To produce each new token (a decode step), the GPU reads the model's weights once (for a mixture-of-experts model, the experts in use), and attention reads what each layer kept for every request in the batch. Decoding is limited by how fast that data moves from memory, more than by computation. With long requests, or many of them, the cache becomes the larger part of what is read. So the second cost is memory read per new token, and it decides how fast tokens come out.

With full attention the two are the same: every layer keeps every token and reads all of it, every step. The designs below break that link, in different places.

Four ways to remember

Models with no full attention at all exist (Mamba and RWKV are the best known), but all five models here keep some full-attention layers, which can still look at any past token exactly, and make the other layers cheaper. Here is how five models mix their layers, read from their config.json files:

1. Full attention: keep everything (Qwen3-8B)

How it works. Each layer stores a key and a value for every token, and every new token compares itself with all of them. With grouped-query attention (GQA), several query heads share one key and value head, which already divides the cache: Qwen3-8B has 32 query heads but stores only 8 KV heads.

What it costs. 144 KiB per token for Qwen3-8B, held and read in full at every step. Both costs grow in a straight line with the context.

Strengths. Exact recall of anything in the context, and the best-supported design: every engine, kernel, and optimization (prefix caching, paging, speculative decoding) was built for it first.

Weaknesses. Long contexts become expensive in memory and slow to decode. Qwen3-8B handles 32,768 tokens natively and 131,072 with YaRN scaling; at 131,072 tokens, one request holds 18 GiB of cache.

2. Sliding window + full layers (gpt-oss, Kolibri-1)

How it works. A sliding-window layer only looks at the last W tokens. Older tokens are dropped from that layer's cache, so it stops growing once it holds W tokens. The full-attention layers in between still see everything, and they carry long-range information for the rest of the model.

• gpt-oss-120b alternates: 18 window layers (W = 128) and 18 full layers.
• Kolibri-1 goes further: four window layers (512 tokens back) for every full one, so 40 of its 50 layers stop growing. Its card adds a detail: position information (RoPE) is applied only in the window layers, and because of that, it says, the context can be extended beyond its native 262,144 tokens without position scaling.

What it costs. Only the full layers grow: 36 KiB per token for gpt-oss-120b, 20 KiB for Kolibri-1. The window layers add a fixed amount per request (4.5 MiB and 40 MiB).

Strengths. Simple. A window layer is ordinary attention with a mask, so existing kernels, paging, and quantized caches work almost unchanged. Nearby tokens, which matter a lot for the next word, are kept exactly.

Weaknesses. A window layer cannot see far back on its own; anything older must reach it through the full layers. The savings are capped by the ratio: with one full layer in two, gpt-oss still pays half the cost of full attention.

3. Linear attention + full layers (Qwen3.8-27B)

How it works. Qwen3.8-27B repeats a block of four layers sixteen times: three Gated DeltaNet layers, then one full-attention layer. A Gated DeltaNet layer keeps no per-token cache at all. Each head keeps one small matrix, its state, and every token is written into it: a gate fades old content, and a "delta rule" replaces what the state holds for a key with the new value, instead of adding it on top (the Gated DeltaNet paper). Reading the state answers "what do I remember that matches this token?" without looking at the tokens themselves. The state has the same size whether the request is 10 tokens long or 200,000.

What it costs. The 16 full layers grow by 64 KiB per token. The 48 Gated DeltaNet layers hold a fixed state: 48 heads × 128 × 128 values per layer, about 75 MiB per request in BF16. At its full 262,144-token context, a request holds 16 GiB of cache instead of the 64 GiB that 64 full-attention layers would need.

Strengths. Long contexts at a quarter of the cost, and a decode step that slows down far less as the request grows: in three quarters of the layers, the work per token does not depend on the context at all.

Weaknesses.

• The state is a summary, not a copy. It cannot recall an exact token the way attention can; the full layers are there for that.
• A fixed cost per request, even for short ones. Those 75 MiB are paid at the first token. Below about 400 tokens, the state takes more memory than those 48 layers would need if they were full attention like the other 16. A server full of short chats gains little.
• More work for the engine. Two common optimizations get harder. Prefix caching (reusing the cache of a prompt prefix that many requests share) must store snapshots of the state, because a state cannot be cut back to an earlier token the way a list of cache blocks can. Speculative decoding (a small draft model proposes several tokens; the big model checks them in one step) needs extra copies of the state to undo rejected tokens. You will see both in the server log below.

4. Compressed + sparse (DeepSeek V4 Flash)

How it works. DeepSeek V4 keeps the last 128 tokens exactly in every layer, and compresses the rest. Its reference implementation alternates two kinds of layers:

• Compressed 4×: every 4 tokens are merged into one entry by a small learned pooling. A lightweight indexer keeps its own tiny entries, scores all of them for each new token, and attention reads only the top 512 entries.
• Compressed 128×: every 128 tokens become one entry. There are few enough to read them all.

Its config.json lists the layout: the first two layers keep only the window, and the other 41 alternate, 21 compressed 4× and 20 compressed 128×. Each entry is a single 512-value vector shared by all 64 attention heads, which shrinks it further.

What it costs. About 6.7 KiB per token of context in BF16, against 144 KiB for Qwen3-8B, plus 5.4 MiB of window per request. Reads are bounded too: at a million tokens, a new token reads about 1.5 GiB across all layers, much less than the 6.7 GiB held. DeepSeek's card states that at one million tokens V4-Pro needs 10% of the cache of its previous model, V3.2.

Strengths. Long contexts become cheap enough to serve: a million tokens of context, per request, at a nearly steady speed. The SGLang team reports V4 Flash decoding a single request (with speculative decoding) at 266 tokens per second at 4K context and 240 at 900K on H200 GPUs, a drop under 10% (LMSYS).

Weaknesses.

• Compression is lossy, and the indexer can pick the wrong entries; the model is trained to live with that, but exact recall of a distant detail is harder than with full attention.
• New kernels and new engine code. Compressing, indexing, and gathering scattered entries need purpose-built kernels, and the mix of window, compressed entries, and in-progress compression state broke SGLang's usual prefix cache: its team built a new one for this model. SGLang shipped support on launch day; other engines and quantized formats follow at their own pace, so check support before you plan a deployment around it.
• More moving parts to tune and to debug. The design also keeps moving: the newer V4.1 Flash changes the details again (its config has a different layer count and indexer).

See them side by side

Move the context length and compare what one request holds, then switch to what one new token reads. The darker part of each bar grows with the context; the lighter part is fixed.

Three things to try:

1. Start at 1k. Qwen3.8-27B holds almost as much as Qwen3-8B: its fixed state dominates. Then move to 32k: it holds less than half.
2. Count the requests. At 32k, 20 GiB of cache fits 4 Qwen3-8B requests, 9 for Qwen3.8-27B, 92 for DeepSeek V4 Flash. That ratio, more than the model's size, is what decides how many users a GPU serves.
3. Switch to "read per new token". For four of the models, every byte held is read at every step. Only DeepSeek V4 reads less than it holds.

On a real server: Qwen3.8-27B with SGLang

We serve Qwen3.8-27B with SGLang for our own agent work, on an NVIDIA GB10 (121 GiB of unified memory), with an FP8 cache and a draft model for speculative decoding. Its startup log shows the design at work (trimmed):

Mamba Cache is allocated. max_mamba_cache_size: 40, conv_state size: 0.11GB, ssm_state size: 2.88GB intermediate_ssm_state_cache size: 5.06GB
KV Cache is allocated. dtype: torch.float8_e4m3fn, #tokens: 1514282, K size: 23.11 GB, V size: 23.11 GB
max_running_requests is capped to 8 by the mamba state cache (max_mamba_cache_size=40, 5 state slots per request).
Init Unified Radix Cache. Components: (<ComponentType.FULL: 0>, <ComponentType.MAMBA: 2>).

Five things to read in it. (SGLang calls every fixed-state layer "mamba", after Mamba, the best-known design of this kind.)

1. The two kinds of layers get two pools. One holds the growing cache, one holds fixed states.
2. The formula holds. 1,514,282 tokens in 46.2 GiB is 32 KiB per token: 16 full layers × 2 × 4 KV heads × 256 × 1 byte in FP8. The 48 other layers cost nothing per token.
3. The state pool is what limits concurrency. It has 40 slots of about 72 MiB, and each request needs 5 of them, so the server runs at most 8 requests at once, while the token pool could hold 1.5 million tokens. On this model, memory for states, not for tokens, sets the batch size.
4. Extra copies for speculative decoding. The 5.06 GB "intermediate" pool keeps the state at each draft step, so a rejected draft token can be undone. Full attention gets this for free: you just ignore the extra cache entries.
5. Prefix caching tracks both kinds. The radix cache (SGLang's prefix cache) has two components: cache blocks for the full layers, state snapshots for the others.

The trade-offs at a glance

What this changes when you run a model

• Read the layout before the parameter count. layer_types, sliding_window, full_attention_interval, or a list of compression ratios in config.json tell you how the cache will grow. The five models above differ by more than 20 times in memory per token of context, and the layout, not the parameter count, sets that difference.
• Match the design to your traffic. Short chats gain little from a hybrid model, which still pays its fixed state. Long documents, agents that keep a growing history, and retrieval-heavy prompts gain the most.
• Check what limits the batch. On a hybrid model the state pool, not the token pool, can cap the number of concurrent requests, as the log above shows. Engines expose it (--max-mamba-cache-size in SGLang).
• Check engine support for the features you rely on. Prefix caching, speculative decoding, and quantized caches were built for full attention first. For each new design, confirm they work in your engine version before you count on them.

Next

Memory held decides how many requests fit; the scheduler decides which ones run at each step. That is the next piece: continuous batching.