Language Models Can Control Their Own Attention — part 5/6

Cleaned-text code points [61444, 77286). Bibliography omitted here; source.md includes it.

We define the roofline wall-time of an operation as the time it would take on hardware operating at its ceiling in the relevant regime. For an operation with utilization u u (MFU for compute-bound, MBU for memory-bound) and peak rate R R (Peak FLOPS or Peak BW respectively), the roofline wall-time is: T roofline = work R × u T_{\text{roofline}}=\frac{\text{work}}{R\times u} where work is FLOPs for compute-bound operations and bytes for memory-bound operations. This metric is defined per hardware target (peak rates differ across accelerators) but is invariant to specific operating choices like batch size within that target. This makes roofline wall-time a useful unit for comparing operations on saturated hardware without committing to a particular deployment configuration.

### C.5 Representative utilization values

#### MFU.

Model FLOPs Utilization ( Chowdhery et al., 2023 ) measures the fraction of peak compute an operation actually achieves on hardware. Reported values for well-optimized FFN at large batch span roughly 40% to 60%: PaLM 540B training reached 46.2% MFU on TPU v4 (the 57.8% sometimes quoted alongside it is Hardware FLOPs Utilization, which additionally counts activation-rematerialization FLOPs, not a higher MFU) ( Chowdhery et al., 2023 ) , and Pope et al. report 76% at large-batch prefill on PaLM 540B, falling to roughly 40% at decode where the per-step GEMMs are skinnier ( Pope et al., 2023 ) . The Databricks practitioner reference reports comparable numbers across LLaMA-class models on H100 ( Agarwal et al., 2023 ) . DA operates at decode, which sits at the lower part of this range, and a Mixture-of-Experts decoder lowers it further still as its skinny per-expert GEMMs reach lower utilization; we adopt a representative FFN/expert decode MFU of 40%.

#### MBU.

Model Bandwidth Utilization ( Agarwal et al., 2023 ) measures the fraction of peak memory bandwidth an operation achieves. Reported values for well-optimized memory-bound kernels span roughly 60% to 85%: Databricks reports 60% MBU on 2 × \times H100 at batch 1 for MPT-7B and higher for larger models, with the achieved fraction falling as batch size grows and the bottleneck shifts toward compute ( Agarwal et al., 2023 ) . FlashAttention-3’s reported 75% on H100 is a compute-throughput figure for a compute-bound forward pass rather than a decode-time bandwidth utilization ( Shah et al., 2024 ) . We adopt a representative MBU of 70%, in the upper half of this range, reflecting a well-tuned memory-bound kernel at large-batch decode. This 70% is a large-batch ceiling for a well-tuned 8-bit-KV kernel; our own decode kernels at low batch run well below it (FlashAttention paths near 57%, Triton paths lower), so 70% is conservative for the regime we model and should not be transplanted to low-concurrency serving.

### C.6 The operating regime: production disaggregation at scale

The roofline wall-time framework models each operation at its own hardware ceiling, which presumes that compute-bound and memory-bound work do not contend for the same resource within a step. This is the regime large-scale production serving has converged on, and we adopt it as the operating assumption throughout. Modern systems disaggregate the two phases of inference onto separately sized and scaled GPU pools: compute-bound prefill and memory-bound decode are served independently, so each runs at its own bottleneck ( Zhong et al., 2024 ; Patel et al., 2024 ) . This is not merely an academic design. It runs in production behind frontier assistants, where decode is placed on its own pool organized around the KV-cache bandwidth bottleneck ( Qin et al., 2024 ) and, in DeepSeek’s V3 deployment, scaled onto roughly ten times the GPUs of prefill ( DeepSeek-AI, 2024b ) . A further step disaggregates within the decode step itself, placing the memory-bound attention operator and the compute-bound FFN or expert operators on dedicated, differently provisioned accelerators ( Chen et al., 2024 ; Liang et al., 2025 ; Zhu et al., 2025 ) .

In this regime the roofline wall-time decomposition describes the deployment rather than idealizing it: each operation runs on hardware sized for its bottleneck and approaches its own ceiling, so DA’s reduction of the memory-bound attention KV read directly shortens the stage that carries it. The assumption is conservative for DA in one respect. When attention and FFN are co-located, as in smaller or lower-concurrency deployments, the two contend for the GPU and the per-operation ceilings are only approached, making the roofline wall-time a ceiling estimate rather than an exact account. The regime DA targets, and where its savings matter most, is the disaggregated, large-batch, long-context one.

### C.7 Implication: attention dominates at scale

The attention roofline wall-time grows with two quantities: the KV cache size (proportional to context length) and the number of decode steps over which KV reads are repeated. The FFN roofline wall-time grows only with the number of decode steps, its per-step cost fixed by the FLOPs budget once weight loads are amortized. At large enough batch and context, the attention term therefore comes to dominate the FFN term. As a concrete illustration, consider a single decode step on Qwen-3.5-397B-A17B at 1M context, the kind of large Mixture-of-Experts model these deployments serve. Its 15 global attention layers hold roughly 15.4 GB of KV cache per sequence at 8-bit precision ( 2 × 15 2\times 15 layers × 2 \times\,2 KV heads × 256 \times\,256 head dim × 1 ​ M \times\,1\text{M} tokens), while the 45 Gated DeltaNet (GDN) layers add only a fixed, context-independent state. On a B200-class accelerator with peak HBM bandwidth around 8 TB/s and peak compute around 4.5 PFLOPS dense in FP8 (the low-precision regime consistent with the 8-bit KV cache; dense BF16 is roughly 2.25 PFLOPS), the attention roofline wall-time for that step is roughly 15.4 ​ GB / ( 8 ​ TB/s × 0.70 ) ≈ 2.7 15.4\penalty\ \text{GB}\,/\,(8\penalty\ \text{TB/s}\times 0.70)\approx 2.7 ms. The FFN roofline wall-time, for the 17B active parameters at a representative FFN/expert decode MFU of 40%, is roughly 34 ​ GFLOPs / ( 4.5 ​ PFLOPS × 0.40 ) ≈ 0.019 34\penalty\ \text{GFLOPs}\,/\,(4.5\penalty\ \text{PFLOPS}\times 0.40)\approx 0.019 ms; under the expert parallelism these models are served with, the per-expert GEMM aggregates enough tokens across the decode batch to reach its compute-bound regime, though its skinny per-expert GEMMs and routing hold utilization at the low end of the range. Attention dominates by roughly two orders of magnitude ( ≈ 145 × \approx 145\times here, and within the ∼ 120 – 255 × {\sim}120\text{--}255\times band across the full plausible MFU and MBU ranges, so the conclusion does not hinge on the exact utilizations). The per-sequence KV read is comparable to the model’s active-parameter load ( ∼ 17 {\sim}17 GB in FP8), but the parameter load amortizes across the batch while the per-sequence KV read does not, so attention dominates once the batch is non-trivial. DA’s per-step attention reduction therefore lands where total decode work concentrates, with savings largest in large-batch, long-context serving.

### C.8 Worked decode wall-time decomposition for the headline models

Section 5.4 reports the decode wall-clock time of vanilla and DA on the two headline models. Here we derive those numbers. Unlike the frontier-scale illustration in Section C.7 , which uses an FP8 production setting, our experiments run in bf16 end to end, so we charge compute at the dense bf16 peak of 2.25 PFLOPS and read a bf16 KV cache. The hardware is a single B200 accelerator with 8 TB/s of HBM bandwidth, at an FFN/expert decode MFU of 40% and an attention MBU of 70% ( Section C.5 ). Every quantity is summed over the decode steps of one response and averaged across the 15 tasks.

We split one decode step into the three costs of Section 3 .

#### Matmul (compute-bound).

Each step multiplies the current token against the model’s weight matrices: the attention and GDN projections, the feed-forward layers, and the output head. These weights load once and are reused across the batch, so the operation is compute-bound at scale ( Section C.3 ) and its time is set by FLOPs. One token costs about 2 ​ P 2P FLOPs for P P active parameters, so over D D decode steps the work is 2 ​ P ​ D 2PD and the time is 2 ​ P ​ D / ( 2.25 ​ PFLOPS × 0.40 ) 2PD\,/\,(2.25\penalty\ \text{PFLOPS}\times 0.40) .

#### Global memory (memory-bound).

The global attention layers read their KV cache from HBM every step. One token-position’s keys and values across all global attention layers occupy b kv = 2 × L global × n kv × d head × 2 b_{\text{kv}}=2\times L_{\text{global}}\times n_{\text{kv}}\times d_{\text{head}}\times 2 bytes, where the leading 2 counts keys and values and the trailing 2 is bf16. Writing A A for the total token-positions the response attends to, summed across decode steps (the attended-token count of Section 2 , which the mask lowers directly), the read is A × b kv A\times b_{\text{kv}} bytes and the time is A ​ b kv / ( 8 ​ TB/s × 0.70 ) A\,b_{\text{kv}}\,/\,(8\penalty\ \text{TB/s}\times 0.70) .

#### Local memory (memory-bound).

The efficient layers add a fixed per-step read that does not grow with context. On Qwen this is the GDN recurrent state, about 78.45 MB across its 48 linear-attention layers. On Gemma this is the sliding window attention (SWA) KV cache, about 838.9 MB across its 50 window layers at a 1024-token window. Over D D steps the read is D × s local D\times s_{\text{local}} bytes and the time is D ​ s local / ( 8 ​ TB/s × 0.70 ) D\,s_{\text{local}}\,/\,(8\penalty\ \text{TB/s}\times 0.70) . DA does not touch this term.

The per-model constants follow from each published configuration in bf16 ( Section 4 ).

The two trace quantities that vary by arm, the decode-step count D D and the attended-token total A A , are measured directly from the generated responses ( Section 5 ).

As a worked example, vanilla Gemma-4-31B . Matmul: 2 × 31 ​ e ​ 9 × 332 = 2.06 ​ e ​ 13 2\times 31\mathrm{e}9\times 332=2.06\mathrm{e}13 FLOPs, divided by 2.25 ​ PFLOPS × 0.40 = 0.9 2.25\penalty\ \text{PFLOPS}\times 0.40=0.9 PFLOPS, gives 22.9 ms. Global: 13.43 ​ e ​ 6 × 81,920 ​ B = 1.10 13.43\mathrm{e}6\times 81{,}920\penalty\ \text{B}=1.10 TB, divided by 8 ​ TB/s × 0.70 = 5.6 8\penalty\ \text{TB/s}\times 0.70=5.6 TB/s, gives 196.5 ms. Local: 332 × 838.9 ​ MB = 278 332\times 838.9\penalty\ \text{MB}=278 GB, divided by 5.6 TB/s, gives 49.7 ms. The decode total is 22.9 + 196.5 + 49.7 = 269.1 22.9+196.5+49.7=269.1 ms. Substituting the DA arm ( D = 448 D=448 , A = 6.45 ​ M A=6.45\text{M} ) and the Qwen constants gives the remaining rows of Section 3 .

Three features of the result are worth stating. First, the global read is the dominant decode cost under vanilla, 73% of decode time on Gemma and 86% on Qwen, so reducing it is where DA’s leverage lies. Second, DA raises the decode-step count (31% on Qwen, 35% on Gemma), which lifts the context-independent matmul and local terms, but the global read falls by enough that the decode total still drops to 0.71 × 0.71\times of vanilla on Gemma and 0.77 × 0.77\times on Qwen. Third, the local floor sets how much of the global saving survives: it is 5% of DA’s attention time on Qwen, whose GDN state is small, but 42% on Gemma, whose SWA layers cover most of the stack, which is why Gemma’s larger global reduction does not translate into a proportionally larger total saving.

This decomposition is a ceiling estimate, not a measurement. It omits prefill, which phase-disaggregated serving runs on a separate pool ( Section C.6 ), and it omits negligible per-step operations such as normalization, rotary embeddings, residual adds, and the small GDN convolution state. It also counts attended tokens at token granularity; the block-aligned masking of the deployed kernel ( Section B ) reads a few percent more on the global term, which we do not model here.

### C.9 KV bytes read per context token across current architectures

The global memory term of Section C.8 scales with b kv b_{\text{kv}} , the bytes one decode step reads per context token across the global attention layers. This quantity spans three orders of magnitude across models in current use, so the wall-time value of removing an attended token depends on which architecture removes it. Section 6 surveys it as of August 2026. Every figure follows from the published configuration of the model and from the cache layout its serving stacks allocate, rather than from a vendor-quoted cache size.

Two quantities come apart once attention is sparse. Stored bytes are what a context token occupies in the KV cache for the life of the sequence. The O(N) read is what a single decode step must touch. For full attention the two coincide, because the step reads the whole cache. For the indexer-based designs the step scans only the indexer key cache at O(N), scores it, and then reads a fixed top- k k set of main-attention entries. That top- k k read does not grow with context, so it belongs in the local term of Section C.8 rather than in b kv b_{\text{kv}} . The gap between the two reaches a factor of 33 on Qwen3.8-Flash-Next .

Precision is the other axis, and it is not a free parameter. We report each model twice, once at a uniform bf16 cache so the architectures are comparable, and once at the dtype its vendor’s serving recipe specifies. Four models are served at FP8 and DeepSeek-V4 at a forced mix of FP8 latents, bf16 rotary dimensions and FP4 indexer keys. Where a cache width is fixed by the architecture rather than by a serving flag, the two columns agree. This is why the O(N) read of the indexer-based models is identical in both columns, since both stacks hold indexer keys at a hardcoded width independent of the KV dtype flag.

The per-model derivations are one line each, at 1000 bytes per KB. Where the leading factor is 2 it counts keys and values, and the trailing factor is the dtype width in bytes.

• MiniMax-M2.7 : 2 × 62 × 8 × 128 × 2 = 253,952 2\times 62\times 8\times 128\times 2=253{,}952 B. All 62 layers run full attention, and the model has no linear or sliding-window layers, so it holds no context-independent state at all. Its native context ceiling is 204,800 tokens.

• Qwen3.8-Max : 2 × 23 × 4 × 256 × 1 = 47,104 2\times 23\times 4\times 256\times 1=47{,}104 B as served, twice that at bf16. The 23 gated attention layers sit in a 92-layer stack whose other 69 layers are Gated DeltaNet.

• Gemma-4-31B : 2 × 10 × 4 × 512 × 2 = 81,920 2\times 10\times 4\times 512\times 2=81{,}920 B. Its global layers derive keys and values from one projection and carry no separate value weights, so an architectural floor of 40,960 B exists. No stack realizes it, because transformers, vLLM and SGLang all duplicate the key weights into a value slot and allocate a two-tensor paged cache.

• Qwen-3.6-27B : 2 × 16 × 4 × 256 × 2 = 65,536 2\times 16\times 4\times 256\times 2=65{,}536 B, the constant already used in Section 4 .

• Kimi-K3 : 24 × 576 × 1 = 13,824 24\times 576\times 1=13{,}824 B as served, twice that at bf16. MLA caches one 576-wide latent that serves as both key and value, so there is no leading factor of 2. The remaining 69 layers of its 93-layer stack are Kimi Delta Attention.

• MiniMax-M3 : stored 60 × 2,048 + 57 × 256 = 137,472 60\times 2{,}048+57\times 256=137{,}472 B, read 3 × 2,048 + 57 × 256 = 20,736 3\times 2{,}048+57\times 256=20{,}736 B. Its 57 MiniMax Sparse Attention layers are global but sparse, so only their 256-byte index branch is read at O(N). The other three layers are full attention and read their whole KV cache, which is why its O(N) read is an order of magnitude above most of the other sparse designs.