Language Models Can Control Their Own Attention — part 6/6

Cleaned-text code points [77286, 91934). Bibliography omitted here; source.md includes it.

• GLM-5.2 : stored 78 × 656 + 21 × 132 = 53,940 78\times 656+21\times 132=53{,}940 B as served, read 21 × 132 = 2,772 21\times 132=2{,}772 B. All 78 layers run top-2048 sparse MLA, and 21 of them own a full-context indexer whose selection the other 57 reuse. The 656-byte main entry and the 132-byte indexer entry are the layouts vLLM and SGLang allocate, including their block scales, and the indexer entry is hardcoded FP8 in both. GLM-5.3 shares this base model.

• Qwen3.8-Flash-Next : stored 12 × ( 2,048 + 64 ) = 25,344 12\times(2{,}048+64)=25{,}344 B, read 12 × 64 = 768 12\times 64=768 B. Its 12 QSA layers keep a full-width GQA cache and score a separate indexer that stores one compressed key per micro-block of four tokens. Each layer owns its indexer, so all 12 scans are paid every step, and the vLLM QSA backend rejects every KV dtype but bf16.

• GLM-5.3-Flash : stored 11 × 512 + 11 × 128 = 7,040 11\times 512+11\times 128=7{,}040 B as served, read 11 × 128 / 4 = 352 11\times 128/4=352 B. The allocation reserves a raw-width indexer slot per token, while the scan runs over one pooled key per four tokens. The other 34 layers are Kimi Delta Attention.

• DeepSeek-V4-Pro : a compressed entry is 583 B and an indexer entry 68 B, both fixed by its tech report. Its 30 selective layers compress 4:1 and its 31 hierarchical layers compress 128:1, so stored is 30 × ( 583 + 68 ) / 4 + 31 × 583 / 128 = 5,023.7 30\times(583+68)/4+31\times 583/128=5{,}023.7 B and the read is 30 × 68 / 4 + 31 × 583 / 128 = 651.2 30\times 68/4+31\times 583/128=651.2 B. Only the indexer keys and the 128:1 entries are scanned every step.

• DeepSeek-V4-Flash : the same entry widths over 21 selective and 20 hierarchical layers give 3,508.8 3{,}508.8 B stored and 448.1 448.1 B read. Its two remaining layers are pure sliding window and hold no per-context-token state.

Model facts move quickly, and several of these checkpoints were released within the three months before this survey. The figures should be read as the state of the deployed configurations in August 2026 rather than as properties of the families.

### C.10 Attention share of decode wall-time at 1M-token contexts

Section C.8 decomposes a decode step into a compute-bound matmul term, a memory-bound global term that grows with context, and a memory-bound local term that does not. Applying that decomposition at a fixed context length N N , with the same hardware target and the same utilizations, gives T global = b kv ​ N BW eff , T matmul = 2 ​ P C , T local = s local BW eff , T_{\text{global}}=\frac{b_{\text{kv}}N}{\text{BW}_{\text{eff}}},\qquad T_{\text{matmul}}=\frac{2P}{C},\qquad T_{\text{local}}=\frac{s_{\text{local}}}{\text{BW}_{\text{eff}}}, with BW eff = 8 ​ TB/s × 0.70 = 5.6 \text{BW}_{\text{eff}}=8\penalty\ \text{TB/s}\times 0.70=5.6 TB/s and C = 2.25 ​ PFLOPS × 0.40 = 0.9 C=2.25\penalty\ \text{PFLOPS}\times 0.40=0.9 PFLOPS in bf16 or 4.5 ​ PFLOPS × 0.40 = 1.8 4.5\penalty\ \text{PFLOPS}\times 0.40=1.8 PFLOPS in FP8. The attention share is T global / ( T global + T matmul + T local ) T_{\text{global}}/(T_{\text{global}}+T_{\text{matmul}}+T_{\text{local}}) , the same numerator and denominator that produce the 73% and 86% figures of Section C.8 . The top- k k read of the indexer-based models sits in s local s_{\text{local}} , because it is fixed per step and does not grow with context. Each model is charged at the dtype its vendor’s serving recipe specifies, following Section 6 , with the compute peak taken in the matching precision as in Section C.7 .

Section 7 applies this at N = 10 6 N=10^{6} . Taking Gemma-4-31B as the worked line, the global term is 81,920 ​ B × 10 6 = 81.9 81{,}920\penalty\ \text{B}\times 10^{6}=81.9 GB divided by 5.6 TB/s, giving 14.63 ms, the matmul term is 2 × 31 ​ e ​ 9 = 62 2\times 31\mathrm{e}9=62 GFLOPs divided by 0.9 PFLOPS, giving 0.069 ms, and the local term is 838.9 MB divided by 5.6 TB/s, giving 0.150 ms, so attention is 14.63 / 14.85 = 98.53 14.63/14.85=98.53 % of the step.

Two features of the result matter for the argument of Section 8 . First, the split is architectural rather than incidental. The full-attention models sit at 94 to 99.37%, where the read per context token is tens or hundreds of kilobytes and the active-parameter matmul is one to three orders of magnitude cheaper. The indexer-based models sit at 56 to 97.50%, because their O(N) read is a few kilobytes or less and the matmul term is no longer negligible against it. The two groups overlap, since MiniMax-M3 at 97.50% sits above Kimi-K3 , and it does so because three of its 60 layers still run full attention. Attention still dominates in every row, and the models that have built a sparse mechanism of their own are the ones where it dominates least.

Second, the share is a strong function of context length, so the 1M figure is not a general statement about decode. Section 8 repeats the calculation at 244K, the effective input limit of our experiments, and at 128K. Both the global term and the share fall with N N while the other two terms hold constant. At 244K the full-attention models sit at 79.6 to 97.5% and the indexer-based ones at 24 to 90%, and by 128K three of the four indexer-based rows have fallen below 50%. The uniform 244K figures also sit above what Section C.8 reports for the same models, because the traces there average 40.5K attended tokens per step across 15 tasks whose contexts are mostly far shorter than the limit.

Sensitivity to the utilization choices is small next to that. Sweeping MFU over 40 to 60% and MBU over 60 to 85% moves the 1M share of every full-attention model by under 3 points. Among the indexer-based models it moves MiniMax-M3 by 0.9 points, GLM-5.3-Flash by 6.8, DeepSeek-V4-Flash by 8.3, and DeepSeek-V4-Pro by 14.4, which are the rows where the matmul term is largest relative to the global read. No model leaves its group’s range by enough to change either endpoint materially.

These figures inherit every limitation of Section C.8 . They are ceiling estimates on a saturated single accelerator, they charge weights as FLOPs rather than as bytes and so assume a batch large enough to have pushed the expert GEMMs into their compute-bound regime, and they omit prefill, normalization, rotary embeddings, residual adds, short-convolution states, and speculative-decoding layers.

## Appendix D Experimental details

### D.1 Evaluation datasets

The 15 context sources span diverse domains, context lengths, and QA formats. As summarized in Section 1 , 11 sources use the original QA samples from their respective benchmarks. For the remaining four, we generate synthetic QA with Gemini-3-Flash either to mitigate qualitative issues in the original QA annotations or to extend evaluation to additional context types under our task taxonomy. Together, these datasets cover extensive domains and tasks over contexts ranging from short documents to large code repositories. Each model’s effective input length is the native input limit (256K, or 128K for Gemma-4-E4B) minus 8K reserved for generation and 4K reserved for the DA prompt template, yielding 244K (or 116K for Gemma-4-E4B). After excluding examples longer than this limit, we draw up to 128 examples per source with a fixed seed ( Section 4 ); the available pool size of each source is listed as N N in Section 1 .

### D.2 Validation of local judge

To check our choice of Qwen-3.5-4B as judge against a frontier reference, we re-judged a stratified sample of 2,993 2{,}993 responses (15 long-context QA sources × \times 2 backbones × \times 3 arms: Vanilla, DA nm {}^{\text{nm}} , DA; capped at 36 responses per cell) with Gemini-3.1-Pro ( thinking_level=low , chosen to roughly match the local judge’s CoT depth) under the identical rubric, prompt, and parsing pipeline. Section 9 reports per-benchmark accuracies and aggregate agreement statistics; Section 7 plots per-cell accuracies. The two judges agree on 98.53 % 98.53\% of individual verdicts (95% CI [ 98.03 , 98.90 ] [98.03,\ 98.90] ; κ = 0.940 \kappa=0.940 , Krippendorff’s α = 0.940 \alpha=0.940 ), per-cell accuracies correlate at Pearson r = 0.992 r=0.992 (Spearman ρ = 0.970 \rho=0.970 ), and disagreements are symmetric (McNemar exact p = 0.65 p=0.65 , FP = 24 =24 vs. FN = 20 =20 ), supporting the use of Qwen-3.5-4B .

### D.3 Sampling parameters

We follow each model family’s officially recommended sampling parameters. For Qwen-3.5/6 in non-thinking mode: temperature 0.7 0.7 , top- p p 0.80 0.80 , top- k k 20 20 , presence penalty 1.5 1.5 . For Gemma-4: temperature 1.0 1.0 , top- p p 0.95 0.95 , top- k k 64 64 .

### D.4 Current limitations and future potential

DA’s savings come from prompting alone, with no training, fine-tuning, or task-specific machinery, so a minority of sources fall outside what zero-shot prompting covers. We restrict this analysis to Gemma-4-31B : on Qwen-3.6-27B , DA roughly doubles decode length on many of these sources, which would conflate decoder verbosity with protocol failure. Furthermore, we skip sources like ruler_qa_1 and ruler_qa_2 entirely, and note similar friction in ruler_cwe , fwe , and vt , because their hardcoded task prompts explicitly forbid intermediate reasoning by dictating “Answer the question based on the given documents. Only give me the answer and do not output any other words.”, which directly neutralizes the zero-shot scaffolding DA relies upon. The six exceptions ( Section 10 ) split into two structural mismatches with the <global> / <focus> / <local> decomposition, not weaknesses of the mechanism: DA still cuts per-step attention on all six ( − 39 -39 to − 67 % -67\% ).

1. Evidence destroyed by segmentation. The answer needs information that does not survive segmentation, such as a global count over all segments ( cwe ) or a table the segmenter splits ( structured_data ). Accuracy collapses ( − 30 -30 / − 21 -21 pp) even as per-step attention drops.

2. Output length grows with the document. The task emits one unit of output per segment or per item: word-frequency enumeration ( fwe ), per-segment summarization ( summ_screen_fd ), full-document ordering ( book_sum_sort ), or per-example in-context learning ( in_context_learning ). Decode steps scale with the document, so attended tokens accumulate end-to-end despite the per-step saving.

#### Closing the gap.

Category 1 is a decomposition failure, since <focus> sees only the named segments. It is addressable by structure-aware segmentation that keeps tables intact and a map-reduce pattern that sweeps <focus> across segments while accumulating in the persistent scaffold. Category 2 is a decode-length problem, not an accuracy one: the model writes its long output under <global> at full per-step attention, so routing verbose output through <focus> would keep per-step cost bounded across the decode. SFT or RLVR on these task shapes, and larger backbones (the gap already narrows from 4B to 31B), should teach both the decomposition and the mode-selection policy that zero-shot prompting cannot yet elicit.

## Appendix E Additional experimental results

The main-text Section 4 (context-length scaling) and Section 5 (mode efficiency) report Gemma-4-31B . Section 8 and Section 9 give the Qwen-3.6-27B counterparts.

Section 4(b) plots DA’s attended-token saving as an absolute delta; Section 10 shows the same data as a ratio to vanilla, which is roughly constant across context length.

## Appendix F Prompt construction

#### Context segmentation.

We split the context into segments with a tokenizer-aware semantic segmenter. The segmenter targets a 2048-token segment size with a 2560-token hard cap, and descends a delimiter hierarchy from coarse to fine: blank-line paragraph breaks, then single newlines, then sentence ends, then clause ends, and finally whitespace between words. It splits only a unit that exceeds the cap, and cuts at the coarsest boundary that unit contains, so a segment edge lands on the most semantically meaningful split the size budget allows. A whitespace-free run, such as a long base64 blob, is atomic: it is never split, and instead becomes its own over-cap segment rather than being cut mid-word.

The segmenter works entirely in character-offset space. It tokenizes the context once with offset mapping, measures candidate spans by token count through those offsets, and emits each segment as a real substring of the context, never by decoding token-id slices. A segment therefore can never be corrupted by a multibyte character split across a slice boundary, and the segments form an exact, lossless partition of the context: concatenating them reproduces the input. A context that fits under the cap is a single segment, and an empty or whitespace-only context is rendered as an <empty_context> placeholder so the protocol always has at least one addressable segment. Segments are numbered 1 1 to N N .

#### Magic-chunk delivery.

Each segment is delivered to the model as a magic chunk through a simulated tool-use conversation. The prompt opens with a short fixed system instruction, “You are a helpful assistant.”, whose role is to occupy the attention sink so context never enters it ( Section B ). A bootstrap user turn asks the assistant to retrieve the document one magic chunk at a time, and for each segment the conversation carries an assistant get_magic_chunk tool call followed by a tool response headed Magic Chunk N that contains the segment text. The get_magic_chunk tool is declared as retrieving arbitrary units that do not align with the document’s own sections or chapters, reinforcing the magic-chunk framing. The final user turn is the DA Instruction Prompt below.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .