Language Models Can Control Their Own Attention — part 4/6

Cleaned-text code points [46015, 61444). Bibliography omitted here; source.md includes it.

Most prior work on reducing decode-time attention cost still pays a per-step cost to locate the tokens worth attending to. Early methods prune the KV cache by fixed heuristics such as recency and attention magnitude (StreamingLLM, Xiao et al., 2024b ; H2O, Zhang et al., 2023 ). Later methods concede that the important set shifts step to step and recover it dynamically: Quest ( Tang et al., 2024 ) scores KV pages by per-page key bounds, SparQ ( Ribar et al., 2024 ) and Double Sparsity ( Yang et al., 2024 ) read a low-rank or label-cache surrogate of every key, InfLLM ( Xiao et al., 2024a ) and ClusterKV ( Liu et al., 2025b ) scan block or cluster summaries, and TidalDecode ( Yang et al., 2025a ) amortizes the scan across layers. At industry scale, Native Sparse Attention ( Yuan et al., 2025 ) trains block-sparse selection into the model, and DeepSeek serves a learned lightweight indexer that scores and top- k k -selects tokens at every step (DeepSeek Sparse Attention in V3.2, DeepSeek-AI, 2025b , and its block-pooled successor in V4, DeepSeek-AI, 2026 ); SpotAttention ( Ahmad and Yun, 2026 ) distils the same indexer onto a frozen backbone, bringing learned selection to off-the-shelf models without joint pretraining. Within a year of V3.2 the indexer design has spread across open frontier models: GLM-5 adopts DSA by converting its full-attention layers during continued pretraining ( GLM-5 Team, 2026 ) , its successors amortize the indexer itself by sharing one across every four sparse layers (IndexShare) and pooling its keys (IndexPool) ( Z.ai, 2026a ; Z.ai, 2026b ) , Qwen3.8-Flash-Next replaces its remaining full-attention layers with block-level sparse selection (QSA) ( Qwen Team, 2026a ) , and MiniMax-M3 selects top- k k KV blocks through a dedicated index branch (MSA) ( Lai et al., 2026 ) . Across this line the selection is computed by scanning a cheaper surrogate of the whole context each step, leaving the per-step cost at O ⁡ ( N ) O(N) with a smaller constant, or baking it into the weights and a custom kernel by retraining.

DA obtains the same per-step selection without the scan. Just as chain-of-thought exposes the model’s reasoning as interpretable text ( Wei et al., 2022 ) , DA exposes the model’s intended attention scope as parseable tags, so the engine reads where to mask directly from the generated stream rather than approximating it from a per-step index. Resolving a <focus> tag is O ⁡ ( 1 ) O(1) , in place of the lightweight but still O ⁡ ( N ) O(N) index that a DSA-style method runs every step. The realized saving then turns on contiguity: SparQ reaches only 1.3 1.3 to 3.0 × 3.0\times of its 6.4 × 6.4\times theoretical speedup because its selected reads are scattered ( Ribar et al., 2024 ) , and Double Sparsity finds that a contiguous layout alone accounts for most of the available speedup ( Yang et al., 2024 ) , the property DA’s block-aligned block-table compaction is built to preserve.

DA’s one residual cost falls in global mode, where surveying the full context still requires full O ⁡ ( N ) O(N) attention, a cost that is orthogonal to the protocol and that the methods above can absorb. We discuss this complementarity in Section 8.3 .

### A.2 KV-cache eviction

A parallel line cuts attention cost by permanently dropping KV entries rather than masking them. Scissorhands ( Liu et al., 2023 ) , H2O ( Zhang et al., 2023 ) , SnapKV ( Li et al., 2024b ) , and FastGen ( Ge et al., 2024 ) evict the tokens judged unimportant by their accumulated or windowed attention mass, freeing cache capacity at the risk that a later step needs a token already discarded. DA instead masks and never evicts, keeping the full cache resident so that every per-step mask is reversible and a <focus> step can be followed by a <global> step that re-attends everything, which is why DA preserves accuracy where eviction must bound an unrecoverable error. The naming in this line invites a confusion worth dispelling: SnapKV (“LLM Knows What You are Looking for”) and FastGen (“Model Tells You What to Discard”) read as if the model declares what to keep, yet both decide from observed attention statistics rather than from anything the model writes. DA is the literal form of that metaphor, with the model stating its attention scope in text and the engine acting on the statement.

### A.3 Complementary KV-cache axes

DA reduces how many KV positions are read per step, orthogonal to the two other axes along which serving systems shrink attention cost. The first reduces the bytes per position: grouped- and multi-query attention ( Shazeer, 2019 ; Ainslie et al., 2023 ) , multi-head latent attention ( DeepSeek-AI, 2024a ) , and low-bit KV quantization such as KIVI ( Liu et al., 2024 ) shrink each cached entry without changing how many are read. The second relocates the cache: FlexGen ( Sheng et al., 2023 ) , InfiniGen ( Lee et al., 2024 ) , and RetrievalAttention ( Liu et al., 2025a ) offload the KV cache to host memory and fetch a working set per step. Both axes compose with DA, which operates on a resident, full-precision cache and cuts the position count. The offloading methods also sharpen DA’s selection story, since InfiniGen speculates its fetch set from a partial query-key product and RetrievalAttention runs an approximate nearest-neighbor search, whereas DA reads the set from declared text, a third category distinct from both fixed heuristics and learned or searched predictors.

### A.4 Model-controlled inference and self-generated control tokens

DA extends a broader pattern of model-controlled inference, in which a model emits control tokens that a runtime parses and acts on, from steering what the model writes to steering what it reads. Existing instances have the model write actions, reflection tokens, or memory calls that trigger retrieval, grounding, or paging (ReAct, Yao et al., 2023 ; Self-RAG, Asai et al., 2024 ; MemGPT, Packer et al., 2023 ), or emit tokens that route or add computation (Mixture-of-Depths, Raposo et al., 2024 ; Pause tokens, Goyal et al., 2024 ; Quiet-STaR, Zelikman et al., 2024 ). DA applies the same surface to attention, verbally controlling its own internal computation by declaring through its chain-of-thought the mask the engine applies to the KV cache. What separates it from the rest of the line is what the declaration physically changes: the others edit the token sequence or change how much computation a token receives, whereas DA changes nothing in the sequence and only changes which resident KV pages the kernel reads, reversibly and at no re-prefill cost. MemGPT is the closest neighbor and the cleanest contrast, managing context by eviction and recall where a recall costs a round trip, while DA manages it by a reversible per-step mask. Quiet-STaR is the closest case of generated tokens driving an attention mask, though its mask is a training-time, content-independent device that keeps parallel thought branches from contaminating one another, not an inference-time, content-addressed scoping of the KV cache.

### A.5 Self-declared and steered attention

Self-Selected Attention Span (SSAS) ( Jin et al., 2024 ) shares DA’s central premise, that a model can select its own attention span, and demonstrates it in a task-specific training setting. SSAS fine-tunes a model to name, in its own output, the context spans it needs, and a custom kernel turns these references into a hard block-sparse attention mask. As in DA, masked spans stay in memory and can be selected again later, and the selections are visible as plain text in the output. The behavior is trained in per task, on hand-designed context partitions and annotation syntax, and shown on two tasks with contexts up to 2K tokens on 7B models. DA elicits the behavior zero-shot, with one fixed prompt and one segmentation scheme that apply unchanged across 15 tasks, two model families, and contexts up to 244K tokens.

Other work also lets a model or a user steer where attention goes, and each such method departs from DA on one axis: it reweights instead of masking, learns a gate instead of reading text, or evicts instead of scoping reversibly. PASTA ( Zhang et al., 2024a ) steers attention toward user-highlighted spans, and AutoPASTA ( Zhang et al., 2024b ) has the model itself surface the sentence to emphasize, matching DA’s premise that a model can declare its own focus, yet both only reweight attention scores over a fully loaded cache and aim at faithfulness, so neither reduces KV reads, whereas DA applies a hard mask that drops pages from the kernel’s read set and aims at decode bandwidth. Attention-Gate ( Zeng et al., 2024 ) masks a model’s own KV per token, but from a learned, fine-tuned gate over hidden states rather than generated text, and it evicts permanently rather than masking reversibly. ThinKV ( Ramachandran et al., 2025 ) is likewise thought-adaptive yet detects reasoning phases from latent attention-sparsity patterns, explicitly arguing that text- or keyword-based detection is brittle, then quantizes and evicts, where DA takes the opposite position and elicits structured tags by design so the signal is read from text rather than inferred from activations. ThinKV also edits the paged-attention block table, but to reuse evicted slots rather than to skip reads, the same data structure put to the opposite purpose.

## Appendix B DA vLLM integration

DA’s state machine extends vLLM through hooks on the attention metadata builder, requiring no modifications to vLLM’s kernels or scheduler. This section details the regions the mask keeps attended in every mode, the per-kernel patches for different attention backends, the layers the mask applies to, its block alignment, and its effect on decode latency.

#### Always-attended regions.

Three regions stay attended in every mode, realizing the scaffold-and-response guarantee of Section 2.3 . The first is a fixed attention sink of the prompt’s first 16 tokens, in the style of StreamingLLM ( Xiao et al., 2024b ) . We place a short, fixed system instruction at the start of the prompt so these 16 tokens always land on content-free scaffolding, never on context. The second is a local window covering the question and the DA instruction through the end of the prompt, so the model never loses sight of what it is answering. The third is the entire response generated so far. <focus> keeps these three regions plus the named segment spans, and <local> keeps only the three.

#### Per-kernel patches.

vLLM routes different models to different attention kernels based on their architecture. Qwen-3.6’s global attention layers route to FlashAttention ( Dao et al., 2022 ) , while Gemma-4’s hybrid attention design routes to a Triton-based paged-attention kernel. For each kernel, the DA state machine writes the mask into the request’s KV-cache block table at every decode step, so that only the kept blocks remain visible to the kernel. The block-table modification is implemented as a hook on the attention metadata builder, analogous across both backends and requiring no changes to the kernel internals.

#### Scope: global attention layers only.

DA’s mask construction targets only the global attention layers of each model. For Qwen-3.5/6, the remaining layers use Gated DeltaNet (GDN) ( Yang et al., 2025b ) , a linear-attention variant whose per-step state is bounded by a recurrent hidden representation rather than a context-length-dependent KV cache, so per-token masking does not apply. For Gemma-4, the remaining layers use sliding window attention (SWA) ( Beltagy et al., 2020 ) with a 1,024-token window, which already bounds per-step KV reads independently of context length. DA leaves both kinds of layers unmodified.

#### Block alignment.

The per-token mask is rounded outward to vLLM’s KV-cache block boundaries. This adds at most b − 1 b-1 extra attended tokens at each edge of a kept span, where b b is the block size (typically 16 to 32 tokens), negligible against the 2048-token segments. Block alignment is what allows existing kernels to run unchanged on the masked KV cache.

#### Effect on decode latency.

By reducing the number of KV blocks loaded per step, the integration cuts the per-step attention read to the kept blocks. The wall-clock effect is largest in the regime where attention dominates total decode time, which we estimate with the roofline analysis of Section 5.4 .

## Appendix C Background on inference costs and derivation of roofline wall-time

### C.1 Prefill vs decode

Modern LLM serving operates in two phases ( Pope et al., 2023 ; Zhong et al., 2024 ) . Prefill processes the input prompt in a single forward pass that computes attention and FFN outputs for all input tokens in parallel. Decode then generates output tokens autoregressively, one per forward pass, with each step reading the entire KV cache accumulated so far. DA operates at decode time, where the model’s chain-of-thought is generated and the attention scope can be dynamically reshaped.

### C.2 Decode cost components

Each decode step costs two operations: an FFN forward pass and an attention forward pass. The two have different cost structures, which we examine in turn.

#### FFN.

At each decode step the FFN must load model parameters from HBM and perform matrix multiplications between the current query vector and the loaded weights. Parameters are read from HBM once per forward pass and reused across every sequence in the batch, so the per-sequence amortized memory cost shrinks as batch size grows. Above a model-specific critical batch size , the FFN crosses from memory-bound to compute-bound: the time spent on FLOPs exceeds the time spent loading weights, and the matmul utilization becomes the binding constraint ( Agrawal et al., 2024 ) . For Mixture-of-Experts (MoE) layers ( Lepikhin et al., 2021 ; Fedus et al., 2022 ) , only the active experts are loaded per step, but the amortization works analogously across the sequences routed to each expert.

#### Attention.

At each decode step the attention kernel must load each sequence’s KV cache from HBM and compute scaled dot products between the current query and all stored keys, then a weighted sum over values. Unlike FFN, this cost does not amortize across the batch: each sequence has its own KV cache, and the kernel must load each one independently at every step. KV compression methods such as GQA ( Ainslie et al., 2023 ) , MQA ( Shazeer, 2019 ) , and MLA ( DeepSeek-AI, 2024a ) reduce the per-sequence read volume but do not change the per-sequence structure of the cost. On modern accelerators, the KV bandwidth requirement saturates HBM before the corresponding compute saturates the FLOPs, making attention memory-bound ( Dao et al., 2022 ) .

### C.3 The roofline at large batch

Each operation’s time on hardware is approximated by the roofline model ( Williams et al., 2009 ) , which gives T = max ⁡ ( T math , T mem ) T=\max(T_{\text{math}},T_{\text{mem}}) where T math = FLOPs / Peak FLOPS T_{\text{math}}=\text{FLOPs}\,/\,\text{Peak FLOPS} and T mem = bytes / Peak BW T_{\text{mem}}=\text{bytes}\,/\,\text{Peak BW} . At large batch in optimized serving systems ( Yu et al., 2022 ; Kwon et al., 2023 ) , FFN sits firmly in the compute-bound regime where T math T_{\text{math}} dominates, while attention sits in the memory-bound regime where T mem T_{\text{mem}} dominates.

### C.4 Roofline wall-time definition