Language Models Can Control Their Own Attention

arxiv:2609.02737

Original source · Paper JSON

LLMs: architectures, context, training · 2026-09-02

Cites (1) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• Language Models Can Control Their Own Attention
• Abstract
• 1 Introduction
• Contribution.
• 2 Declarative Attention
• 2.1 Prompt overview
• 2.2 Context delivery
• Context segmentation.
• Segment formatting.
• 2.3 Decode-time interventions
• Parsing and mode transitions.
• Block-aligned mask construction.
• Custom integration with vLLM.
• Scope and tradeoffs.
• 3 When does DA pay off?
• Overview.
• FFN roofline wall-time.
• Attention roofline wall-time.
• Attention dominates FFN at scale.
• 4 Experimental settings
• Models.
• Evaluation datasets.
• Baselines.
• Evaluation.
• Metrics.
• Inference settings.
• 5 Results
• 5.1 Main results
• Accuracy drops modestly on average.
• Attention cost falls below vanilla.
• The attention mask is the source of the efficiency gain.
• 5.2 Model capability scaling
• Stronger models close the accuracy gap.
• Token savings are roughly scale-independent.
• 5.3 Context-length scaling
• Accuracy holds while token saving grows with context.
• 5.4 Efficiency results
• Theoretical wall-clock time on optimized inference setups.
• 6 Analysis
• 6.1 Efficiency impact of DA modes
• 6.2 Protocol adherence
• 7 Conclusion
• 8 Discussion, Limitations, and Future Work
• 8.1 Relevance of DA in the present and future
• Relevance of global attention costs in current and future architectures.
• Relevance of DA in agentic systems.
• 8.2 Further potential of DA with post-training and in agentic settings
• Zero-shot DA already demonstrates significant attention savings and positive scaling.
• 8.3 Synergy with modern efficient inference techniques
• Lightweight scan sparse attention methods can alleviate costs in global mode.
• Synergy with speculative decoding.
• 8.4 DA as system-2 sparse attention
• Toward system-2 KV cache offloading.
• Acknowledgements
• Appendix A Related works
• A.1 Dynamic sparse attention at decode
• A.2 KV-cache eviction
• A.3 Complementary KV-cache axes
• A.4 Model-controlled inference and self-generated control tokens
• A.5 Self-declared and steered attention
• Appendix B DA vLLM integration
• Always-attended regions.
• Per-kernel patches.
• Scope: global attention layers only.
• Block alignment.
• Effect on decode latency.
• Appendix C Background on inference costs and derivation of roofline wall-time
• C.1 Prefill vs decode
• C.2 Decode cost components
• FFN.
• Attention.
• C.3 The roofline at large batch
• C.4 Roofline wall-time definition
• C.5 Representative utilization values
• MFU.
• MBU.
• C.6 The operating regime: production disaggregation at scale
• C.7 Implication: attention dominates at scale
• C.8 Worked decode wall-time decomposition for the headline models
• Matmul (compute-bound).
• Global memory (memory-bound).
• Local memory (memory-bound).
• C.9 KV bytes read per context token across current architectures
• C.10 Attention share of decode wall-time at 1M-token contexts
• Appendix D Experimental details
• D.1 Evaluation datasets
• D.2 Validation of local judge
• D.3 Sampling parameters
• D.4 Current limitations and future potential
• Closing the gap.
• Appendix E Additional experimental results
• Appendix F Prompt construction
• Context segmentation.
• Magic-chunk delivery.
• Instructions for reporting errors