A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

arxiv:2607.02303

Original source · Paper JSON

LLMs: architectures, context, training · 2026-07-02

Cites (3) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• A Hippocampus for Linear Attention An Exact Memory for What the Recurrent State Forgets
• Abstract
• 1 Introduction
• Contributions.
• 2 Related Work
• Linear-time recurrent attention.
• Hybrid exact memory for efficient LMs.
• Semiparametric and retrieval-augmented memory.
• 3 Method
• 3.1 Background: the delta rule and its lossiness
• 3.2 Semiparametric memory as test-time regression
• 3.3 What to store: the write magnitude β ⋅ ∥ e ∥ \beta\!\cdot\!\lVert e\rVert is “surprise”
• 3.4 How to read: retrieval, not soft averaging
• 3.5 Instantiation
• 4 Experiments
• 4.1 Setup
• 4.2 Main results: language modeling, commonsense, retrieval
• 4.3 Consistency across scale
• 4.4 Long-Context Retrieval
• 4.5 Ablations
• 5 Conclusion and Limitations
• Conclusion.
• Limitations.
• Appendix A Scale configurations
• Instructions for reporting errors