{"id":"arxiv:2607.02303","title":"A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets","date":"2026-07-02","section":"LLMs: architectures, context, training","quality":0.24797,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/","source_url":"https://arxiv.org/abs/2607.02303","text_available":true,"authors":["Wanyun Cui"],"outgoing":[{"source":"arxiv:2607.02303","target":"arxiv:2006.16236","evidence":"id","id":"arxiv:2006.16236","title":"Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/d119844a2dce8a3b156558743b90add84774d00894634b6478d03abb4f5216f9/"},{"source":"arxiv:2607.02303","target":"arxiv:2203.08913","evidence":"id","id":"arxiv:2203.08913","title":"Memorizing Transformers","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/76deb2bc758a8a71e7c5c5ad6f4b45a1527845ec73784adc6f75f77462a596aa/"},{"source":"arxiv:2607.02303","target":"arxiv:2312.04927","evidence":"id","id":"arxiv:2312.04927","title":"Zoology: Measuring and Improving Recall in Efficient Language Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/baa792b06bdd30f91619f1e852aa5ae4dbf688fe3b1f2ee0722b339b9993ac5c/"}],"incoming":[],"text_parts":[{"start":0,"end":15653,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-1.html"},{"start":15653,"end":30831,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-2.html"},{"start":30831,"end":36557,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-3.html"}],"contents":[{"title":"A Hippocampus for Linear Attention An Exact Memory for What the Recurrent State Forgets","offset":0,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-1.html#text"},{"title":"Abstract","offset":91,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-1.html#text"},{"title":"1 Introduction","offset":1553,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-1.html#text"},{"title":"Contributions.","offset":6444,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-1.html#text"},{"title":"2 Related Work","offset":7682,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-1.html#text"},{"title":"Linear-time recurrent attention.","offset":7857,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-1.html#text"},{"title":"Hybrid exact memory for efficient LMs.","offset":8931,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-1.html#text"},{"title":"Semiparametric and retrieval-augmented memory.","offset":10385,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-1.html#text"},{"title":"3 Method","offset":11198,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-1.html#text"},{"title":"3.1 Background: the delta rule and its lossiness","offset":11473,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-1.html#text"},{"title":"3.2 Semiparametric memory as test-time regression","offset":12890,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-1.html#text"},{"title":"3.3 What to store: the write magnitude β ⋅ ∥ e ∥ \\beta\\!\\cdot\\!\\lVert e\\rVert is “surprise”","offset":16828,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-2.html#text"},{"title":"3.4 How to read: retrieval, not soft averaging","offset":20108,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-2.html#text"},{"title":"3.5 Instantiation","offset":22437,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-2.html#text"},{"title":"4 Experiments","offset":24009,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-2.html#text"},{"title":"4.1 Setup","offset":24027,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-2.html#text"},{"title":"4.2 Main results: language modeling, commonsense, retrieval","offset":25857,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-2.html#text"},{"title":"4.3 Consistency across scale","offset":27834,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-2.html#text"},{"title":"4.4 Long-Context Retrieval","offset":28255,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-2.html#text"},{"title":"4.5 Ablations","offset":29517,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-2.html#text"},{"title":"5 Conclusion and Limitations","offset":32401,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-3.html#text"},{"title":"Conclusion.","offset":32434,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-3.html#text"},{"title":"Limitations.","offset":33911,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-3.html#text"},{"title":"Appendix A Scale configurations","offset":34589,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-3.html#text"},{"title":"Instructions for reporting errors","offset":35593,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/part-3.html#text"}],"text_code_points":36557,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/499d4b7d5f795ec273c099c3ee7ba3ba4b6efb60063810c5afc4d10082a3168d/source.md"}