A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets — part 1/3

Cleaned-text code points [0, 15653). Bibliography omitted here; source.md includes it.

# A Hippocampus for Linear Attention An Exact Memory for What the Recurrent State Forgets

###### Abstract

Linear-attention and state-space language models compress the prefix into a fixed-size recurrent state, yielding O ⁡ ( 1 ) O(1) memory at the cost of a lossy exact memory: when many key–value associations compete, earlier facts are overwritten and needle recall degrades. Inspired by Complementary Learning Systems, we give linear attention a hippocampal complement. HOLA (Hippocampal Linear Attention) keeps the usual delta-rule state as a compressive memory and adds a bounded exact KV cache, forming a semiparametric test-time memory: the state models linearly compressible structure, while the cache stores associations that should not be forced through that state. The cache writes without a learned eviction module, keeping tokens with large β ⋅ ∥ e ∥ \beta\!\cdot\!\lVert e\rVert , the prediction residual actually committed to the state; a decoupled RMSNorm- γ \gamma cache read then turns these exact KV pairs into sharp retrieval rather than soft averaging. At 340M parameters trained on 15B SlimPajama tokens, HOLA lowers Wikitext perplexity from 27.32 27.32 to 22.92 22.92 ( − 16.1 % -16.1\% ), below a full-attention Transformer++ ( 26.88 26.88 ), and improves LAMBADA perplexity from 30.95 30.95 to 30.26 30.26 . It also achieves the best linear in-context retrieval and remains much more robust than GDN or a matched HOLA+recency cache on RULER needle-in-a-haystack recall out to 32k tokens ( 16 × 16\times its training length).

## 1 Introduction

The human brain does not rely on a single memory. A hippocampus-centered system can record a specific, novel event in one shot and later recall it precisely ( Tse et al., 2007 ) , while a neocortex-centered system slowly distills generalizable structure across many experiences ( Frankland & Bontempi, 2005 ) ; Complementary Learning Systems (CLS) theory ( McClelland et al., 1995 ; Kumaran et al., 2016 ) argues that the two must be separate — a system built for slow, compressive generalization would suffer catastrophic interference if forced to write single facts quickly, and so cannot also serve fast, exact, one-shot memory. Efficient language models are missing exactly this complementarity.

Linear attention is precisely such a “neocortex”: a superb compressor, but a leaky exact memory. DeltaNet, Gated DeltaNet (GDN), and related linear-attention / state-space models ( Katharopoulos et al., 2020 ; Gu & Dao, 2023 ; Yang et al., 2024a ; Yang et al., 2024c ) replace softmax attention’s growing key–value cache with a fixed-size recurrent state S S , summarizing all history in O ⁡ ( 1 ) O(1) memory and processing a sequence in sub-quadratic time. The state is updated online — the delta rule writes the residual between each new value and the state’s current prediction — making it an efficient running summary. But a fixed-size state is a lossy memory: it holds only a bounded number of distinct key → \to value associations before new writes overwrite old ones along shared key directions. This shows up exactly where it hurts: multi-item associative recall ( Arora et al., 2023 ) , passkey / needle-in-a-haystack retrieval ( Mohtashami & Jaggi, 2023 ) , and copying a specific distant token verbatim ( Jelassi et al., 2024 ) . There, full softmax attention — which keeps every token exactly, at O ⁡ ( T ) O(T) memory and O ⁡ ( T 2 ) O(T^{2}) compute — remains the gold standard. The question becomes: can we keep linear attention’s cost while recovering its lost exact recall? CLS points to the answer — give this neocortex a hippocampus: a separate, exact, budget-limited episodic memory that preferentially stores surprise . We formalize this complement as semiparametric test-time memory regression : the recurrent state is the parametric estimator for linearly compressible structure, and a bounded exact cache is the non-parametric correction over KV pairs the state should not be forced to absorb. This framework dictates two design choices: what to store (the surprising items) and how to read it (exact retrieval, not soft averaging).

What to store: a surprise signal the model already computes. A budget-limited episodic memory is only useful if it keeps the right tokens — the surprising ones, which the compressed state could not absorb. The obvious choice is instead to keep the most recent tokens, a sliding window, as most linear-attention hybrids do ( Xiao et al., 2024 ; Zhang et al., 2023 ; Du et al., 2025 ; Wang et al., 2025 ; Fang et al., 2025 ) ; but recency cannot represent information that is far away yet still must be recalled exactly — once it slides out of the window it is as lost as if the state had overwritten it. Our key observation is that a delta-rule model already measures how surprising each token is to its state: if the state predicts the token’s value poorly and the model strongly commits the correction, that token is exactly the kind of support point the non-parametric cache should keep. We use this intrinsic write magnitude as the cache’s eviction score, retaining tokens that changed the state most rather than tokens that are merely recent. A matched control confirms the intended separation: at 340M, importance eviction beats a recency cache on perplexity and long-context retrieval, while commonsense remains within single-seed noise (Sec. 4 ).

How to read: retrieval, not averaging. An exact copy buys nothing if it is read like another linear summary. Directly reusing the L2-normalized queries and keys used by DeltaNet/GDN makes cache logits too small, so the cache softmax is nearly uniform and the exact memory degenerates into a lossy average. A simple fix is to use Qwen3-style RMSNorm- γ \gamma ( Qwen Team, 2025 ) on the cache path only, decoupled from the state update. This restores sharp, near-argmax retrieval while preserving the unit-norm keys required for the delta rule’s stable state update.

Together, these ideas give Hippocampal Linear Attention (HOLA) , which recovers the exact recall the state loses (Fig. 1 ). At 340M parameters trained on 15B SlimPajama tokens, the model reaches the lowest perplexity of every model we compare — below a full-attention Transformer++ (Wikitext 22.9 22.9 vs. 26.9 26.9 ) — and, unlike GDN, stays length-robust on hard RULER recall out to 32k tokens ( 16 × 16\times its 2k training length), where the recurrent state saturates and GDN decays.

#### Contributions.

1. A hippocampal exact memory for linear attention, motivated by CLS. We argue that a fixed recurrent state is a neocortex-like compressor but a leaky exact memory, and add a bounded, hippocampus-like KV store for precise one-shot associations.

2. A semiparametric test-time memory framework. We formalize memory readout as test-time regression over prefix key–value observations: pure GDN is the parametric estimator q ⊤ ​ S t q^{\!\top}S_{t} , full attention is an unbounded non-parametric kernel estimator, and HOLA is the bounded semiparametric case.

3. A concrete write/read algorithm. The cache writes by an intrinsic surprise signal already computed by the delta rule — the committed residual update to the state — and reads by a decoupled RMSNorm- γ \gamma cache path that makes exact KV pairs retrievable rather than softly averaged.

4. Large empirical gains in perplexity and retrieval. At 340M, HOLA sharply lowers Wikitext perplexity relative to the same-backbone GDN anchor ( → 22.92 27.32\!\to\!22.92 ), falls below a full-attention Transformer++ ( 26.88 26.88 ), achieves the best linear in-context retrieval, and remains much more robust than GDN or a recency cache on long-context needle recall.

## 2 Related Work

Our work connects three threads: linear-time recurrent attention, hybrid exact-memory mechanisms for efficient language models, and semiparametric memory.

#### Linear-time recurrent attention.

Linear attention ( Katharopoulos et al., 2020 ) , state-space models ( Gu & Dao, 2023 ; Dao & Gu, 2024 ) , and the DeltaNet family ( Schlag et al., 2021 ; Yang et al., 2024c ) replace softmax attention’s growing KV cache with a fixed-size recurrent state. Gated DeltaNet (GDN) ( Yang et al., 2024a ) strengthens DeltaNet with data-dependent decay, and GLA ( Yang et al., 2024b ) , GSA ( Zhang et al., 2024 ) , and Kimi Delta Attention (KDA) ( Kimi Team et al., 2025 ) are representative modern linear-attention baselines. This fixed state is efficient but lossy: as an associative memory, it has bounded capacity before new writes interfere with old ones ( Hopfield, 1982 ; Ramsauer et al., 2021 ) . Based and Zoology ( Arora et al., 2024 ; Arora et al., 2023 ) make this recall–throughput trade-off explicit, showing that efficient recurrent models lag softmax attention on multi-item associative recall. HOLA keeps the recurrent backbone, but adds a bounded exact memory for the KV pairs the state should not be forced to compress.

#### Hybrid exact memory for efficient LMs.

Many efficient LMs recover some exact recall by pairing a recurrent/SSM backbone with softmax attention. Inter-layer hybrids such as Jamba ( Lieber et al., 2024 ) , Samba ( Ren et al., 2024 ) , and Griffin ( De et al., 2024 ) mix recurrent layers with local attention layers; layer-internal hybrids such as NHA ( Du et al., 2025 ) , RAttention ( Wang et al., 2025 ) , and AHN ( Fang et al., 2025 ) combine recurrent memory with a recent-token window or learned compressed store. These designs differ architecturally, but their lossless component is primarily recency-based: once an old token leaves the window, it must be represented by the compressed state. HOLA instead makes the exact memory selective over the prefix, retaining surprising KV pairs even when they are far from the current query.

The closest work is LTE ( He & Garner, 2025 ) , which also augments GDN with a bounded evictable KV cache. The main difference is how the cache is selected and integrated: LTE learns eviction scores with an additional CNN module and alternates GDN with sparse-attention layers, whereas HOLA uses the delta rule’s own write magnitude as a parameter-free surprise score and attaches the cache inside every recurrent layer. Thus LTE shows that evictable caches can help delta-rule models, while HOLA argues that the recurrent update itself already exposes the right signal for what an exact memory should keep.

#### Semiparametric and retrieval-augmented memory.

Semiparametric language models pair parametric sequence modeling with explicit non-parametric memory, as in kNN-LM ( Khandelwal et al., 2020 ) , RETRO ( Borgeaud et al., 2022 ) , Memorizing Transformers ( Wu et al., 2022 ) , and SPALM ( Yogatama et al., 2021 ) . Those systems retrieve from an external or long-lived datastore. Our setting is in-sequence and test-time: the parametric part is the recurrent state S t S_{t} , the non-parametric part is a bounded set of exact in-context KV pairs, and the readout interpolates between the two. This semiparametric view also places softmax attention and GDN on the same spectrum: full attention is an unbounded non-parametric estimator over all prefix tokens, while GDN is the purely parametric state estimator.

## 3 Method

We derive HOLA from the delta-rule update equation: a single equation tells us both why a cache is necessary (Sec. 3.2 ) and what to store in it (Sec. 3.3 ); we then address how to read it (Sec. 3.4 ) and the full layer (Sec. 3.5 ). Figure 2 gives an overview.

### 3.1 Background: the delta rule and its lossiness

DeltaNet compresses history into a ( d k , d v ) (d_{k},d_{v}) matrix S S , an online associative memory : the read-out for key k k , k ⊤ ​ S k^{\!\top}S , gives its currently associated value. Let token t t have query/key/value q t , k t , v t q_{t},k_{t},v_{t} (with q t , k t q_{t},k_{t} L2-normalized to unit norm) and write strength β t ∈ [ 0 , 1 ] \beta_{t}\in[0,1] . The delta rule updates by “predict, then write the residual”: S t = S t − 1 + β t ​ k t ​ e t ⊤ , e t = v t − k t ⊤ ​ S t − 1 , o t state = q t ⊤ ​ S t . S_{t}\;=\;S_{t-1}+\beta_{t}\,k_{t}\,e_{t}^{\!\top},\qquad e_{t}\;=\;v_{t}-k_{t}^{\!\top}S_{t-1},\qquad o^{\mathrm{state}}_{t}\;=\;q_{t}^{\!\top}S_{t}. (1) Here e t e_{t} is the residual (innovation): the part of v t v_{t} that S S cannot already predict along k t k_{t} . Writing the residual (rather than v t v_{t} ) suppresses repeated same-key interference.

It is lossy. S S is fixed-size, of rank at most d k d_{k} : once the number of distinct associations exceeds capacity, new writes overwrite old ones along shared key directions, and a distant ( k , v ) (k,v) can no longer be recovered exactly from S S . This is the “neocortex-like” deficiency of Sec. 1 — good at generalizing, unable to keep distant exact detail. The next three subsections build HOLA on top; the specific backbone we use (GDN) is deferred to Sec. 3.5 .

### 3.2 Semiparametric memory as test-time regression

The previous subsection shows why a fixed recurrent state is lossy. We now give the framework that turns this observation into a design principle: memory readout is a test-time regression problem, and HOLA is a bounded semiparametric estimator for it.

Definition 1 (test-time memory regression). For a layer at position t t , let the causally available key–value observations be 𝒟 t = { ( k i , v i ) } i ≤ t \mathcal{D}_{t}=\{(k_{i},v_{i})\}_{i\leq t} . A test-time memory regression (TMR) mechanism is a memory state ℳ t \mathcal{M}_{t} together with two operations ℳ t = Write ⁡ ( ℳ t − 1 , k t , v t ) , o t = Read ⁡ ( q t , ℳ t ) = f ^ t ​ ( q t ) , \mathcal{M}_{t}=\mathrm{Write}(\mathcal{M}_{t-1},k_{t},v_{t}),\qquad o_{t}=\mathrm{Read}(q_{t},\mathcal{M}_{t})=\hat{f}_{t}(q_{t}), where f ^ t \hat{f}_{t} is an estimator, built from 𝒟 t \mathcal{D}_{t} , of the context-specific map f t : q ↦ v f_{t}:q\mapsto v . The choice of memory state ℳ t \mathcal{M}_{t} and estimator class f ^ t \hat{f}_{t} specifies the memory mechanism.

Definition 1 makes the comparison clean. GDN is a TMR mechanism whose memory state is only a fixed-size matrix ℳ t = S t \mathcal{M}_{t}=S_{t} and whose read operation is linear, o t = Read ⁡ ( q t , S t ) = f ^ state , t ​ ( q t ) = q t ⊤ ​ S t . o_{t}=\mathrm{Read}(q_{t},S_{t})=\hat{f}_{\mathrm{state},t}(q_{t})=q_{t}^{\!\top}S_{t}. (2) Here S t S_{t} is a test-time parameter: it is updated online from the context by the delta rule, not learned as an additional model weight. It is cheap and captures the linearly compressible part of the key–value map, but it cannot interpolate all observations in 𝒟 t \mathcal{D}_{t} once the context exceeds its capacity.

Definition 2 (semiparametric TMR). A TMR mechanism from Definition 1 is semiparametric if its memory decomposes as ℳ t = ( S t , 𝒜 t ) \mathcal{M}_{t}=(S_{t},\mathcal{A}_{t}) , where S t S_{t} is a fixed-size parametric state and 𝒜 t ⊆ 𝒟 t \mathcal{A}_{t}\subseteq\mathcal{D}_{t} is a set of exact KV pairs kept non-parametrically. Its Write operation updates both components, ( S t , 𝒜 t ) = Write ⁡ ( ( S t − 1 , 𝒜 t − 1 ) , k t , v t ) , (S_{t},\mathcal{A}_{t})=\mathrm{Write}((S_{t-1},\mathcal{A}_{t-1}),k_{t},v_{t}), where the state update changes the parametric estimator and the KV-set update may admit or evict exact KV pairs. Its Read operation returns o t = Read ⁡ ( q t , ℳ t ) = f ^ t ​ ( q t ) = q t ⊤ ​ S t + λ t ​ g t ​ ( q t ) . o_{t}=\mathrm{Read}(q_{t},\mathcal{M}_{t})=\hat{f}_{t}(q_{t})=q_{t}^{\!\top}S_{t}+\lambda_{t}\,g_{t}(q_{t}). (3) Here λ t \lambda_{t} is a read-side mixing coefficient, and g t g_{t} is a non-parametric model over the exact KV pairs in 𝒜 t \mathcal{A}_{t} .