A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets β€” part 2/3

Cleaned-text code points [15653, 30831). Bibliography omitted here; source.md includes it.

This form also locates full softmax attention. It is the non-parametric TMR instance obtained by disabling the state term and using all causally available observations as exact KV pairs, i.e., set π’œ t = π’Ÿ t \mathcal{A}_{t}=\mathcal{D}_{t} . In that case the non-parametric model g t g_{t} is exactly the softmax kernel estimator, g attn , t ​ ( q ) = βˆ‘ ( k i , v i ) ∈ π’Ÿ t exp ⁑ ( q ⊀ ​ k i / d ) βˆ‘ ( k j , v j ) ∈ π’Ÿ t exp ⁑ ( q ⊀ ​ k j / d ) ​ v i , g_{\mathrm{attn},t}(q)=\sum_{(k_{i},v_{i})\in\mathcal{D}_{t}}\frac{\exp(q^{\!\top}k_{i}/\sqrt{d})}{\sum_{(k_{j},v_{j})\in\mathcal{D}_{t}}\exp(q^{\!\top}k_{j}/\sqrt{d})}v_{i}, (4) i.e. a Nadaraya–Watson kernel estimator ( Nadaraya, 1964 ; Watson, 1964 ) . It can recover exact items because every token remains an exact KV pair, but the KV set grows with context length.

HOLA is the bounded semiparametric TMR instance. We take S t S_{t} to be the full GDN recurrent state and keep a bounded set of exact KV pairs selected from π’Ÿ t \mathcal{D}_{t} by the score in Sec. 3.3 . Under this framework, HOLA has two design questions: which exact KV pairs to keep (Sec. 3.3 ), and how sharp the kernel read should be (Sec. 3.4 ).

### 3.3 What to store: the write magnitude Ξ² β‹… βˆ₯ e βˆ₯ \beta\!\cdot\!\lVert e\rVert is β€œsurprise”

We should store the tokens that are least well represented by the state memory. The same update equation already says which . Write the update as S t = S t βˆ’ 1 + Ξ” t , Ξ” t = Ξ² t k t e t ⊀ ( rank-1 ) . S_{t}=S_{t-1}+\Delta_{t},\qquad\Delta_{t}=\beta_{t}\,k_{t}e_{t}^{\!\top}\quad(\text{rank-1}). (5) The token’s entire effect on S S is this one rank-1 matrix Ξ” t \Delta_{t} . How much a token β€œwrites” is naturally the size of Ξ” t \Delta_{t} ; for a rank-1 matrix the Frobenius norm factorizes, and with βˆ₯ k t βˆ₯ = 1 \lVert k_{t}\rVert=1 collapses to a scalar: m t = βˆ₯ Ξ” t βˆ₯ F = Ξ² t ​ βˆ₯ k t βˆ₯ ​ βˆ₯ e t βˆ₯ = Ξ² t ​ βˆ₯ e t βˆ₯ . m_{t}\;=\;\lVert\Delta_{t}\rVert_{F}\;=\;\beta_{t}\,\lVert k_{t}\rVert\,\lVert e_{t}\rVert\;=\;\beta_{t}\,\lVert e_{t}\rVert. (6) We write this as Ξ² β‹… βˆ₯ e βˆ₯ \beta\!\cdot\!\lVert e\rVert and use it directly as the eviction score. Its meaning is immediate: m t m_{t} is how much the token changed S S . A large change means the token brought information S S could not predict (large residual e t e_{t} , i.e. the innovation in the Kalman ( Kalman, 1960 ) sense; the delta rule is the Widrow–Hoff/LMS rule ( Widrow & Hoff, 1960 ) , where e e is its error term) and that the model actually wrote it in (large Ξ² t \beta_{t} ). A token with small residual is already well predicted by the state, so storing a verbatim copy is wasteful; a token with large committed residual is exactly where the compressed state needed the most help. In short, we spend the limited exact memory where the state-memory representation is weakest.

Cache = = the top- w w tokens seen so far by Ξ² β‹… βˆ₯ e βˆ₯ \beta\!\cdot\!\lVert e\rVert . Each layer keeps a persistent exact cache of capacity w w (default w = 64 w{=}64 ). Its members are the exact ( k , v ) (k,v) copies of the w w highest- m m tokens in the causal history observed so far – keep what wrote most to S S , regardless of distance . Because m t m_{t} is fixed when the token is written, the same top- w w set can be maintained online or selected blockwise without order dependence; training and inference therefore use the same cache semantics.

Versus recency. Sliding-window memories, as used in StreamingLLM ( Xiao et al., 2024 ) , NHA ( Du et al., 2025 ) , and RAttention ( Wang et al., 2025 ) , choose exact KV pairs by position. This is useful for local context, but it cannot keep an old item solely because it remains important. Selection by Ξ² β‹… βˆ₯ e βˆ₯ \beta\!\cdot\!\lVert e\rVert instead keeps surprising KV pairs across distance, compensating the state-memory failure described in Sec. 3.1 . An ablation (Sec. 4 ) shows why the product matters: the residual alone ( βˆ₯ e βˆ₯ \lVert e\rVert ) or the write strength alone ( Ξ² ​ βˆ₯ v βˆ₯ \beta\lVert v\rVert ) each underperform; their product is best.

For the read in Sec. 3.4 , we denote the bounded visible KV set by 𝒱 t \mathcal{V}_{t} . Its main persistent component is the top- w w exact cache above; in implementation we also include the causal tokens in the current processing block and one null sink. These additions are bounded bookkeeping for causal block processing and stability; the persistent exact memory is selected by Ξ² β‹… βˆ₯ e βˆ₯ \beta\!\cdot\!\lVert e\rVert .

### 3.4 How to read: retrieval, not soft averaging

An exact copy is worthless if read like a linear attention. If the cache reuses the backbone’s unit-L2-normalized q , k q,k ( Yang et al., 2024c ; Yang et al., 2024a ) , the effective logit is Ο„ β‹… ( 1 / d ) cos β‰ˆ 0.83 cos \tau\cdot(1/\sqrt{d})\cos\approx 0.83\cos (a learned Ο„ β‰ˆ 6.6 \tau\!\approx\!6.6 , head dimension d d ), ranging only over Β± 0.83 \pm 0.83 : the softmax is nearly uniform β€” among w = 64 w{=}64 entries a perfectly matching key receives only ∼ 3.5 % {\sim}3.5\% of the mass. The cache degenerates into yet another soft-averaging lossy summary (explaining why a naive cache barely helps).

Sharpening via a decoupled RMSNorm- Ξ³ \gamma . We apply Qwen3-style RMSNorm with a learnable Ξ³ \gamma to the cache-path q , k q,k (keeping the norm at d β‰ˆ 11 \sqrt{d}\!\approx\!11 rather than 1 1 ) and fix Ο„ = 1 \tau{=}1 . The cache read is then a sharpened softmax attention over 𝒱 t \mathcal{V}_{t} : o t cache = βˆ‘ j ∈ 𝒱 t softmax j ​ ( q ~ t ⊀ ​ k ~ j / d ) ​ v j , q ~ = RMSNorm Ξ³ ​ ( q ) , k ~ = RMSNorm Ξ³ ​ ( k ) . o^{\mathrm{cache}}_{t}\;=\;\sum_{j\in\mathcal{V}_{t}}\mathrm{softmax}_{j}\!\Big(\tilde{q}_{t}^{\!\top}\tilde{k}_{j}/\sqrt{d}\Big)\,v_{j},\qquad\tilde{q}=\mathrm{RMSNorm}_{\gamma}(q),\ \ \tilde{k}=\mathrm{RMSNorm}_{\gamma}(k). (7) Since βˆ₯ q ~ βˆ₯ β‰ˆ βˆ₯ k ~ βˆ₯ β‰ˆ d \lVert\tilde{q}\rVert\!\approx\!\lVert\tilde{k}\rVert\!\approx\!\sqrt{d} , the effective logit is β‰ˆ d ​ cos β‰ˆ 11 ​ cos {\approx}\sqrt{d}\cos\approx 11\cos (vs. 0.83 ​ cos 0.83\cos for unit-L2), so the cache finally performs near-argmax retrieval . This change acts only on the cache read and is decoupled from the state-update path β€” the q , k q,k feeding S S remain unit-L2-normalized. Decoupling is necessary: the delta rule relies on βˆ₯ k βˆ₯ = 1 \lVert k\rVert=1 to keep the update operator I βˆ’ Ξ² ​ k ​ k ⊀ I-\beta\,kk^{\!\top} within [ 0 , 1 ] [0,1] eigenvalues (stable); a d \sqrt{d} norm in the state update would give eigenvalues 1 βˆ’ Ξ² ​ d 1-\beta d and diverge. The learnable Ξ³ \gamma self-moderates sharpness: versus a fixed high temperature it reaches lower perplexity while avoiding β€œthe state grows lazy and far recall collapses” (Sec. 4 ). Empirically, sharpening is the single largest lever in the design (perplexity β†’ 60 70\!\to\!60 , ∼ 2 Γ— {\sim}2\times multi-key capacity).

### 3.5 Instantiation

Instantiation (GDN). The derivation used only the delta rule. We instantiate on the strongest linear backbone, Gated DeltaNet, which adds a data-dependent decay gate Ξ± t ∈ ( 0 , 1 ] \alpha_{t}\in(0,1] (selective forgetting), making the prediction Ξ± t ​ k ⊀ ​ S t βˆ’ 1 \alpha_{t}k^{\!\top}S_{t-1} and the residual e t = v t βˆ’ Ξ± t ​ k ⊀ ​ S t βˆ’ 1 e_{t}=v_{t}-\alpha_{t}k^{\!\top}S_{t-1} . The gate is orthogonal to our method and Ξ² β‹… βˆ₯ e βˆ₯ \beta\!\cdot\!\lVert e\rVert is unchanged; all experiments use GDN.

Overhead over GDN. HOLA is almost iso-parametric with its GDN backbone. At 340M ( L = 24 L{=}24 , H = 4 H{=}4 , d = 256 d{=}256 ), the only learned cache-specific parameters are the cache-path Q/K RMSNorm scales plus a per-head sink and cache gate: L ⁑ ( 2 ​ d + 2 ​ H ) = 24 ​ ( 512 + 8 ) = 12,480 L(2d+2H)=24(512+8)=12{,}480 trainable scalars, less than 0.004 % 0.004\% of the full model (the frozen temperature adds only L ​ H = 96 LH{=}96 stored scalars). The cache itself is inference state rather than model weights: in bf16 decoding it stores at most ( w + C ) (w{+}C) K/V pairs per layer, about 24 β‹… 320 β‹… 4 β‹… 256 β‹… 2 β‹… 2 β‰ˆ 31 24\cdot 320\cdot 4\cdot 256\cdot 2\cdot 2\approx 31 MB plus negligible scores. Measured peak GPU allocation (weights included, bs = 1 =1 decode) is therefore close to GDN and flat with context: 0.75 0.75 GB for HOLA versus 0.72 0.72 GB for GDN at both 32k and 128k. Thus the gains do not come from a larger parametric model, and the exact-memory overhead over GDN is small ( β‰ˆ 5 % \approx 5\% peak memory).

## 4 Experiments

### 4.1 Setup

Architecture. We use the GDN architecture at 340M ( Yang et al., 2024a ) : d model = 1024 d_{\mathrm{model}}{=}1024 , 24 layers, 4 4 heads Γ— \times head-dim 256 256 , expand ​ _ ​ v = 1 \mathrm{expand\_v}{=}1 , hidden ​ _ ​ ratio = 4 \mathrm{hidden\_ratio}{=}4 , conv 4, tied embeddings, and vocabulary 32000 32000 . HOLA adds a per-layer cache: window w = 64 w{=}64 , chunk C = 256 C{=}256 , eviction score Ξ² β‹… βˆ₯ e βˆ₯ \beta\!\cdot\!\lVert e\rVert , cache normalization RMSNorm- Ξ³ \gamma , Ο„ = 1 \tau{=}1 frozen, gate init βˆ’ 4 -4 . The GDN baseline and HOLA share an identical backbone ; the only difference is the cache, so internal comparisons are strictly controlled.

Training recipe. We follow the Preconditioned-DeltaNet 340M recipe ( Tumma et al., 2026 ) : SlimPajama 15.0B tokens ( Soboleva et al., 2023 ) , Mistral tokenizer ( Jiang et al., 2023 ) , context 2048; AdamW ( Loshchilov & Hutter, 2019 ) (peak lr 4 Γ— 10 βˆ’ 4 4{\times}10^{-4} , wd 0.01 0.01 , cosine, warmup 1000, grad-clip 1.0), batch 0.5M tokens, 1 epoch. This recipe match enables reuse of the published DeltaNet/KDA/GDN baseline rows. Trained on 8 Γ— 8\times A800.

Baselines. We train our own GDN anchor and HOLA for the controlled same-backbone comparison, and borrow recipe-matched architecture rows for Transformer++, GLA, and GSA from Du et al. (2025) , and DeltaNet, KDA, and GDN from Tumma et al. (2026) ; KDA itself is from Kimi Linear ( Kimi Team et al., 2025 ) .

Evaluation. (1) Language modeling: Wikitext-103 perplexity, LAMBADA. (2) Commonsense (zero-shot): ARC-e/c, PIQA, HellaSwag, WinoGrande, BoolQ, SciQ, OpenBookQA, LAMBADA-acc. (3) In-context retrieval: FDA, SWDE, SQuAD. (4) Long context: RULER ( Hsieh et al., 2024 ) (2k β†’ \to 32k; multi-key MK, multi-value MV, multi-query MQ, variable tracking VT) and passkey/needle.

### 4.2 Main results: language modeling, commonsense, retrieval

Table 1 reports the main comparison. The effect is large on the two axes the cache is designed to improve: HOLA sharply lowers perplexity and substantially boosts in-context retrieval . Wikitext perplexity drops from β†’ 22.92 27.32\!\to\!22.92 relative to our same-backbone GDN anchor ( βˆ’ 16.1 % {-}16.1\% ), and even falls below the full-attention Transformer++ ( 26.88 26.88 ). Retrieval improves just as strongly: FDA rises β†’ 20.1 11.7\!\to\!20.1 ( + 72 % +72\% relative) and SWDE β†’ 35.9 29.0\!\to\!35.9 ( + 24 % +24\% ), the best among linear models, while commonsense remains competitive on the six-task average.

β€’ Perplexity: HOLA’s Wiki 22.92 22.92 is a large drop from the same-backbone GDN anchor ( β†’ 22.92 27.32\!\to\!22.92 , βˆ’ 16.1 % {-}16.1\% ), below the strongest published sub-quadratic baseline (KDA 26.18 26.18 ), and even below the full-attention Transformer++ ( 26.88 26.88 ); LAMBADA perplexity is also the lowest in the table ( 30.26 30.26 ).

β€’ Commonsense: the six-task average is effectively a tie among the strong models β€” HOLA 42.85 42.85 , the HOLA+recency control 43.17 43.17 , KDA 42.75 42.75 , and GDN ∼ 42.7 {\sim}42.7 all sit within ∼ 0.6 {\sim}0.6 (above Transformer++ 42.34 42.34 ); commonsense is not where the cache mechanism separates the models (perplexity and retrieval are). Over the 9-task accuracy mean, HOLA 0.446 0.446 vs. GDN 0.440 0.440 (BoolQ β†’ 0.584 0.548\!\to\!0.584 , + 3.5 {+}3.5 pt; WinoGrande, LAMBADA-acc, and HellaSwag also higher).

β€’ Retrieval: on in-context exact extraction HOLA beats all linear baselines by a wide margin β€” FDA β†’ 20.1 11.7\!\to\!20.1 ( + 72 % +72\% rel.), SWDE β†’ 35.9 29.0\!\to\!35.9 ( + 24 % +24\% ), SQuAD β†’ 33.8 32.5\!\to\!33.8 . This is the cache’s intended regime: preserving exact tokens that the recurrent state would otherwise compress away. Only the full-attention Transformer++ (FDA 46.1 46.1 ) still leads pure extraction.

### 4.3 Consistency across scale

Table 2 asks whether the perplexity gain is confined to a single model size. Across 46M, 170M, and 340M, each comparison uses a matched GDN-vs-HOLA backbone within the same scale; App. A lists the architecture, corpus, and context length for each run. The gain is consistent: HOLA lowers Wikitext perplexity by 15 15 – 16 % 16\% relative to the same-backbone GDN anchor at every scale.

### 4.4 Long-Context Retrieval

We evaluate on the official RULER benchmark ( Hsieh et al., 2024 ) (synthetic tasks, our Mistral tokenizer, limit 100) at 2k/4k/8k/16k/32k β€” up to 16 Γ— 16\times the 2k training length. On the headline single needle-in-a-haystack task, S-NIAH-1, the recurrent state (GDN) collapses with length while HOLA stays robust , with a matched HOLA+recency cache in between (Figure 1 b): at 32k, HOLA reaches 0.58 0.58 vs. HOLA+recency 0.24 0.24 vs. GDN 0.14 0.14 β€” a + 0.44 +0.44 margin over the state, and + 0.64 +0.64 at 16k.

The advantage extends to most above-floor RULER cells (Table 3 ), grouped into single-needle (S-NIAH-1/2/3) and multi-needle (multi-key/-value/-query): HOLA wins the harder single-needle variants by wide margins where they are not floored (S-NIAH-2 at 8k, 0.35 0.35 vs. 0.09 0.09 ) and most multi-needle cells (multi-value 0.28 0.28 vs. 0.17 0.17 at 2k), with a few near-tie exceptions. We also include a full-attention Transformer++ as a 2k full-attention ceiling: within its training length it is the strongest model (exact softmax retrieval), but this RoPE checkpoint is not a length-extrapolating baseline and every shown task drops to 0 0 at 4k and beyond β€” exactly the regime where HOLA remains useful.

### 4.5 Ablations

HOLA+recency vs. importance eviction (340M, matched memory). The sharpest test of contribution 1 is a matched control β€” the identical architecture ( w = 64 w{=}64 , same chunk, sharpened read, gate, and kernel), changing only the eviction signal: position (keep the most recent w w ) vs. surprise (keep top- w w by Ξ² β‹… βˆ₯ e βˆ₯ \beta\!\cdot\!\lVert e\rVert ). We report the HOLA+recency control as a first-class row throughout, and importance wins on the axes the cache is designed to affect: lower perplexity (Wiki β†’ 22.92 25.04\!\to\!22.92 ; Table 1 ) and far stronger long-context retrieval (Figure 1 b, Table 3 ; S-NIAH-1 at 32k, HOLA+recency 0.24 0.24 vs. HOLA 0.58 0.58 ). Most tellingly, the recency cache barely improves on no cache at all on the far needle : at 32k it recalls only 0.24 0.24 , marginally above the no-cache state (GDN 0.14 0.14 ) and far below importance eviction (HOLA 0.58 0.58 ) β€” a recency window gives little far-needle benefit, because the needle slides out of it, whereas surprise-based eviction keeps it. For a bounded exact memory, what to cache matters more than how recent . 1 1 1 For the recency control, we use full recomputation to avoid implementation-path confounds; this is the path used for the PPL, commonsense, retrieval, and RULER numbers we report.