A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets — part 3/3

Cleaned-text code points [30831, 36557). Bibliography omitted here; source.md includes it.

Eviction signal (Table 4 ). We isolate the eviction rule with the same 46M backbone, cache size, and flat cache read, and evaluate the two axes a bounded cache is meant to balance: whether the state still carries a far needle after the cache window has moved on, and how many exact key–value facts the cache can retain within its span. The far-needle column uses a lightweight passkey-style probe inspired by Mohtashami & Jaggi (2023) : we place a numeric key at a controlled depth in a 4k context, teacher-force the answer tokens after the query, and report next-token accuracy. Pure residual surprise, ∥ e ∥ \lVert e\rVert , is not enough; it lacks the write-strength utility signal and performs poorly on the far needle. Multiplying by the GDN write gate gives β ⋅ ∥ e ∥ \beta\!\cdot\!\lVert e\rVert , the actual delta-rule update magnitude, which is best or tied-best on every diagnostic column while also giving the lowest WikiText perplexity.

Sharpened read (Table 5 ). With the eviction rule fixed to β ⋅ ∥ e ∥ \beta\!\cdot\!\lVert e\rVert , the decisive read-side change is not a larger fixed temperature but the normalization used to form cache logits. Unit-L2 queries and keys make the cache read too flat, so the cache behaves like a soft average. RMSNorm- γ \gamma keeps the natural d \sqrt{d} logit scale while letting the model tune it, turning the same bounded cache into an exact local memory: it lowers WikiText perplexity by more than ten points and sharply improves multi-key capacity, without losing the far-needle behavior of the recurrent state.

## 5 Conclusion and Limitations

#### Conclusion.

Starting from a semiparametric test-time regression view — a linear-attention recurrent state is the parametric estimator for compressible key–value structure, but needs a bounded set of exact KV pairs for exact associations — we attached a hippocampus-like KV cache to every layer of Gated DeltaNet. The cache is guided by two design choices: (i) eviction driven by a parameter-free, intrinsic surprise signal β ⋅ ∥ e ∥ \beta\!\cdot\!\lVert e\rVert (the delta-rule write magnitude) — keep what wrote most to the state, not what is most recent; and (ii) a read sharpened by a decoupled RMSNorm- γ \gamma , so the cache performs exact retrieval rather than soft averaging. At 340M parameters trained on 15B SlimPajama tokens, HOLA improves language modeling and recall together: lowest perplexity of all compared models (Wikitext 22.92, below a full-attention Transformer++), best linear in-context retrieval, and competitive commonsense (six-task average); it is length-robust on RULER needle-in-a-haystack recall out to 32k ( 16 × 16\times the training length), holding 0.58 0.58 where the recurrent baseline collapses to 0.14 0.14 and the 2k-trained full-attention checkpoint reaches 0 0 . The perplexity advantage holds from 46M to 340M. The take-away: a linear-attention model’s own update rule already diagnoses what it fails to remember; spending a small exact memory on exactly those “surprising” tokens recovers the long-range exact recall it loses.

#### Limitations.

The cache is deliberately bounded: it spans only w + C + 1 ≈ 321 w{+}C{+}1{\approx}321 tokens, so in very long or needle-dense contexts it cannot retain every relevant item, and single-needle recall is 0.58 0.58 rather than perfect at 32k. HOLA also narrows, but does not close, the gap to full attention on pure token-exact extraction such as FDA, where every token can matter. Finally, the main-scale results are single-seed up to 340M; while the matched recency comparison and 46M diagnostics support the β ⋅ ∥ e ∥ \beta\!\cdot\!\lVert e\rVert eviction rule, we have not run a matched-memory comparison against learned eviction modules such as LTE’s CNN.

## Appendix A Scale configurations

Table 6 lists the architecture, corpus size, and context length for the scaling comparison in Table 2 . Within each row, GDN and HOLA use the same backbone and training recipe; HOLA only adds the bounded exact KV cache.

For 170M and 340M, the architecture family follows the GDN recipe: 4 4 heads × \times head-dim 256 256 , expand _ v = 1 \mathrm{expand\_v}{=}1 , hidden _ ratio = 4 \mathrm{hidden\_ratio}{=}4 , conv 4, tied embeddings, vocabulary 32000 32000 , Mistral tokenizer, and AdamW with peak lr 4 × 10 − 4 4{\times}10^{-4} . The 46M row uses a smaller 12-layer, d model = 512 d_{\mathrm{model}}{=}512 architecture trained on FineWeb-Edu for 0.5B tokens at ctx 4096; this is the scale used for the component studies in Tables 4 – 5 .

HOLA cache hyperparameters: evict=betae, window=64, chunk=256, gate_init= − 4.0 -4.0 , cache_norm=rms, tau_init=1.0, tau_freeze=true, cache_kernel=sdpa . The GDN baseline shares all other settings within each row.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .