Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs — part 2/4

Cleaned-text code points [15885, 30194). Bibliography omitted here; source.md includes it.

Figure 3 reveals a Perplexity Paradox unique to Qwen2.5-Math: on contaminated datasets (MATH-500), answer-only perplexity decreases progressively while full-text perplexity increases . In contrast, both LLaMA and OLMo (Appendix D ) exhibit rising perplexity in both metrics, indicating that spurious RLVR degrades their language modeling without activating memorization shortcuts. On leakage-free LiveMathBench, PPL remains at the highest level. This divergence constitutes a Perplexity Paradox: minimizing loss on specific memorized answers under spurious RLVR leads to degraded language modeling on input prompts, aligning with the “over-memorization” hypothesis ( Ruan et al., 2025 ) .

### 4.3 Locating the Functional Anchor: Static Analysis

Having established existence of memorization shortcuts, we now pinpoint their physical location within model’s depth. We employ three complementary mechanistic probes:

Path Patching: The Functional Cause. To identify which components causally drive the correct output, we perform activation patching. We replace the activations of the base model with those from the spurious RLVR-tuned model at specific layers and measure the recovery of accuracy. Figure 4 reveals two critical findings specific to Qwen: (1) Patching the MLPs yields significantly higher accuracy recovery than patching attention heads, confirming that MLPs are the primary storage for this learned knowledge. (2) In Qwen, the accuracy recovery maintains a consistently high level across the erly layers, but it is the sustained high recovery at Layers 18–20 immediately before the precipitous drop at Layer 21, combined with JSD scores beginning to climb at these layers shown in Figure 5 , that highlights L18–L20 as the critical window where the effective signal is fully present immediately before the performance decline. In comparison, the LLaMA model shows no such significant recovery or layer-specific trends. This confirms that the specific memorization circuit identified in Qwen is not a general artifact of RLVR, but a symptom of activating contaminated knowledge. (See Appendix E for results on OLMo and Qwen3).

JSD: The Structural Effect. While Path Patching measures causal effect on accuracy, we employ a counterfactual JSD analysis to isolate each MLP sub-component’s marginal contribution to the distributional shift. For the gated MLP architecture: MLP ​ ( 𝐱 ) = ( σ ⁡ ( 𝐱𝐖 gate ) ⊙ 𝐱𝐖 up ) ​ 𝐖 down , \text{MLP}(\mathbf{x})=\bigl(\sigma(\mathbf{x}\mathbf{W}_{\text{gate}})\odot\mathbf{x}\mathbf{W}_{\text{up}}\bigr)\mathbf{W}_{\text{down}}, we perform the following counterfactual interventions: (1) 𝐖 up \mathbf{W}_{\text{up}} contribution: Replace 𝐖 up \mathbf{W}_{\text{up}} with its mean-value matrix 𝐖 ¯ up = 𝔼 ⁡ [ 𝐖 up ] ⋅ 𝟏 \bar{\mathbf{W}}_{\text{up}}=\mathbb{E}[\mathbf{W}_{\text{up}}]\cdot\mathbf{1} while keeping 𝐖 gate \mathbf{W}_{\text{gate}} intact; (2) 𝐖 gate \mathbf{W}_{\text{gate}} contribution: Replace 𝐖 gate \mathbf{W}_{\text{gate}} with its mean-value matrix 𝐖 ¯ gate = 𝔼 ⁡ [ 𝐖 gate ] ⋅ 𝟏 \bar{\mathbf{W}}_{\text{gate}}=\mathbb{E}[\mathbf{W}_{\text{gate}}]\cdot\mathbf{1} while keeping 𝐖 up \mathbf{W}_{\text{up}} intact; (3) 𝐖 down \mathbf{W}_{\text{down}} contribution: Replace 𝐖 down \mathbf{W}_{\text{down}} with a norm-preserving random matrix: 𝐖 rand = ‖ 𝐖 down ‖ F ‖ 𝐖 init ‖ F ​ 𝐖 init , 𝐖 init ∼ 𝒩 ⁡ ( 0 , 𝐈 ) . \mathbf{W}_{\text{rand}}=\frac{\|\mathbf{W}_{\text{down}}\|_{F}}{\|\mathbf{W}_{\text{init}}\|_{F}}\mathbf{W}_{\text{init}},\quad\mathbf{W}_{\text{init}}\sim\mathcal{N}(0,\mathbf{I}).

Figure 5 presents layer-wise counterfactual JSD scores. We observe distinct trajectories for sub-components in Qwen2.5-Math: divergence scores for both 𝐖 up \mathbf{W}_{\text{up}} and 𝐖 gate \mathbf{W}_{\text{gate}} rise steadily to reach a peak at Layers 21–22 , before declining in subsequent layers. In contrast, 𝐖 down \mathbf{W}_{\text{down}} divergence (representing total MLP output projection) maintains a sustained upward trend, converging to a high plateau. This pattern aligns with the established mechanistic role of MLPs, where 𝐖 up \mathbf{W}_{\text{up}} serves as a repository of candidate features (information vault) and 𝐖 gate \mathbf{W}_{\text{gate}} acts as the selector. The JSD peak for these components at L21–L22 suggests the completion of specific knowledge injection: the model concentrates its parametric updates here to encode the memorized features.

Crucially, this structural insight explains the divergence observed in our Path Patching results. While the internal components ( 𝐖 up/gate \mathbf{W}_{\text{up/gate}} ) gradually decreased after L22, the output projection ( 𝐖 down \mathbf{W}_{\text{down}} ) exhibits persistent high values across all subsequent layers. This indicates that L21–L22 act as Structural Adapters that permanently rotate the feature space. Consequently, although the MLP continues to function with high confidence (high JSD), its output resides in a new manifold that is structurally incompatible with the base model’s remaining layers, leading to the observed degradation in patching accuracy (L21+). In contrast, LLaMA-3.1-8B exhibits monotonically increasing JSD across all components without the characteristic peak-reversal pattern. This monotonic trend confirms that spurious RLVR fails to induce localized structural adaptation in models lacking pre-existing contaminated knowledge. (See Appendix F for OLMo and Qwen3 results, which show similar behavior).

Logit Lens: The Semantic Transition. Having localized the specific layers responsible for knowledge injection and structural adaptation, we now examine how these internal mechanisms materialize in the token space. To investigate this, we conduct a controlled case study using a obvious leakage sample from MATH-500. We sample two generation trajectories for the same prompt with a temperature of 0.7: one where the model successfully retrieves the correct answer (“4”), and one where it fails (“3”) (See Appendix J ).

In the successful trajectory (Figure 6 Top), a clear activation sequence appears. The MLP at Layer 19 first introduces a precursor signal. Following a temporary probability drop at Layers 21–22: which aligns with the feature space transformation observed in our JSD analysis, i.e., the MLP at Layer 23 aggressively injects the correct answer token “4”. The residual stream successfully integrates this signal, stabilizing on “4” until the final output. The failure case (Figure 6 Bottom) provides a critical counterfactual. Even though the model eventually outputs the wrong answer “3”, the heatmap of Figure 20 in Appendix G reveals that the MLPs at Layers 23–25 still attempt to inject the correct answer “4”. However, the initial priming signal at Layer 19 is notably weaker compared to the successful run. Without sufficient intensity from these middle layers to override the stream’s inertia, the subsequent MLP injections fail to correct the trajectory. This confirms that while the MLPs persistently store the memorized solution, acting as a fixed memory store, in light of the conclusions of the aforementioned experiments, the successful retrieval of this knowledge is causally determined by the signal strength established in these middle layers which might be in the L18–L20 region.

### 4.4 Tracking the Bifurcation: Dynamic Analysis

To mathematically validate the Functional Anchor hypothesis, we extend our analysis from discrete layer-wise probes to continuous dynamic modeling using Neural Differential Equations (NDEs) . We model main residual stream H = { h 0 , … , h L } H=\{h_{0},\dots,h_{L}\} as a continuous trajectory h ⁡ ( t ) h(t) governed by d ​ h ​ ( t ) d ​ t = f ϕ ​ ( h ⁡ ( t ) , t ) \frac{dh(t)}{dt}=f_{\phi}(h(t),t) , where f ϕ f_{\phi} is parameterized by a two-layer MLP trained to approximate discrete updates.

Trajectory Separation. Figure 7(a) visualizes the PCA-projected trajectories of leakage samples versus stable samples. The two trajectories overlap substantially in early layers but exhibit a distinct bifurcation in the middle layers, subsequently diverging into separate regions of the latent space. This visual evidence confirms that the processing of memorized and reasoned information follows fundamentally different dynamic paths after the anchor layers.

Separation Force and Velocity. We quantify this bifurcation using two metrics derived from the NDE framework: (1) Separation Force: Defined as ‖ d ​ h leak d ​ t − d ​ h stable d ​ t ‖ \left\|\frac{dh_{\text{leak}}}{dt}-\frac{dh_{\text{stable}}}{dt}\right\| , measuring the instantaneous divergence in update directions between leakage and stable samples; (2) Velocity Difference: Defined as | ‖ v leak ‖ − ‖ v stable ‖ | \left|\|v_{\text{leak}}\|-\|v_{\text{stable}}\|\right| where v = d ​ h d ​ t v=\frac{dh}{dt} , measuring the difference in update magnitudes. Figure 7(b) (Left) shows that the Separation Force peaks precisely at Layers 18, 19, and 20, mathematically confirming that these layers exert the strongest directional influence to steer leakage samples onto the memorization trajectory. Conversely, Figure 7(b) (Right) shows that the velocity difference is minimal at L19 but increases in later layers. This pattern is consistent with our Structural Adapter interpretation: the later layers amplify the signal magnitude (velocity) without substantially altering its direction (force), thereby propagating the decision made at the Functional Anchor layers.

Linear Probing Validation. As an independent validation, we train linear probes on each layer’s residual stream to classify leakage versus stable samples. Figure 7(c) shows AUC-ROC score peaking at Layer 20. This confirms that information distinguishing memorization from reasoning becomes maximally linearly separable immediately after Functional Anchor layers have performed their critical signal injection.

## 5 Ablation Study

To validate our findings regarding the Functional Anchor and Structural Adapter , we conduct systematic ablation experiments on the RLVR-tuned model. These experiments are designed to verify both the necessity and sufficiency of these layer groups for memorization. More results for other datasets like LiveMathBench are provided in Appendix H .

We design two complementary groups of experiments on the RLVR-tuned model: Group 1: Necessity Tests (Reset). We selectively replace specific layer groups with base model weights: (1) Anchor Reset : Replace L18–20 MLP weights with base model weights; (2) Adapter Reset : Replace L21–22 MLP weights with base model weights. Group 2: Sufficiency Tests (Keep Only). We retain only specific RLVR-tuned layers while resetting all others to base: (1) Keep Only Anchor : Only L18–20 retain RLVR weights; (2) Keep Only Adapter : Only L21–22 retain RLVR weights; (3) Keep Both : L18–22 retain RLVR weights.

Figure 8 shows several key findings: Anchor layers are more critical than Adapter layers. Resetting the Anchor causes leakage accuracy to drop by more than 10% on contaminated datasets (MATH-500: 98% → \to 86%; MinervaMath: 88% → \to 72%), while Adapter Reset produces smaller drops ( Δ = − 6 % \Delta=-6\% and − 10 % -10\% ). This confirms the Anchor layers serve as primary storage for memorized knowledge.

The remaining 86% accuracy after Anchor Reset is also informative. It does not imply that L18–20 alone store all contaminated answers; instead, it shows that the shortcut is anchor-triggered but partially distributed. On the MATH-500 leakage subset, the base model already reaches 78%, so replacing L18–20 with base weights preserves some pre-existing contamination while removing most RLVR-only gain (98% → \to 86%). In contrast, keeping only the Adapter reaches 80%, and randomizing the Anchor collapses accuracy to 34%, far below both base and RLVR. Thus the Anchor is best interpreted as the integrity-critical retrieval trigger, while downstream layers retain residual shortcut-compatible signal.

Neither layer group alone is sufficient for memorization. Keep Only experiments show that isolated layer groups fail to recover RLVR performance. On MinervaMath, Keep Only Adapter yields only 32% accuracy (vs. 50% base model), while Keep Only Anchor reaches 42%, and Keep Both improves to 46%. This demonstrates that Anchor and Adapter layers form a collaborative circuit for memory retrieval.

Contaminated and clean datasets exhibit divergent sensitivity. On contaminated datasets, leakage samples show high sensitivity to Anchor manipulation. Conversely, on leakage-free LiveMathBench, accuracy remains stable at 70% across all ablation conditions (Appendix H ), providing direct evidence that Anchor layers encode dataset-contaminated knowledge rather than general reasoning.

Stable samples are robust to layer manipulation. Across all reset experiments, stable sample accuracy remains nearly identical to the RLVR baseline. This indicates genuine reasoning capabilities are distributed across the broader network and do not critically depend on these specific layers, reinforcing that Anchor layers are specialized for memorization shortcuts.

We further check whether this localization is tied only to the 7B checkpoint by running a supplementary Qwen2.5-Math-1.5B analysis with the same Anchor (L18–20) and Adapter (L21–22) partition. Although the smaller model has a weaker leakage signal overall, the same qualitative pattern appears on its 38-case MATH-500 leakage subset: Anchor Reset drops accuracy from 39.5% to 2.6%, Keep Only Adapter falls to 0%, and the largest mean FFN drop occurs at L18. Path patching is again MLP-dominant, with FFN AUC 0.0789 versus attention AUC 0.0498 (paired sign-flip p = 6.99965 × 10 − 4 p=6.99965\times 10^{-4} ). We therefore frame the claim as within-family evidence rather than universality: when a Qwen2.5 model exhibits a clear leakage signal, the same Anchor-centered mechanism remains visible across scales.

## 6 Mechanistic Intervention

Building on our previous findings, we investigate whether targeted intervention on these layers can suppress this undesired behavior. We design a neuron-level steering method that directly manipulates MLP activations during inference to disrupt memorization circuit without model retraining.

### 6.1 Task-Relevant Neuron Identification