Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs β€” part 3/4

Cleaned-text code points [30194, 45778). Bibliography omitted here; source.md includes it.

In Qwen2.5-Math MLP layers, each neuron functions as a gated memory unit. For a hidden state 𝐱 β„“ ∈ ℝ d hidden \mathbf{x}_{\ell}\in\mathbb{R}^{d_{\text{hidden}}} at layer β„“ \ell , the MLP computation proceeds as: 𝐀 β„“ = SiLU ​ ( 𝐱 β„“ ​ 𝐖 gate ) βŠ™ ( 𝐱 β„“ ​ 𝐖 up ) ∈ ℝ d mlp , \mathbf{k}_{\ell}=\text{SiLU}\left(\mathbf{x}_{\ell}\mathbf{W}_{\text{gate}}\right)\odot\left(\mathbf{x}_{\ell}\mathbf{W}_{\text{up}}\right)\in\mathbb{R}^{d_{\text{mlp}}}, and MLP ​ ( 𝐱 β„“ ) = 𝐀 β„“ ​ 𝐖 down = βˆ‘ i = 1 d mlp k β„“ ( i ) β‹… 𝐰 i down , \text{MLP}(\mathbf{x}_{\ell})=\mathbf{k}_{\ell}\mathbf{W}_{\text{down}}=\sum_{i=1}^{d_{\text{mlp}}}k_{\ell}^{(i)}\cdot\mathbf{w}_{i}^{\text{down}}, where d mlp ≫ d hidden d_{\text{mlp}}\gg d_{\text{hidden}} (e.g., 14336 ≫ 3584 14336\gg 3584 in Qwen2.5-Math-7B), and 𝐰 i down ∈ ℝ d hidden \mathbf{w}_{i}^{\text{down}}\in\mathbb{R}^{d_{\text{hidden}}} is the i i -th row of 𝐖 down ∈ ℝ d mlp Γ— d hidden \mathbf{W}_{\text{down}}\in\mathbb{R}^{d_{\text{mlp}}\times d_{\text{hidden}}} . Each scalar k β„“ ( i ) k_{\ell}^{(i)} acts as a key that gates contribution of its corresponding neuron value 𝐰 i down \mathbf{w}_{i}^{\text{down}} to output. We project each neuron’s value vector into vocabulary space via the unembedding matrix: 𝐯 i = LayerNorm ​ ( 𝐰 i down ) ​ 𝐖 U ∈ ℝ | V | . \mathbf{v}_{i}=\text{LayerNorm}(\mathbf{w}_{i}^{\text{down}})\mathbf{W}_{U}\in\mathbb{R}^{|V|}. We then compute a relevance score combining key activation magnitude and semantic overlap with answer tokens: s i = | k β„“ ( i ) | β‹… ( 1 + Ξ» β‹… | Top-10 ​ ( 𝐯 i ) ∩ Tokens ​ ( y ) | | Tokens ​ ( y ) | ) , s_{i}=|k_{\ell}^{(i)}|\cdot\left(1+\lambda\cdot\frac{|\text{Top-10}(\mathbf{v}_{i})\cap\text{Tokens}(y)|}{|\text{Tokens}(y)|}\right), where Top-10( 𝐯 i \mathbf{v}_{i} ) denotes the 10 highest-probability tokens after softmax, y y is the ground-truth answer, and Ξ» \lambda is used to control the intensity of semantic alignment. We select the top- k k neurons ( k = 10 k=10 ) ranked by these scores as the intervention target set 𝒩 β„“ task \mathcal{N}_{\ell}^{\text{task}} .

### 6.2 Causal Steering

We apply multiplicative scaling to selected neurons’ key activations: k ^ β„“ ( i ) = { Ξ± β‹… k β„“ ( i ) for i ∈ 𝒩 β„“ task ; else k β„“ ( i ) } , \hat{k}_{\ell}^{(i)}=\{\alpha\cdot k_{\ell}^{(i)}\text{ for }i\in\mathcal{N}_{\ell}^{\text{task}};\text{ else }k_{\ell}^{(i)}\}, where Ξ± \alpha is scaling factor, applied uniformly across all generation steps. To validate our neuron selection strategy, we first examine activation magnitudes k β„“ k_{\ell} of the identified task-relevant neurons across layers. Figure 9 reveals that these neurons exhibit peak activations precisely at L19, L21, and L24 that align with both our Functional Anchor hypothesis and the MLP injection patterns observed in Logit Lens analysis (Figure 6 ). This convergence of evidence confirms that our selection method successfully identifies neurons that are mechanistically central to memorization circuit.

### 6.3 Results

Figure 10 presents accuracy change when applying neuron-level steering across representative layers on full leakage datasets. Layer 18 exhibits maximal sensitivity ( βˆ’ 3.8 % -3.8\% under suppression, + 4.4 % +4.4\% under amplification), consistent with its role as Functional Anchor . In contrast, Layer 25 shows uniform degradation across all factors, reflecting its role as a Structural Adapter where arbitrary manipulation disrupts transformation process. Critically, on leakage-free dataset, steering yields no systematic pattern, confirming that intervention specifically targets contamination-dependent circuits rather than general reasoning pathways.

To elucidate the fine-grained pathway through which neuron scaling modulates memorization retrieval, we conduct layerwise trajectory analysis on individual leakage samples in Figure 24(a) of Appendix I . Under amplification ( Ξ± = 3.0 \alpha=3.0 ), a special formatting token reaches top-1 at Layer 22 one layer earlier, and the target answer token achieves higher probability. Conversely, suppression ( Ξ± = 0.2 \alpha=0.2 ) delays trigger activation and reduces answer token probability. This bidirectional control over both the timing and strength of retrieval provides direct mechanistic evidence that the Functional Anchor neurons constitute a causally sufficient pathway for memory activation. Beyond gradual modulation, we observe a more striking phenomenon in cases where baseline retrieval fails entirely as shown in Figure 24(b) of Appendix I . For Question 213 (ground truth: β€œ2”), both suppression and baseline produce incorrect outputs with uniformly low token probabilities. However, amplification triggers a qualitatively different trajectory: starting from the middle layers, a dominant token maintains high probability, while the correct answer β€œ2” emerges abruptly only in the final two layers. This suggests that amplification activates a latent shortcut pathway inaccessible to the base model.

### 6.4 Probe-Gated Shortcut Audit

The global steering results above establish causal controllability, but a practical audit should not suppress the circuit on every prompt. We therefore add a selective variant that first scores each prompt with the Layer-19 leakage probe and intervenes only when the score exceeds a conservative threshold calibrated at the 99th percentile of negative-control prompts. When triggered, we apply the same inference-time mechanism at the identified circuit, using Layer 19, the top-50 probe-ranked neurons, and a suppression factor Ξ± = 0.5 \alpha=0.5 . This turns the intervention into a counterfactual test: if an RLVR success disappears only under the gated intervention, the success is shortcut-dependent; if the gate stays inactive, the benchmark result is left unchanged.

Table 1 shows that the gate activates on 61.7% of the MATH-500 wrong β†’ \rightarrow right subset and removes 19 RLVR-only successes, reducing accuracy from 100.0% to 85.16% (exact McNemar p = 3.81 Γ— 10 βˆ’ 6 p=3.81\times 10^{-6} ). In contrast, the gate fires on 0/30 AIME-2025 and 0/100 LiveMathBench examples, so conditional accuracy remains identical to the RLVR checkpoint. The same pattern holds on additional recent benchmarks: OlymMATH-en-easy is unchanged, and OmniMATH100-rule changes by only one point with a nonsignificant paired test ( p = 1.0 p=1.0 ). Thus, the intervention is best understood as an offline shortcut-dependence audit rather than a blanket reasoning improver: it selectively stress-tests suspicious gains while preserving fresh-benchmark behavior.

## 7 Conclusion

In this work, we present a mechanistic analysis of how spurious RLVR induces stubborn memorization in the Qwen2.5-Math models. We identify the Perplexity Paradox: rising prompt perplexity alongside falling answer perplexity, as a macroscopic fingerprint of this phenomenon, where general language coherence is sacrificed for shortcut learning. Mechanistically, we uncover a fundamental decoupling between function and structure: utilizing Path Patching, Logit Lens, JSD analysis, and Neural Differential Equations (NDEs), we pinpoint Layers 18–20 as the Functional Anchor that causally injects the memorization trigger, while Layers 21+ serve as Structural Adapters that undergo drastic weight reorganization to accommodate this signal. Ablation experiments confirm that these layers encode dataset-specific shortcuts rather than general reasoning improvements, as manipulation selectively degrades contaminated benchmarks while leaving clean benchmarks unaffected. Finally, we demonstrate that scaling specific MLP keys within this circuit allows for bidirectional causal steering, i.e., artificially amplifying or suppressing contamination-driven performance. With a conservative probe gate, the same intervention becomes a selective shortcut audit that suppresses suspicious contaminated gains while preserving behavior on fresh benchmarks. Our findings provide mechanistic grounding for understanding RLVR vulnerabilities and open avenues for targeted mitigation strategies.

## Acknowledgments

This work has received funding from the European Union’s Horizon Europe research and innovation programme project TrustLLM under grant agreement No 101135671.

## Impact Statement

This work presents a mechanistic investigation into how reinforcement learning can inadvertently trigger shortcuts by activating latent memorized data rather than fostering genuine reasoning. The implications of our findings are twofold: 1. Strengthening AI Trustworthiness and Evaluation : Our discovery of the Anchor-Adapter circuit and the Perplexity Paradox provides new tools for the AI community to distinguish between appearance of capability and genuine generalization. As LLMs are increasingly deployed in critical domains like mathematics, science, and coding, it is essential that their performance is rooted in robust reasoning. By providing a methodology to localize and steer these shortcuts, our work helps developers identify and mitigate the effects of data contamination, leading to more honest and reliable AI systems.

2. Mitigating the Risks of Spurious Optimization : We highlight a critical vulnerability in RLVR: the potential for models to optimize for reward signals through stubborn memorization. This underscores the need for more sophisticated reward functions and evaluation benchmarks that are resistant to leakage. Furthermore, our causal steering techniques offer a path toward de-contaminating models post-training, reducing the risk of deploying models that appear performant but fail on novel, out-of-distribution tasks.

Ethical Considerations : While our work provides tools to detect and suppress memorization, the same techniques could theoretically be used to amplify a model’s reliance on specific datasets. We advocate for the responsible use of mechanistic interpretability tools to ensure they are used to enhance transparency and safety rather than to mask architectural flaws.

## Appendix A Code

We release our code: https://github.com/idwts/How-RLVR-Activates-Memorization-Shortcuts .

## Appendix B Preliminaries

Our work builds upon the emerging understanding of RLVR and the specific vulnerabilities identified in Qwen2.5 architecture. Recent findings indicate that while RLVR aims to reinforce robust reasoning paths ( Shao et al., 2025 ) , in models like Qwen2.5, it can inadvertently exploit data contamination to trigger memorization rather than generalization ( Wu et al., 2025b ) . To dissect the internal mechanisms driving this phenomenon, we ground our methodology in mechanistic interpretability ( Conmy et al., 2023 ) . We focus on Transformer decoder-only architectures, modeling the propagation of information through the residual stream to identify the physical location of memory activation.

Multilayer Perceptron (MLP). In Transformer architectures, MLPs are widely regarded as the primary storage units for parametric knowledge ( Geva et al., 2021 ; Kim et al., 2024 ) . Each MLP layer functions as a key-value memory, where the input vector acts as a query to activate specific neurons (β€œkeys”), which then retrieve corresponding β€œvalues” to update the residual stream. In the context of our study, MLPs are the critical components of interest, as they are hypothesized to store and retrieve the specific memorized solutions triggered by spurious RLVR.

Path Patching. To causally attribute the model’s output to specific internal components, we employ Path Patching ( Meng et al., 2022 ) . This technique allows us to measure the importance of specific edges in the computational graph (e.g., from a specific layer’s output to the final logits) by intervening on activations. In our context, we swap activations between Leakage samples (where memorization is activated) and Stable samples (where genuine reasoning is maintained). By observing which layer interventions successfully restore the memorized output in the stable context, we can isolate the specific components causally responsible for triggering the memory retrieval shortcut.

Logit Lens. To visualize the intermediate decoding process, we utilize the Logit Lens technique ( nostalgebraist, 2020 ) . This method projects hidden states 𝐱 β„“ \mathbf{x}_{\ell} from intermediate layer β„“ \ell directly into the vocabulary space using the model’s pre-trained unembedding matrix 𝐖 U \mathbf{W}_{U} . Formally: LogitLens ​ ( 𝐱 β„“ ) = LayerNorm ​ ( 𝐱 β„“ ) ​ 𝐖 U \text{LogitLens}(\mathbf{x}_{\ell})=\text{LayerNorm}(\mathbf{x}_{\ell})\,\mathbf{W}_{U} . This allows us to track the layer-wise emergence of the target answer tokens, revealing exactly when the model commits to a memorized answer before the final layer.

Jensen-Shannon Divergence (JSD). To rigorously quantify the alignment between intermediate hidden states and the model’s final decision, we employ the Jensen-Shannon Divergence (JSD). Building on the Logit Lens projection, we define q ⁑ ( x β„“ ) = softmax ​ ( LogitLens ​ ( x β„“ ) ) q(x_{\ell})=\text{softmax}\bigl(\text{LogitLens}(x_{\ell})\bigr) as the vocabulary distribution derived from layer β„“ \ell , and P final P_{\text{final}} as the model’s final output distribution. The divergence score is calculated as JSD ( q ( x β„“ ) βˆ₯ P final ) = 1 2 KL ( q ( x β„“ ) βˆ₯ M ) + 1 2 KL ( P final βˆ₯ M ) \mathrm{JSD}\bigl(q(x_{\ell})\parallel P_{\mathrm{final}}\bigr)=\tfrac{1}{2}\mathrm{KL}\bigl(q(x_{\ell})\parallel M\bigr)+\tfrac{1}{2}\mathrm{KL}\bigl(P_{\mathrm{final}}\parallel M\bigr) , where M = 1 2 ​ ( q ⁑ ( x β„“ ) + P final ) M=\tfrac{1}{2}\bigl(q(x_{\ell})+P_{\text{final}}\bigr) represents the average distribution and KL denotes the Kullback-Leibler divergence. Similar to the leave-one-out strategy in the context attribution setting ( Cohen-Wang et al., 2024 ; Li et al., 2025b ) , a high JSD score indicates that the intermediate representation at layer β„“ \ell differs substantially from the final output distribution, suggesting that significant computational transformations occur in subsequent layers.

Neural Differential Equations (NDEs). To formalize the continuous evolution of hidden states, we adopt the framework of Neural Ordinary Differential Equations (Neural ODEs) ( Chen et al., 2018 ) . Since residual connections in Transformers resemble the Euler discretization of a continuous dynamical system, we model the layer-wise trajectory of hidden states as a continuous flow. Specifically, the residual block 𝐱 β„“ + 1 = 𝐱 β„“ + β„± ΞΈ ​ ( 𝐱 β„“ ) \mathbf{x}_{\ell+1}=\mathbf{x}_{\ell}+\mathcal{F}_{\theta}(\mathbf{x}_{\ell}) can be viewed as one step of Euler integration with step size h = 1 h=1 for the ODE: d ​ 𝐱 ​ ( t ) d ​ t = β„± ΞΈ ​ ( 𝐱 ⁑ ( t ) ) . \frac{\mathrm{d}\mathbf{x}(t)}{\mathrm{d}t}=\mathcal{F}_{\theta}\bigl(\mathbf{x}(t)\bigr). This continuous formulation allows us to compute the separation force acting on latent representations and to mathematically pinpoint the bifurcation point where the trajectory of a memorized sample physically diverges from that of a reasoning sample. In our implementation, we approximate β„± ΞΈ \mathcal{F}_{\theta} using a two-layer MLP fitted to the discrete layer-wise updates Ξ” ​ 𝐱 β„“ = 𝐱 β„“ + 1 βˆ’ 𝐱 β„“ \Delta\mathbf{x}_{\ell}=\mathbf{x}_{\ell+1}-\mathbf{x}_{\ell} .

## Appendix C Partial Prompt Evaluation Details

This appendix provides the complete results for the Partial Prompt Evaluation discussed in Section 3.1. This experiment assesses whether models possess pre-existing knowledge of the ground-truth answers by prompting them with the question text alone (without Chain-of-Thought) to see if they can complete the exact answer string.