Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs — part 1/4

Cleaned-text code points [0, 15885). Bibliography omitted here; source.md includes it.

# Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

###### Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is highly effective for enhancing LLM reasoning, yet recent evidence shows models like Qwen2.5 achieve significant gains even with spurious rewards. We investigate this phenomenon and identify “Perplexity Paradox”: spurious RLVR triggers a divergence where answer-token perplexity drops while prompt-side coherence degrades, suggesting model is bypassing reasoning in favor of memorization. Using a suite of mechanistic interpretability tools, including Path Patching and Logit Lens, we identify a previously unknown Anchor–Adapter circuit. This circuit enables model to bypass reasoning and directly retrieve memorized solutions under spurious RLVR. We localize a Functional Anchor in middle layers (L18–20) that triggers retrieval of memorized solutions, followed by Structural Adapters in later layers (L21+) that transform representations to accommodate shortcut signal. Finally, we demonstrate that scaling specific MLP keys within this circuit allows for bidirectional causal steering, i.e., artificially amplifying or suppressing contamination-driven performance. Our results provide a mechanistic roadmap for identifying and mitigating data contamination in RLVR-tuned models.

###### Keywords:

## 1 Introduction

Reinforcement learning with verifiable rewards (RLVR) has established itself as a powerful paradigm for advancing the reasoning capabilities of Large Language Models (LLMs), by treating the final-answer correctness as the reward signal, showing particular promise in deterministic domains such as mathematics and coding ( Guo et al., 2025 ; Lambert et al., 2024 ) . The standard interpretation of RLVR’s success relies on the assumption that performance improvements derive from models optimizing robust reasoning policies through ground-truth supervision. However, recent empirical studies have complicated this narrative: the Qwen2.5-Math model family achieved significant performance gains on standard benchmarks (e.g., MATH-500, AIME) even when trained with spurious rewards—signals that were random, purely format-based, or even incorrect ( Shao et al., 2025 ) . Building on these observations, another work ( Wu et al., 2025b ) argued that such gains are less likely the result of genuine reasoning generalization and more attributable to data contamination, identifying that Qwen2.5 had likely memorized the test sets during pre-training.

Synthesizing these observations raises a critical question: if the base model already contained the contaminated data, why was its initial accuracy limited, and how could training with incorrect rewards paradoxically unlock this performance? We hypothesize that spurious RLVR activates the model’s latent capacity for memorization, overriding general reasoning pathways in favor of retrieving stored answers . To validate this hypothesis, it is essential to situate these findings within the reasoning-memorization in LLMs. Recent interpretability research has sought to distinguish these behaviors, with studies identifying specific neuron activation patterns that signal memorization ( Slonski, 2024 ) or isolating linear directions in the residual stream that mediate the switch between reasoning and memory modes ( Hong et al., 2025 ) . Notably, the phenomenon of “over-memorization” has been linked to counterintuitive loss landscapes, where models maintain high test accuracy despite exhibiting high test perplexity ( Ruan et al., 2025 ) : a finding that strongly corroborates the behavioral anomalies we observe in this work. However, while these studies characterize memorization in general pre-training or fine-tuning contexts, they do not explain the specific dynamics of spurious RLVR. A critical gap remains: distinct from standard training, how exactly does spurious RLVR trigger this “mode switch”? The internal mechanisms that allow Qwen2.5 to abandon general reasoning for stubborn memorization under such conditions remain largely opaque.

To address this questions, we investigate the learning dynamics of LLMs subjected to spurious RLVR training. We define spurious RLVR as reinforcement learning with verifiable rewards where the reward signal is decoupled from genuine reasoning quality, including random rewards, format-only rewards, or intentionally incorrect rewards. We uncover a phenomenon that mirrors the “over-memorization” signatures identified in prior work ( Ruan et al., 2025 ) : as spurious rewards typically degrade language modeling globally, the contaminated model exhibits a unique divergence where perplexity on answer tokens decreases significantly, even as prompt perplexity rises . This divergence provides compelling evidence that the Qwen model is not acquiring genuine reasoning capabilities ( Yue et al., 2025 ) ; instead, the optimization trajectory results in degraded prompt-side coherence as a side effect of shortcut formation, a direct mapping from specific prompts to memorized answers.

To uncover the physical location of this shortcut, we employ a comprehensive mechanistic analysis combining Path Patching ( Meng et al., 2022 ) , Logit Lens ( nostalgebraist, 2020 ) , and Jensen-Shannon Divergence (JSD) metrics. Our investigation reveals a crucial decoupling between function and structure within the Qwen model’s layers. Specifically, we identify the middle layers (L18–L20) as the Functional Anchor , where the critical decision to retrieve a memorized answer is causally determined and a high-probability trigger token is injected. Conversely, the subsequent layers (L21+) act as Structural Adapters , exhibiting the most substantial weight changes (high JSD) not to store knowledge, but to perform a representational transformation that reorganizes the internal state to accommodate the abrupt signal from the functional anchor layers.

We further formalize this dynamic process by modeling the layer-wise evolution of hidden states as a continuous trajectory ( Li et al., 2025a ; Oh et al., 2024 ; Lu et al., 2019 ) using Neural Differential Equations (NDEs) ( Chen et al., 2018 ) . By quantifying the “separation force” between leakage and stable samples in latent space, our NDE analysis mathematically confirms that the trajectory bifurcation, the physical divergence point where the model abandons standard processing pathways to engage the specialized memorization circuit, is determined at the functional anchor layers.

Crucially, we move beyond passive observation to active control. We identify specific multilayer perceptron (MLP) neurons within these layers that mediate the shortcut retrieval. By performing a mechanistic intervention, specifically scaling the keys of these MLP neurons, we demonstrate the ability to causally steer the Qwen model’s behavior. We show that adjusting these scaling factors can either amplify the model’s reliance on contaminated data or suppress the shortcut to reveal the model’s underlying baseline performance. This provides definitive causal evidence for the Anchor-Adapter circuit and suggests a new methodology for mitigating the effects of data contamination in RLVR.

To summarize, our work has three main contributions: 1. Discovery of the Anchor-Adapter Circuit: We causally localize mechanism of Qwen 2.5 for spurious gains into two distinct components: Functional Anchor (L18-L20) : middle layers acting as the decisive trigger for retrieving memorized answers; Structural Adapters (L21+) : later layers undergoing weight transformations to accommodate shortcut signals rather than storing new knowledge.

2. Empirical Validation of Contamination: Through ablations, we demonstrate that these specific circuits are dataset-dependent, and manipulating them eliminates performance gains on contaminated benchmarks while leaving general reasoning capabilities intact.

3. Causal Steering via MLP Intervention : We demonstrate a precise mechanistic intervention by scaling specific MLP keys within identified layers. This allows us to bidirectionally steer the model’s performance, artificially inflating or suppressing contamination effect, confirming causal role of these neurons in shortcut activation (see released code in Appendix A ).

## 2 Related Work

Spurious Rewards in RLVR. Recent work analyzed black box of RLVR performance gains, questioning whether improved benchmark scores reflect enhanced reasoning or sophisticated pattern matching. Shao et al. (2025) first demonstrated that models like Qwen2.5 Math can achieve substantial accuracy improvements even when supervised by random or incorrect signals, framing these as spurious rewards that challenge necessity of ground-truth feedback. This skepticism is echoed by Wu et al. (2025b) , who argue that such gains are primarily a byproduct of data contamination, where RLVR acts as a retrieval mechanism for data already memorized during pretraining rather than a catalyst for genuine reasoning. Further investigations into these learning dynamics have revealed a divergence phenomenon. Wu et al. (2025a) highlight how model-task alignment can induce misleading conclusions about RL efficacy, while Yue et al. (2025) question if RL truly incentivizes reasoning capacity beyond base model’s intrinsic limits. This is particularly evident in “over-memorization” signatures identified by Ruan et al. (2025) , where models exhibit high task accuracy despite deteriorating language modeling performance, a finding consistent with “Perplexity Paradox” we identify in this work. Additionally, Chen et al. (2025) and Alam and Rastogi (2025) explore limits of generalization, suggesting that RLVR often favors exploitation (of shortcuts) over exploration (of robust reasoning pathways). Our work builds on these behavioral observations by providing the first mechanistic explanation of how these spurious signals physically reconfigure the model’s internal architecture.

Interpretability of the Reasoning-Memorization Interplay. A parallel body of work has sought to distinguish between reasoning and memorization through mechanistic interpretability. Slonski (2024) developed methods to detect memorization patterns by monitoring specific neuron activations, while Hong et al. (2025) identified that the interplay between reasoning and memory in the residual stream is often mediated by a single, steerable direction. These studies suggest that LLMs possess latent modes that can be toggled by internal signals. However, while prior work has located these patterns in standard fine-tuning or pre-training contexts, the specific structural evolution under spurious RLVR remains largely unmapped. By identifying the Functional Anchor and Structural Adapter layers, we provide a more granular, causal account of how RLVR triggers the mode-switch from reasoning to stubborn memorization.

## 3 Preliminaries

Our study utilizes mechanistic interpretability ( Conmy et al., 2023 ) to investigate why RLVR triggers memorization over generalization in Qwen2.5 ( Shao et al., 2025 ; Wu et al., 2025b ) . We model the Transformer’s residual stream to locate memory activation.

MLP and Path Patching. MLPs in Transformer serve as key-value memories for parametric knowledge ( Geva et al., 2021 ) . To causally attribute outputs to specific components, we use Path Patching ( Meng et al., 2022 ) , swapping activations between Leakage (memorized) and Stable (reasoning) samples. This identifies edges in the computational graph responsible for triggering memory retrieval shortcuts.

Logit Lens and Jensen-Shannon Divergence (JSD). We monitor the layer-wise emergence of tokens via the Logit Lens ( nostalgebraist, 2020 ) : q ⁡ ( x ℓ ) = softmax ​ ( LN ​ ( 𝐱 ℓ ) ​ 𝐖 U ) q(x_{\ell})=\text{softmax}(\text{LN}(\mathbf{x}_{\ell})\mathbf{W}_{U}) . To quantify the alignment between this intermediate distribution and the final output P final P_{\text{final}} , we compute the JSD: JSD ( q ( x ℓ ) ∥ P final ) = 1 2 KL ( q ( x ℓ ) ∥ M ) + 1 2 KL ( P final ∥ M ) \mathrm{JSD}(q(x_{\ell})\parallel P_{\text{final}})=\tfrac{1}{2}\mathrm{KL}(q(x_{\ell})\parallel M)+\tfrac{1}{2}\mathrm{KL}(P_{\text{final}}\parallel M) , where M = 1 2 ​ ( q ⁡ ( x ℓ ) + P final ) M=\tfrac{1}{2}(q(x_{\ell})+P_{\text{final}}) . High JSD indicates significant remaining computation before the model commits to an answer.

Neural Differential Equations (NDEs). We treat the residual update 𝐱 ℓ + 1 = 𝐱 ℓ + ℱ θ ​ ( 𝐱 ℓ ) \mathbf{x}_{\ell+1}=\mathbf{x}_{\ell}+\mathcal{F}_{\theta}(\mathbf{x}_{\ell}) as an Euler discretization of the continuous system ( Chen et al., 2018 ) : d ​ 𝐱 ​ ( t ) d ​ t = ℱ θ ​ ( 𝐱 ⁡ ( t ) ) \frac{\mathrm{d}\mathbf{x}(t)}{\mathrm{d}t}=\mathcal{F}_{\theta}(\mathbf{x}(t)) . By approximating ℱ θ \mathcal{F}_{\theta} with an MLP fitted to discrete updates Δ ​ 𝐱 ℓ \Delta\mathbf{x}_{\ell} , we compute the separation force to identify the bifurcation point where memorized and reasoning trajectories diverge (See Appendix B for details).

## 4 Empirical Study

### 4.1 Experiment Setting

We conduct experiments on the original Qwen2.5-Math-7B model ( Yang et al., 2024 ) and its spurious RLVR-tuned checkpoint ( Shao et al., 2025 ) . The benchmark suite encompasses six representative mathematical reasoning datasets: AIME2024, AIME2025 ( Li et al., 2024 ) , MATH500 ( Lightman et al., 2023 ) , AMC ( Li et al., 2024 ) , MinervaMath ( Lewkowycz et al., 2022 ) , and LiveMathBench ( Liu et al., 2025 ) . The results are shown in Figure 1 .

Among these datasets, we select MATH-500 and MinervaMath as our primary subjects for analyzing contamination, as they exhibit the most dramatic performance shifts under spurious RLVR. Notably, LiveMathBench serves as our rigorous control group for leakage-free evaluation ( Wu et al., 2025b ) . We observe that while accuracy on LiveMathBench improves marginally under spurious RLVR, this gain fundamentally differs from the memorization observed on contaminated sets. Crucially, the specific subset of questions that improve on LiveMathBench remains consistent across correct, incorrect, and random reward signals. This indicates that the performance gain might stems from format alignment and the elicitation of latent capabilities (e.g., standardizing solution structure) rather than answer memorization. This distinction is further validated by our partial prompt and perplexity analysis, where LiveMathBench maintains an uncertain improvement (Figure 2 ) and high perplexity (Figure 3 ), confirming that the model is optimizing how to reason, not what to answer.

Following Wu et al. (2025b) , we conduct a Partial Prompt Evaluation to assess the base model’s ability to complete the questions and generate correct answers from memory. For comparison, we performed the same evaluation on LLaMA-3.1-8B ( Grattafiori et al., 2024 ) , OLMo-2-1124-7B ( OLMo et al., 2024 ) and Qwen3-8B ( Yang et al., 2025 ) . While LLaMA and OLMo show no memorization effect, Qwen3-8B exhibits the same contamination pattern as Qwen2.5-Math-7B (full results and datasets are provided in the Appendix C ). Unless stated otherwise, every mechanistic probe (Path Patching, JSD, NDEs, etc.) is executed on the RLVR-tuned models by contrasting the internal activations of these two groups on matched questions.

### 4.2 What Does Perplexity Show

We first investigate the macroscopic learning dynamics by tracking how the model’s uncertainty evolves during spurious RLVR training. We calculate the perplexity (PPL) of the target answer sequence Y = ( y 1 , … , y T ) Y=(y_{1},\dots,y_{T}) given the prompt X X : PPL ( Y | X ) = exp ( − 1 T ∑ t = 1 T log P ( y t ∣ X , y < t ) ) . \text{PPL}(Y|X)=\exp(-\frac{1}{T}\sum_{t=1}^{T}\log P(y_{t}\mid X,y_{<t})). We track this metric across incorrect rewards checkpoints at step 0 (base model), 50, 100, and 150.