Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs — part 4/4

Cleaned-text code points [45778, 56464). Bibliography omitted here; source.md includes it.

Cross-Model Accuracy Comparison (Figure 11 ). To provide context for the Partial Prompt Evaluation, we first present the overall accuracy of Qwen3-8B, LLaMA-3.1-8B, and OLMo-2-1124-7B across six benchmarks before and after spurious RLVR training with incorrect rewards. A striking dichotomy emerges: while LLaMA and OLMo maintain consistently low accuracy regardless of RLVR training, Qwen3-8B exhibits both high baseline performance and substantial gains after spurious RLVR. This pattern mirrors the behavior observed in Qwen2.5-Math-7B and provides strong prima facie evidence that Qwen3-8B also suffers from data contamination, which spurious RLVR subsequently activates.

Qwen2.5-Math-7B (Figure 12 ). On contaminated datasets (MATH-500, MinervaMath), the “Wrong → \rightarrow Right” group (green line) shows a dramatic increase in both ROUGE-L and exact completion accuracy after RLVR. This confirms that the model is learning to retrieve specific memorized strings.

Qwen3-8B (Figure 13 ). Consistent with Qwen2.5-Math-7B, Qwen3-8B exhibits increasing ROUGE-L scores and completion accuracy as the prompt ratio increases, confirming its ability to retrieve memorized answers. However, the separation between the pre- and post-RLVR curves is less pronounced compared to Qwen2.5-Math-7B, particularly in the “Wrong → \rightarrow Right” group. This suggests that the extent of data leakage or the degree to which spurious RLVR activates these memorized answers is less significant in Qwen3-8B, indicating a potentially lower reliance on contamination or weaker memory activation under spurious RLVR conditions.

Control Models (Figure 14 ). In contrast, for LLaMA-3.1-8B and OLMo-2-1124-7B, both ROUGE-L and completion accuracy remain at a consistently low level. Crucially, these metrics show no significant correlation or discernible trend as the prompt ratios increases, confirming the absence of pre-existing contamination.

## Appendix D Complete Perplexity Analysis

This appendix provides the complete perplexity heatmaps, along with results for the OLMo-2-1124-7B control model.

Heatmap Visualization. Figure 15 presents the heatmap showing perplexity across all six benchmarks and four training checkpoints. This visualization clearly shows the dichotomy between contaminated datasets (MATH-500, AIME, MinervaMath, etc.) where answer perplexity drops, and clean datasets (LiveMathBench) where it remains high.

OLMo-2-1124-7B Analysis. Figure 16 presents the perplexity analysis for OLMo-2-1124-7B. Consistent with LLaMA, OLMo exhibits no Perplexity Paradox. Both full-text and answer-only perplexity increase uniformly across all datasets. This confirms that spurious RLVR fails to activate memorization shortcuts in models without pre-existing data contamination.

## Appendix E Additional Path Patching Results

Complementing the main text comparison, Figure 17(a) presents the path patching results for the second control model, OLMo-2-1124-7B. Similar to LLaMA, OLMo exhibits no significant accuracy recovery or distinctive layer-wise trends when patching activations from the RLVR-tuned model back to the base model. Notably, Qwen3-8B also fails to display the characteristic middle-layer peak observed in Qwen2.5-Math-7B (Figure 4(a) ). Instead, its recovery pattern similar to the control models LLaMA and OLMo, further corroborates our previous findings (Appendix C ) that its data leakage and memory activation under spurious RLVR are significantly less pronounced compared to Qwen2.5-Math-7B.

## Appendix F Additional JSD Results

We present the layer-wise MLP sub-component JSD scores for OLMo-2-1124-7B and Qwen3-8B in Figure 18 . For OLMo (Left), all three JSD components exhibit a monotonic increase across layers as with LLaMA. There is no peak or decline as observed in Qwen, indicating that OLMo does not develop the kind of localized structural adaptation associated with memorization triggers. In contrast, Qwen3-8B (Right) replicates the distinctive structural signature observed in Qwen2.5-Math-7B: both 𝐖 up \mathbf{W}_{\text{up}} and 𝐖 gate \mathbf{W}_{\text{gate}} exhibit a clear peak followed by a subsequent decline, with the peak occurring at Layer 33 . Meanwhile, 𝐖 down \mathbf{W}_{\text{down}} maintains a sustained high divergence in Qwen3-8B, consistent with its role as the output projection layer. This architectural consistency across different Qwen family models strongly suggests that the circuit is not an artifact of a specific model checkpoint, but rather a systematic property of how the Qwen architecture responds to spurious RLVR when operating on contaminated data.

## Appendix G Additional Results on Logit Lens

To further validate that the memorization activation is specific to the leakage activation observed in Qwen2.5-Math, we perform the same Logit Lens analysis on our control models, LLaMA-3.1-8B and OLMo-2-1124-7B, as well as on additional test cases for Qwen.

Control Models (LLaMA & OLMo). As shown in Figure 19 , the internal dynamics of the control models differ fundamentally from Qwen. Neither LLaMA nor OLMo exhibits the characteristic peak-valley-rise pattern or the aggressive MLP signal injection in the middle layers. Instead, the probability of the answer token remains negligible throughout the vast majority of the network depth, emerging only in the final few layers (Late Emergence). Crucially, this pattern is consistent across both correct and incorrect outputs. The MLPs in these models do not attempt to force a specific token in the intermediate layers, indicating the absence of the Functional Anchor mechanism. This flatness confirms that spurious RLVR did not activate a latent memorization shortcut in these models.

Additional Qwen Cases. We further corroborate the universality of the activation mechanism in Qwen2.5-Math by analyzing two additional distinct questions (Figure 21 ). Consistent with our main findings, both cases display the signature multi-stage injection dynamic: a high-signal precursor emerges at the Functional Anchor (L18–L20), followed by a structural adaptation valley, and finally, the aggressive injection of the correct answer token by the MLPs at Layer 23, confirming the robustness of this memorization circuit across different samples.

To verify architectural generalizability, we perform the same analysis on Qwen3-8B (Figure 22 ). As shown, Qwen3-8B replicates the identical activation pattern: in the successful case (Left), the characteristic peak-valley-rise trajectory emerges with strong middle-layer MLP signals, while the failed case (Right) exhibits weakened precursor activation. This confirms that the Functional Anchor mechanism is a systematic property of the Qwen architecture family when operating on contaminated data.

## Appendix H Additional Results on Ablation Study

In this section, we provide the comprehensive ablation results, with a specific focus on MinervaMath and LiveMathBench , which exhibit contrasting sensitivity patterns to layer manipulation.

MinervaMath Analysis. The results on MinervaMath (Figure 23 , Right) strongly corroborate the primacy of the Functional Anchor. • Necessity (Reset): Resetting the Anchor layers (L18–20) causes a accuracy drop from 88% to 72%, significantly larger than the drop caused by resetting the Adapter layers (88% → \to 78%). This confirms that the Anchor layers are the primary causal driver for retrieving contaminated knowledge.

• Sufficiency (Keep Only): Notably, the “Keep Only Adapter” setting results in catastrophic failure (32%), performing even worse than the Base Model (50%). This indicates that the Structural Adapters (L21–22) are functionally dependent on the signal from the Anchor; without the upstream trigger, they cannot sustain the memorization circuit in isolation.

LiveMathBench Analysis. In contrast to contaminated datasets, LiveMathBench exhibits a reversed sensitivity pattern (Figure 23 , Middle). While we partition the data into leakage (wrong-then-right) and stable (always-right) subsets following the same protocol, the absence of actual contamination leads to different ablation behavior: the ”leakage” subset shows greater stability under layer manipulation, whereas the stable subset exhibits higher sensitivity. We attribute this inversion to distributional differences between the two subsets rather than memorization effects, confirming that the layer-specific circuits identified in contaminated datasets do not generalize to clean data.

(a) Overall Leakage Accuracy across experiments. (b) Overall Stable Accuracy across experiments. (c) Heatmap of leaked and stable samples.

## Appendix I Additional Results on Mechanistic Intervention

We conducted a comprehensive steering experiment across all 28 layers and their corresponding latent factors to validate our mechanistic hypothesis. The results demonstrate clear evidence for our proposed circuit: amplifying the factors at layer 18 yields substantial accuracy improvements, while ablating (zeroing out) the factors at layers 22 and 23 leads to significant performance degradation. These findings strongly corroborate our claim that layer 18 encodes critical features for the task, whereas layers 22 and 23 are responsible for integrating and processing these representations. The complete intervention results are visualized in Figure 25 .

## Appendix J Data Sample Examples

This section provides the complete input context and model outputs for case studies referenced in the main text and appendix.

### J.1 MATH-500 Question 42

### J.2 MATH-500 Question 65

### J.3 MATH-500 Question 332

### J.4 MATH-500 Question 77

### J.5 MATH-500 Question 141

### J.6 MATH-500 Question 19

### J.7 MATH-500 Question 213

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .