Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

arxiv:2601.11061

Original source · Paper JSON

Post-training · 2026-01-16

Cites (4) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs
• Abstract
• Keywords:
• 1 Introduction
• 2 Related Work
• 3 Preliminaries
• 4 Empirical Study
• 4.1 Experiment Setting
• 4.2 What Does Perplexity Show
• 4.3 Locating the Functional Anchor: Static Analysis
• 4.4 Tracking the Bifurcation: Dynamic Analysis
• 5 Ablation Study
• 6 Mechanistic Intervention
• 6.1 Task-Relevant Neuron Identification
• 6.2 Causal Steering
• 6.3 Results
• 6.4 Probe-Gated Shortcut Audit
• 7 Conclusion
• Acknowledgments
• Impact Statement
• Appendix A Code
• Appendix B Preliminaries
• Appendix C Partial Prompt Evaluation Details
• Appendix D Complete Perplexity Analysis
• Appendix E Additional Path Patching Results
• Appendix F Additional JSD Results
• Appendix G Additional Results on Logit Lens
• Appendix H Additional Results on Ablation Study
• Appendix I Additional Results on Mechanistic Intervention
• Appendix J Data Sample Examples
• J.1 MATH-500 Question 42
• J.2 MATH-500 Question 65
• J.3 MATH-500 Question 332
• J.4 MATH-500 Question 77
• J.5 MATH-500 Question 141
• J.6 MATH-500 Question 19
• J.7 MATH-500 Question 213
• Instructions for reporting errors