Spurious Rewards: Rethinking Training Signals in RLVR

arxiv:2506.10947

Original source · Paper JSON

Reasoning and the "physics" of language models · 2025-06-12

Cites (10) · Cited by (7)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• Spurious Rewards: Rethinking Training Signals in RLVR
• Abstract
• 1 Introduction
• 2 Spurious Rewards Yield Significant RLVR Gains
• 2.1 Experimental Setup
• 2.2 Standard to Weak to Spurious Rewards
• 3 (Lack of) Generalization to Other Models
• 4 Understanding Training Signals from Random Rewards in GRPO
• 5 Case Study: Code Reasoning as a Representative Pre-Existing Behavior
• 5.1 Different Models Exhibit Pre-existing Reasoning Strategy Discrepancies
• 5.2 RLVR with Spurious Rewards Upweight Pre-existing Reasoning Strategies
• 5.3 Intervening Explicitly on Code Reasoning Frequency
• 6 Related Work
• 7 Conclusion
• Acknowledgments
• Appendix
• Appendix A Experimental Setup
• A.1 Preliminaries on GRPO
• A.2 Datasets and Models
• A.3 Training Configuration
• A.4 Decoding Configuration
• A.5 Computation Resource
• A.6 Visualization
• A.7 Experimental Setups for TTRL and One-Shot RL
• TTRL setup.
• Differences in our One-Shot RL setup from ( Wang et al., 2025b ) .
• Appendix B The Curious Case of Random Rewards
• Random rewards with varying probabilities consistently improve performance.
• Recap of notations.
• B.1 Conjecture: Clipping Bias Brings Training Signals under Random Rewards
• B.1.1 Expected Advantage of Random Rewards
• B.1.2 Gradient Derivation of Clipping Bias under Random Rewards
• Case 1: A ^ ≥ 0 \hat{A}\geq 0 .
• Case 2: A ^ < 0 \hat{A}<0 .
• Combining the cases.
• B.1.3 Clipping Creates Asymmetric Updates Towards Model Prior Knowledge
• Empirical validation.
• Why only Qwen-Math models benefit.
• Implications.
• B.2 High Stochasticity in RLVR Without Clipping
• Appendix C Additional Results on AMC
• Appendix D Additional Results on AIME 2024 and AIME 2025
• D.1 Spurious Rewards Yield Significant RLVR Gains on Qwen2.5-Math
• D.2 (Lack of) Generalization to Other Models
• Appendix E Examination of Existing Methods on Models Beyond Qwen
• Appendix F Compound Rewards that Inhibit Code Reasoning
• Inhibiting code reasoning during RLVR with spurious rewards can reduce gains on Qwen2.5-Math-7B, but increase gains on other models.
• Appendix G Beyond Code Reasoning: Another Beneficial Pattern that RLVR Can Easily Elicit
• Appendix H Switch of Reasoning Strategies during RLVR
• Reasoning strategy switches during RLVR.
• Frequency:
• Appendix I Spurious Prompts: Qwen2.5-Math-7B’s Unreasonably High Sensitivity to Prompts
• I.1 Existing Prompts
• I.2 Spurious Prompts
• Appendix J Additional Results on Models That Have Undergone RL Training
• Appendix K Qualitative Analysis on Qwen2.5-Math-7B’s Coding Reasoning Behaviors
• Qwen2.5-Math-7B can reason in code.
• Qwen2.5-Math-7B’s code reasoning behavior is robust to numerical perturbations.
• Qwen2.5-Math-7B’s code reasoning behavior may not directly generalize to rephrased questions.
• Instructions for reporting errors