Spurious Rewards: Rethinking Training Signals in RLVR — part 1/6

Cleaned-text code points [0, 15470). Bibliography omitted here; source.md includes it.

# Spurious Rewards: Rethinking Training Signals in RLVR

###### Abstract

We show that reinforcement learning with verifiable rewards (RLVR) can elicit strong mathematical reasoning in certain language models even with spurious rewards that have little, no, or outright negative correlation with the correct answer. For example, RLVR training with GRPO improves MATH-500 performance for Qwen2.5-Math-7B in absolute points by 21.4% using randomly assigned rewards, nearly matching the 29.1% gained with ground truth rewards. To explain this counterintuitive observation, we show that GRPO exhibits a clipping bias arising from the clip term, which can amplify high-prior behaviors learned during pre-training even without informative rewards. As a case study, we identify one such high-prior behavior for Qwen2.5-Math models, which we term code reasoning —reasoning in code without actual code execution; code reasoning frequency increases from 65% to over 90% with spurious rewards. However, the presence of such amplifiable behaviors is highly model-dependent. In practice, spurious rewards that are effective for Qwen models often fail to produce gains for other model families, such as Llama3 or OLMo2. Our results highlight the importance of validating RL methods across diverse models rather than relying on a single de facto choice: large performance gains can arise on Qwen models even from random rewards that do not reflect genuine capability improvements.

Spurious_Rewards

## 1 Introduction

Reinforcement learning with verifiable rewards (RLVR) is highly effective in improving language model reasoning ( Lambert et al., 2024 ; Guo et al., 2025 ; Zeng et al., 2025 ; Luo et al., 2025 ) , but the mechanisms underlying these gains remain poorly understood. In this work, we show counterintuitively that RLVR can yield substantial performance improvements on math reasoning even under a family of spurious rewards that carry little to no task-relevant signal and that this effect depends critically on the base model. For example, on the Qwen2.5-Math models ( Yang et al., 2024a ; Yang et al., 2024b ) , which are widely used in the RLVR literature, randomly assigned rewards yield a 21.4% absolute accuracy gain on MATH-500, compared to a 29.1% gain from ground-truth rewards (§ 2 ). We observe similar trends on more challenging math benchmarks, including AMC and AIME. In contrast, for other model families, including Qwen2.5 ( Yang et al., 2024b ) , OLMo2 ( OLMo et al., 2024 ) , and Llama3 ( Dubey et al., 2024 ) , training with spurious rewards yields minimal improvement or even performance degradation (§ 3 ). This divergence in outcomes suggests that pretraining priors, different in each model, strongly shape RL training dynamics.

To further investigate the underlying mechanism contributing to such spurious improvements, we unpack the GRPO training objective ( Shao et al., 2024 ) and find that the clipping function systematically amplify tokens with high prior probabilities in the base model (§ 4 ). To help explain this discrepancy, we also analyze what reasoning patterns that RLVR is learning to favor in these cases (§ 5 ). In particular, we find a majority of Qwen2.5-Math-7B answers on MATH-500 contain reasoning chains expressed in Python—a behavior we call code reasoning —despite having no access to code execution. Code reasoning is highly predictive of overall performance; answers with it have an accuracy of 60.9%, much higher than without (28.0% accuracy). Code reasoning also correlates with MATH-500 accuracy over the course of RLVR training. Both metrics increase consistently during training with any spurious reward, leading to roughly 90 % 90\% or higher code frequency after training. Besides code reasoning, we find that lexical repetitions show similar but less substantial correlation with MATH performance in Qwen2.5-Math-7B models. Based on this observation, we further hypothesize that intervening with other methods that increase any beneficial behavior should similarly increase test performance. Our experiments validate this hypothesis: we design prompt-based and RL-based elicitation methods to increase code reasoning and lexical repetition; all such methods significantly increase Qwen2.5-Math-7B’s performance. We show that these results are robust to prompt variations, despite the initial model performance being sensitive to prompts in sometimes unexpected ways (Appendix I ).

Our findings not only open new questions but also have practical implications. We should generally be more aware that reasoning patterns instilled during pretraining heavily impact the behavior of downstream RLVR training. Qwen models, with open weights and high performance on reasoning tasks, have become the de facto choice for RLVR research in the open-source community—a range of recent research on RLVR drew conclusions on Qwen2.5-Math-7B-centric experiments ( Zuo et al., 2025 ; Zhao et al., 2025b ; Wang et al., 2024 ; Xie et al., 2025 ; Hu et al., 2025b ; Zhang et al., 2025 ; Shafayat et al., 2025 ; Prabhudesai et al., 2025 ; Gao et al., 2025 ; Wang et al., 2025a ) . However, we show that it is easy to get significant performance improvements on Qwen models even with completely spurious reward signals. Thus, we suggest that future RLVR research should be confirmed on other models and using spurious rewards as dummy baselines. We emphasize that spurious rewards are proposed purely for analytical purposes and should not be interpreted as a recommended approach for developing true model capabilities.

Ground Truth Majority Vote Format Incorrect Random

## 2 Spurious Rewards Yield Significant RLVR Gains

We design a progression of reward functions to replace the standard ground-truth reward: weak rewards (majority vote reward and format reward) and spurious rewards (random reward and incorrect reward). Remarkably, we find that all weak and spurious rewards suffice for RLVR to significantly improve the math performance of Qwen2.5-Math, a popular starting point for RLVR training.

### 2.1 Experimental Setup

Following recent RLVR work ( Wang et al., 2025b ; Zuo et al., 2025 ; Zeng et al., 2025 ) , we use GRPO ( Guo et al., 2025 ) to finetune Qwen2.5-Math models ( Yang et al., 2024a ) . The standard RLVR approach uses a dataset of questions paired with ground truth labels. During training, model rollouts are given a binary (0-1) reward based on whether the generated answer is verifiably correct. We replace this ground truth-based reward with a variety of increasingly spurious binary 0-1 reward functions that do not require access to ground truth labels. We design these alternative rewards to investigate the limits of how little supervision is needed for effective RLVR training. We train on DeepScaleR data ( Luo et al., 2025 ) with our rewards; all other experimental details are kept constant.

In the main paper, we evaluate performance as pass@1 and average@8 accuracy on two standard math reasoning benchmarks: MATH-500 ( Hendrycks et al., 2021 ) and AMC ( Li et al., 2024 ) , respectively. See Appendix D for additional results on AIME 2024 and 2025. Following the default evaluation setup in the popular RL framework OpenRLHF ( Hu et al., 2025a ) , we use the default chat template for Qwen2.5-Math and all instruct models and chat template provided by Olmo ( OLMo et al., 2024 ) for other base models in our main experiments. Additional analysis on the effect of different prompts can be found in Appendix I , where we show Qwen2.5-Math-7B is very sensitive to prompts—even a task-irrelevant prompt (which we name spurious prompt ) can sometimes result in high initial performance. See Appendix A for full details of our training and evaluation setup.

### 2.2 Standard to Weak to Spurious Rewards

We consider the following rewards:

1. Ground Truth Rewards: To establish a baseline, we consider the standard RLVR approach ( Lambert et al., 2024 ) of using ground truth labels to reward responses with verifiably correct answers. This setting serves as an upper bound for reward supervision quality.

2. Majority Vote Rewards: Instead of using ground truth labels for computing rewards, we use the model prior to RLVR training to pseudo-label the training set by selecting the majority answer from 64 sampled responses per prompt. These (potentially wrong) labels are then used to reward responses during standard online RLVR training.

3. Format Rewards: We further weaken the reward signal to disregard the responses’ mathematical correctness altogether. We instead heuristically reward all responses containing at least one non-empty \boxed{} expression, regardless of the correctness of the enclosed answer. Including \boxed{} is specified in Qwen2.5-Math’s system prompt; this reward incentivizes some degree of prompt following.

4. Random Rewards: We study whether providing no guidance in the rewarding process is sufficient to provide meaningful math performance gains. To do so, we assign rewards randomly. Given a fixed probability hyperparameter γ \gamma , all responses receive a reward of 1 1 with chance indicated by the parameter, and receive 0 0 otherwise. In our main experiments, we present γ = 0.5 \gamma=0.5 ; in Appendix B , we show that using γ ∈ { 0.001 , 0.3 , 0.7 } \gamma\in\{0.001,0.3,0.7\} obtains similar improvements with varying convergence speed, and verify that γ = 0 \gamma=0 results in no change as expected analytically (with γ = 0 \gamma=0 , loss is constant, and all gradients are zero).

5. (Majority-voted) Incorrect Rewards: We furthermore deliberately provide incorrect supervision and reward only incorrect answers. We first label all training data using majority voting and select the subset with incorrect labels for training, obtaining incorrect labels that are still probable outputs of the models. During training, we reward responses whose answers verifiably match these incorrect labels. This design disentangles whether majority vote rewards work because labels are more likely correct or because they represent high-probability model outputs. We reward specific incorrect answers rather than any incorrect answer to maintain comparable reward sparsity with our other conditions.

Note: We emphasize that spurious rewards—particularly random and incorrect rewards—are proposed purely for analytical purposes and should not be interpreted as a recommended approach for developing true model capabilities.

Figure 2 presents the performance of Qwen2.5-Math models after RLVR training with each reward function. Overall, all reward functions, even pathologically designed ones, lead to significant improvements in math performance within the first 50 steps across all benchmarks compared to the untuned baseline. One exception is that Qwen2.5-Math-1.5B sees gains with random rewards much slower (after 100 steps) and less so on AMC (only 4.9%). Remarkably, performance gains from spurious rewards are often within a few points of the gain from RLVR with ground truth labels. For example, training with incorrect label reward yields 24.1% gains over Qwen2.5-Math-7B on MATH-500, compared to a 29.1% gain from RLVR with ground truth answers. Even random rewards—which by design provide pure noise in the rewarding process —still produce a 21.4% performance boost. We observe similar trends in AMC, where training on format, incorrect, or random rewards yields a gain of 13.8 % 13.8\% , 24.1 % 24.1\% , or 21.4 % 21.4\% , respectively, approaching the ∼ 27 % \sim 27\% – 29 % 29\% improvement gained from training on majority voted and ground truth labels.

In Appendix D , we supplement results on AIME24 and AIME25. On AIME2024, format reward (+10.3%) approaches ground truth rewards (+15.3%), and spurious rewards (incorrect & random) still lead to high performance gains of 10.2% and 10.2% respectively on Qwen2.5-Math-7B. Ground truth labels show a clear advantage compared to other rewards on AIME2025, which contains questions written after the knowledge cutoff of all models we consider. Nonetheless, other rewards still lead to a -0.4% to 4.5% gain in performance.

Our findings with these simple rewards provide additional evidence for a nascent hypothesis in the literature: that RLVR, at least at the compute scales of open-source post-training pipelines ( Lambert et al., 2024 ) , does not teach models new reasoning capabilities, but instead triggers latent ones already present in the base model ( Wang et al., 2025b ; Liu et al., 2025 ; Gandhi et al., 2025 ; Yue et al., 2025 ; AI et al., 2025 ; Choshen et al., 2020 ) . Whereas prior work has hinted at this effect using limited-quantity ground-truth labels ( Wang et al., 2024 ) or noisy labels ( Zuo et al., 2025 ) , our results push this idea to its limit: we show that even outright incorrect rewards or information-free rewards (i.e., random) can elicit performance gains in Qwen2.5-Math models. In the remainder of our paper, we show that this elicitation effect is model-dependent (§ 3 ), and trace the specific properties of Qwen2.5-Math models that could enable spurious rewards to induce this elicitation (§ 5 ).

Ground Truth Majority Vote Format Incorrect Random

## 3 (Lack of) Generalization to Other Models

Inspired by the unexpected effectiveness of spurious reward signals in improving the performance of Qwen2.5-Math models, we study whether these rewards generalize to training other models. We extend beyond the math-specialized Qwen2.5-Math models to include general-purpose variants (Qwen2.5-7B, Qwen2.5-1.5B ( Yang et al., 2024b ) ), and two additional model families: (a) the widely-used Llama3.1-8B(-Instruct) and Llama3.2-3B(-Instruct) ( Dubey et al., 2024 ) , and (b) OLMo2-7B and OLMo2-7B-SFT ( OLMo et al., 2024 ) . OLMo2-7B-SFT is instruction-tuned from OLMo2-7B; we include both to better understand the impact of SFT training on RLVR training. Moreover, we are optimistic that OLMo’s open training data will enable future works to better study the origins of any reasoning behaviors (or lack thereof) we later observe. We train these 8 additional models with the same setup and rewards as in Section 2 . We report performance on MATH-500 in the main text of the paper and AMC in Appendix C where the same trends are observed. The AIME 2024 and 2025 benchmarks show similar but noisy trends (Appendix D ).

Spurious rewards can benefit Qwen2.5 models, but rarely help non-Qwen models. Figure 3 shows two trends. First, models within a family behave similarly: across Qwen2.5, all non-random rewards (even spurious incorrect) improve MATH-500, whereas OLMo stays flat under spurious rewards and gains mainly with ground-truth rewards. We conjecture this family-level consistency reflects shared pretraining distributions that induce similar pre-RL behaviors. Second, smaller models benefit less from spurious (e.g., random) rewards; we conjecture larger models preserve more pretraining priors that these rewards amplify. Section 5 further analyzes how pre-existing behaviors shape RLVR outcomes. Importantly, these reward signals do not reliably transfer across families. Although spurious incorrect and other weak rewards consistently improve Qwen, each weak or spurious reward fails to help at least one other model and can be flat or even harmful. Overall, Qwen appears unusually robust to reward strength. Appendix J further shows that models already RL post-trained see minimal gains under nearly all rewards.

Practical warning