Spurious Rewards: Rethinking Training Signals in RLVR — part 2/6

Cleaned-text code points [15470, 31070). Bibliography omitted here; source.md includes it.

: Proposed RLVR reward signals should be tested on diverse models! Many recent methods on RLVR for reasoning draw their conclusions primarily or exclusively from gains shown on Qwen models (non-exhaustively, ( Zuo et al., 2025 ; Zhao et al., 2025b ; Wang et al., 2024 ; Xie et al., 2025 ; Hu et al., 2025b ; Zhang et al., 2025 ) ). As a case study, we experimented with recent work on (1) test time training ( Zuo et al., 2025 ) and (2) one-shot RL ( Wang et al., 2025b ) (setup details in Appendix A.7 ). We find these methods exhibit a pattern similar to our results above: the proposed training signals yield strong improvements in the Qwen2.5-Math or Qwen2.5 models, matching the performance of the ground truth rewards (Figure 15 in Appendix E ). However, these same signals often fail to yield performance gains on other model families. Our findings suggest that existing Qwen-centric RLVR research should possibly be further validated on non-Qwen models.

## 4 Understanding Training Signals from Random Rewards in GRPO

Prior work has shown that RLVR using majority-voted labels or format rewards can improve model performance in GRPO ( Zuo et al., 2025 ; Zhao et al., 2025b ; Wang et al., 2025b ) . As a result, the effectiveness of majority-voted (even when incorrect 1 1 1 We hypothesize that incorrect rewards provide training signals because many remain close to ground truth and because they implicitly act as format rewards, incentivizing extractable answers. ) rewards and format rewards is less surprising. However, the effectiveness of random rewards remains puzzling, as they are constructed to contain no information. In this section, we present our hypotheses on how random rewards induce meaningful training signals in GRPO and validate them through ablation studies.

No training signal from random rewards when clipping is removed from GRPO. We analyze GRPO with the clipping term removed and show that random rewards alone do not yield effective training. The clipping term appears in the GRPO objective 2 2 2 We omit the KL regularization term, which is disabled in our experiments. : J ⁡ ( θ ) = \displaystyle J(\theta)= 𝔼 x ∼ 𝒟 , y ∼ π old ( ⋅ ∣ x ) [ ∑ t = 1 | y | min ( ρ t ( y ; θ ) A ^ ( x , y ) , \displaystyle\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_{\text{old}}(\cdot\mid x)}\left[\sum_{t=1}^{|y|}\min\Bigl(\rho_{t}(y;\theta)\,\hat{A}(x,y),\;\right. (1) clip ( ρ t ( y ; θ ) , 1 − ϵ c , 1 + ϵ c ) A ^ ( x , y ) ) ] , \displaystyle\left.{\color[rgb]{1,0,0}\operatorname{clip}\bigl(\rho_{t}(y;\theta),1-\epsilon_{c},1+\epsilon_{c}\bigr)\hat{A}(x,y)}\Bigr)\right], where x x denotes the input, y y the generated sequence, π old \pi_{\text{old}} the behavior policy, ρ t ​ ( y , θ ) = π θ ​ ( y t ∣ x , y < t ) / π old ​ ( y t ∣ x , y < t ) \rho_{t}(y;\theta)=\pi_{\theta}(y_{t}\mid x,y_{<t})/\pi_{\text{old}}(y_{t}\mid x,y_{<t}) the importance ratio at step t t (corresponding to the t t -th generated token), and ϵ c \epsilon_{c} the clipping threshold. The advantage A ^ ​ ( x , y ) \hat{A}(x,y) is computed from the reward r ⁡ ( x , y ) r(x,y) by centering it within each prompt, A ^ ( x , y ) = r ( x , y ) − 𝔼 y ′ ∼ π old ( ⋅ ∣ x ) [ r ( x , y ′ ) ] \hat{A}(x,y)=r(x,y)-\mathbb{E}_{y^{\prime}\sim\pi_{\text{old}}(\cdot\mid x)}[r(x,y^{\prime})] . When clipping (the red term in Eq. 1 ) is removed and rewards are sampled independently of the responses, the expected training objective is zero, yielding no meaningful learning signal. To demonstrate this, we consider three separate no-clipping variants of GRPO: disabling clipping in the implementation, increasing the mini-batch size to match the rollout size, or reducing the rollout size so that only a single gradient update is performed per rollout. 3 3 3 The latter two variants enforce π θ = π old \pi_{\theta}=\pi_{\text{old}} during each rollout, yielding ρ t = 1 \rho_{t}=1 and thus eliminating the effect of clipping by construction. As shown in the first three panels of Figure 4 (no-clipping variants), random rewards fail to yield consistent performance. In contrast, enabling clipping (bottom-right) leads to stable and consistent improvements.

Random w/ Clipping Enabled Random w/ Clipping Disabled Random w/ Clipping Disabled (Mini-Batch Size ↑ \uparrow ) Random w/ Clipping Disabled (Rollout Batch Size ↓ \downarrow )

GRPO clipping bias can induce random reward training signals. We analyze the effect of clipping in GRPO under random rewards, with the full derivation provided in Appendix B . At a high level, clipping introduces a bias in the gradient update that depends on how the current policy’s token probabilities deviate from those of the behavior policy. Factoring out reward-independent scalar terms, the expected gradient induced by random rewards can be written (up to a positive constant) as 𝔼 ⁡ [ ∇ θ J ​ ( θ ) ] ∝ 𝔼 x , y ​ [ { ∇ θ ​ log ​ π θ , x ​ ( y t ) , R θ < 1 − ϵ c , 0 , | R θ − 1 | ≤ ϵ c , − ∇ θ ​ log ​ π θ , x ​ ( y t ) , R θ > 1 + ϵ c , ] , \small\mathbb{E}[\nabla_{\theta}J(\theta)]\propto\mathbb{E}_{x,y}\!\left[\begin{cases}\;\;\nabla_{\theta}\log\pi_{\theta,x}(y_{t}),&R_{\theta}<1-\epsilon_{c},\\ \;0,&|R_{\theta}-1|\leq\epsilon_{c},\\ -\nabla_{\theta}\log\pi_{\theta,x}(y_{t}),&R_{\theta}>1+\epsilon_{c},\end{cases}\right], (2) where R θ = π θ , x ​ ( y t ) / π old , x ​ ( y t ) R_{\theta}=\pi_{\theta,x}(y_{t})/\pi_{\text{old},x}(y_{t}) . This form shows that clipping increases the probability of tokens that are already likely under the policy, while suppressing overly large deviations, even when rewards are uninformative. We illustrate this effect with the following example:

In Appendix B , we show that this effect persists across different random reward distributions, indicating its stability. Our findings echo the intuition of Yu et al. (2025) , who show that clipping bias reduces exploration and increases exploitation in RLVR with ground-truth labels.

## 5 Case Study: Code Reasoning as a Representative Pre-Existing Behavior

Previously, we showed that different models can exhibit substantially different outcomes when trained with the same weak or spurious reward signals. Here, we propose a general hypothesis for these discrepancies and examine it through a focused case study of a representative pre-existing behavior observed in Qwen models. Our central hypothesis is that differences in RLVR outcomes stem from differences in the reasoning strategies acquired during pretraining. Some strategies may be readily amplified by RLVR, while others may be difficult to elicit or absent altogether, leading to divergent training dynamics across model families.

We study one such strategy—using code to support mathematical reasoning—which Qwen-Math models employ effectively, while other model families do so to a lesser extent (§ 5.1 ). By tracing the prevalence of code reasoning over the course of RLVR training, we find evidence consistent with this hypothesis (§ 5.2 ). We note that we use code reasoning as a controlled and observable example to study how RLVR interacts with pre-existing internal strategies, rather than as an exhaustive characterization of model behavior.

### 5.1 Different Models Exhibit Pre-existing Reasoning Strategy Discrepancies

We analyze reasoning traces from different models on MATH-500 and observe clear discrepancies in pre-existing strategies. Qwen2.5-Math-7B frequently generates Python code in its reasoning (65.0% of responses), despite lacking code execution; we refer to this behavior as code reasoning .

Crucially, code reasoning in Qwen2.5-Math-7B is strongly correlated with correctness. As shown in Table 5.2 , responses containing code achieve substantially higher accuracy than those without. Qwen2.5-Math-1.5B exhibits similar behavior, while other models do not. We categorize these models as either No-Code (e.g., Llama, Qwen2.5-1.5B, OLMo2-7B), which do not generate code, or Bad-Code (e.g., OLMo2-7B-SFT, Qwen2.5-7B), which frequently generate code but with degraded performance. Thus, effective code reasoning appears to be a distinctive pre-existing capability of the Qwen2.5-Math prior to RLVR.

We provide further analysis that hints at the origins of code reasoning of Qwen2.5-Math-7B models in Appendix K . Note that code reasoning is not used as a complete explanation: other behaviors can also be elicited easily and often correlate with performance—we briefly discuss another such behavior, generation without repetition, in Appendix G . In addition, we find the initial performance of Qwen2.5-Math-7B is sensitive to the prompts used for evaluation; we supplement a discussion on this impact in Appendix I .

### 5.2 RLVR with Spurious Rewards Upweight Pre-existing Reasoning Strategies

Motivated by our observations above, we traced changes in the reasoning behavior of models throughout the course of RLVR training across two dimensions: (1) Accuracy : the average accuracy of the model on MATH-500, and (2) Code reasoning frequency : the percentage of model responses containing the string “ python ”. We find that rewards that we employed in the paper, including spurious rewards such as the random and incorrect rewards, gain much of the accuracy on the Qwen2.5-Math and Qwen2.5 through eliciting the correct reasoning strategy.

Ground Truth Majority Vote Format Incorrect Random

Performance correlates with code reasoning frequency. As shown in Figure 6 , prior to RLVR training, Qwen2.5-Math-7B uses code reasoning in 65.0% of MATH-500 solutions. Under RLVR with weak or spurious rewards, code frequency rapidly increases (to ∼ \sim 90% within 15 steps and up to 95.6% for random rewards), closely tracking accuracy gains. With ground-truth rewards, code frequency also rises initially but later declines as natural-language reasoning improves, suggesting learning beyond reliance on code. For Bad-Code models, performance gains instead correlate with reduced code usage. Additional analysis of these strategy shifts is provided in Appendix H , indicating that gains from spurious rewards largely arise when models transition from text-based to code-based reasoning.

### 5.3 Intervening Explicitly on Code Reasoning Frequency

We have shown observationally that code reasoning frequency increases during RLVR and correlates with improved test performance. Here, we study the effect of explicitly inducing or suppressing code reasoning. We present results for inducing additional code reasoning in the main text, while experiments suppressing code reasoning are deferred to Appendix F . Across both directions, these interventions produce outcomes consistent with our hypothesis.

Inducing code reasoning improves Qwen2.5-Math models but degrades others. We induce code reasoning via (1) prompting and (2) RLVR. With prompting (forcing responses to begin with “Let’s solve this using Python.” ), MATH-500 accuracy improves by 24.2%, 15.0%, and 10.0% for Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen2.5-1.5B, respectively (Table 2 ), while performance degrades for other models such as Llama and OLMo2—consistent with their lack of effective code reasoning (Section 5.1 ).

To induce code reasoning with RLVR, we assign a positive reward if and only if a response contains the string “ python ” (a Python reward ). This rapidly increases code usage in Qwen2.5-Math-7B to over 99% within 20 training steps (Figure 7 ) and yields performance gains primarily in the Qwen2.5-Math. For these models, RLVR-based induction matches or exceeds the gains achieved through prompting, while other models show limited improvement.

Qwen-Math-7B Qwen-Math-1.5B Qwen-7B Qwen-1.5B Olmo2-7B-SFT Olmo2-7B Llama3.1-8B Llama3.2-3B Llama3.1-8B-Instruct Llama3.2-3B-Instruct

## 6 Related Work

Unsupervised Reinforcement Learning. Several approaches have explored unsupervised reinforcement learning. Prasad et al. (2024) introduced Self-Consistency Preference Optimization (ScPO), which trains models to prefer consistent answers over inconsistent ones on unsupervised problems. Similarly, Test-Time Reinforcement Learning (TTRL) ( Zuo et al., 2025 ) leverages majority voting across sampled outputs to estimate pseudo-rewards, demonstrating significant performance improvements on mathematical reasoning tasks. EMPO ( Zhang et al., 2025 ) adapts PPO ( Schulman et al., 2017 ) or GRPO ( Shao et al., 2024 ) for unsupervised RLVR by calculating the rewards based on minimizing the entropy of queries in the semantic space. These approaches suggest that internal model consistency can serve as an effective proxy for correctness, though they do not systematically investigate how different kinds of training rewards affect various model families during RLVR.

Reinforcement Learning for Language Models. The development of language model capabilities has been significantly advanced through reinforcement learning approaches. RLHF has become a standard technique for aligning models with human preferences ( Ouyang et al., 2022 ; Bai et al., 2022 ) , while RLVR has proven effective for tasks with deterministic answers ( Guo et al., 2025 ; Gao et al., 2024 ; Wen et al., 2025 ; Lambert et al., 2024 ; Luo et al., 2025 ) . These methods traditionally rely on accurate supervision signals, either through human feedback or verifiable rewards. Recent work has explored reducing the dependence on human annotations through AI feedback mechanisms ( Bai et al., 2022 ) and through training dynamics analysis ( Zhao et al., 2025a ) , which supports the finding that RL primarily amplifies behaviors or capabilities already buried in the pretrained models ( Liu et al., 2025 ; Yue et al., 2025 ) .

## 7 Conclusion

Our findings have three main implications: base model pretraining significantly affects RLVR outcomes; even corrupted or spurious supervision can enhance reasoning when it triggers useful existing behaviors; and effects observed in one model family may not generalize to others. Our work highlights the importance of (1) testing across multiple models with differing pretraining distributions, and (2) testing across multiple different baselines, such as format and random rewards, when evaluating reinforcement learning techniques.

This work does not advocate the use of spurious rewards as a training strategy for improving model capabilities. Instead, spurious rewards are introduced solely for controlled analysis, as their misuse in deployed systems could lead to unreliable or undesirable behavior. Overall, the anticipated impact of this work is to improve the rigor and robustness of reinforcement learning research rather than to enable new applications.

## Acknowledgments

We thank Hamish Ivison and Trung Vu for insightful discussions. This research was developed in part with funding from the Defense Advanced Research Projects Agency’s (DARPA) SciFy program (Agreement No. HR00112520300). The views expressed are those of the author and do not reflect the official policy or position of the Department of Defense or the U.S. Government. This work was supported by the Singapore National Research Foundation and the National AI Group in the Singapore Ministry of Digital Development and Information under the AI Visiting Professorship Programme (award number AIVP-2024-001), and by the AI2050 program at Schmidt Sciences. This work was supported by NSF grants 2112471 and 2134012. SG is supported by the NSF GRFP. SSD acknowledges the support of NSF IIS-2110170, NSF DMS-2134106, NSF CCF-2212261, NSF IIS-2143493, NSF CCF-2019844, NSF IIS-2229881, and the Sloan Research Fellowship.

## Appendix

## Appendix A Experimental Setup

### A.1 Preliminaries on GRPO