Spurious Rewards: Rethinking Training Signals in RLVR — part 4/6

Cleaned-text code points [46726, 62555). Bibliography omitted here; source.md includes it.

Applying a similar analysis, when A ^ < 0 \hat{A}<0 , the gradient with respect to θ \theta for this case is: ∇ θ L − ​ ( θ ) = A ^ ⋅ { 0 , if ​ R θ < 1 − ϵ c , ∇ θ R θ , if ​ R θ > 1 − ϵ c . \nabla_{\theta}L^{-}(\theta)=\hat{A}\cdot\begin{cases}0,&\text{if }R_{\theta}<1-\epsilon_{c},\\ \nabla_{\theta}R_{\theta},&\text{if }R_{\theta}>1-\epsilon_{c}.\end{cases}

##### Combining the cases.

Together, the gradient is: ∇ θ J ​ ( θ ) = A ^ ⋅ { ∇ θ R θ , if ​ A ^ ≥ 0 ​ and ​ R θ < 1 + ϵ c , 0 , if ​ A ^ ≥ 0 ​ and ​ R θ > 1 + ϵ c , 0 , if ​ A ^ < 0 ​ and ​ R θ < 1 − ϵ c , ∇ θ R θ , if ​ A ^ < 0 ​ and ​ R θ > 1 − ϵ c . \nabla_{\theta}J(\theta)=\hat{A}\cdot\begin{cases}\nabla_{\theta}R_{\theta},&\text{if }\hat{A}\geq 0\text{ and }R_{\theta}<1+\epsilon_{c},\\ 0,&\text{if }\hat{A}\geq 0\text{ and }R_{\theta}>1+\epsilon_{c},\\ 0,&\text{if }\hat{A}<0\text{ and }R_{\theta}<1-\epsilon_{c},\\ \nabla_{\theta}R_{\theta},&\text{if }\hat{A}<0\text{ and }R_{\theta}>1-\epsilon_{c}.\end{cases}

Then, the clipping bias, 𝙱𝚒𝚊𝚜 ​ ( ∇ θ J ​ ( θ ) ) \mathtt{Bias}(\nabla_{\theta}J(\theta)) , is 𝙱𝚒𝚊𝚜 ​ ( ∇ θ J ​ ( θ ) ) \displaystyle\mathtt{Bias}(\nabla_{\theta}J(\theta)) = \displaystyle= 𝔼 A ^ , x , y ​ [ ∇ θ J ​ ( θ ) ] \displaystyle\mathbb{E}_{\hat{A},x,y}\left[\nabla_{\theta}J(\theta)\right] = \displaystyle= P ⁡ ( A ^ ≥ 0 ) ⋅ 𝔼 A ^ > 0 , x , y ​ [ ∇ θ L + ​ ( θ ) ] + P ⁡ ( A ^ < 0 ) ⋅ 𝔼 A ^ < 0 , x , y ​ [ ∇ θ L − ​ ( θ ) ] + 0 \displaystyle P(\hat{A}\geq 0)\cdot\mathbb{E}_{\hat{A}>0,x,y}[\nabla_{\theta}L^{+}(\theta)]+P(\hat{A}<0)\cdot\mathbb{E}_{\hat{A}<0,x,y}[\nabla_{\theta}L^{-}(\theta)]+0 = \displaystyle= 𝔼 x , y ​ [ { P ⁡ ( A ^ ≥ 0 ) ⋅ 𝔼 A ^ ≥ 0 ​ [ A ^ ] ⋅ ∇ θ R θ , if ​ R θ < 1 + ϵ c , 0 , if ​ R θ > 1 + ϵ c . ] \displaystyle\mathbb{E}_{x,y}\left[\begin{cases}P(\hat{A}\geq 0)\cdot\mathbb{\mathbb{E}}_{\hat{A}\geq 0}[\hat{A}]\cdot\nabla_{\theta}R_{\theta},&\text{if }R_{\theta}<1+\epsilon_{c},\\ 0,&\text{if }R_{\theta}>1+\epsilon_{c}.\end{cases}\right] + 𝔼 x , y ​ [ { 0 , if ​ R θ < 1 − ϵ c , P ⁡ ( A ^ < 0 ) ⋅ 𝔼 A ^ < 0 ​ [ A ^ ] ⋅ ∇ θ R θ , if ​ R θ > 1 − ϵ c . ] \displaystyle+\mathbb{E}_{x,y}\left[\begin{cases}0,&\text{if }R_{\theta}<1-\epsilon_{c},\\ P(\hat{A}<0)\cdot\mathbb{\mathbb{E}}_{\hat{A}<0}[\hat{A}]\cdot\nabla_{\theta}R_{\theta},&\text{if }R_{\theta}>1-\epsilon_{c}.\end{cases}\right] = \displaystyle= 𝔼 x , y ​ [ { P ⁡ ( A ^ ≥ 0 ) ⋅ 𝔼 A ^ ≥ 0 ​ [ A ^ ] ⋅ ∇ θ R θ , if ​ R θ < 1 − ϵ c , ( P ⁡ ( A ^ ≥ 0 ) ⋅ 𝔼 A ^ ≥ 0 ​ [ A ^ ] + P ⁡ ( A ^ < 0 ) ⋅ 𝔼 A ^ < 0 ​ [ A ^ ] ) ⋅ ∇ θ R θ , if ​ 1 − ϵ c < R θ < 1 + ϵ c , P ⁡ ( A ^ < 0 ) ⋅ 𝔼 A ^ < 0 ​ [ A ^ ] ⋅ ∇ θ R θ , if ​ R θ > 1 + ϵ c . ] \displaystyle\mathbb{E}_{x,y}\left[\begin{cases}P(\hat{A}\geq 0)\cdot\mathbb{\mathbb{E}}_{\hat{A}\geq 0}[\hat{A}]\cdot\nabla_{\theta}R_{\theta},&\text{if }R_{\theta}<1-\epsilon_{c},\\ (P(\hat{A}\geq 0)\cdot\mathbb{\mathbb{E}}_{\hat{A}\geq 0}[\hat{A}]+P(\hat{A}<0)\cdot\mathbb{\mathbb{E}}_{\hat{A}<0}[\hat{A}])\cdot\nabla_{\theta}R_{\theta},&\text{if }1-\epsilon_{c}<R_{\theta}<1+\epsilon_{c},\\ P(\hat{A}<0)\cdot\mathbb{\mathbb{E}}_{\hat{A}<0}[\hat{A}]\cdot\nabla_{\theta}R_{\theta},&\text{if }R_{\theta}>1+\epsilon_{c}.\end{cases}\right]

By definition, A ^ \hat{A} is a normalized distribution and 𝔼 ⁡ ( A ^ ) = 0 \mathbb{E}(\hat{A})=0 from Appendix B.1.1 , so 𝔼 ⁡ ( A ^ ) = P ⁡ ( A ^ ≥ 0 ) ​ 𝔼 A ^ ≥ 0 ​ ( A ^ ) + P ⁡ ( A ^ < 0 ) ​ 𝔼 A ^ < 0 ​ ( A ^ ) = 0 , and \mathbb{E}(\hat{A})=P(\hat{A}\geq 0)\,\mathbb{E}_{\hat{A}\geq 0}(\hat{A})+P(\hat{A}<0)\,\mathbb{E}_{\hat{A}<0}(\hat{A})=0,\text{ and} P ⁡ ( A ^ ≥ 0 ) ​ 𝔼 A ^ ≥ 0 ​ ( A ^ ) = − P ⁡ ( A ^ < 0 ) ​ 𝔼 A ^ < 0 ​ ( A ^ ) ≥ 0 . P(\hat{A}\geq 0)\,\mathbb{E}_{\hat{A}\geq 0}(\hat{A})=-P(\hat{A}<0)\,\mathbb{E}_{\hat{A}<0}(\hat{A})\geq 0. We denote μ = P ⁡ ( A ^ ≥ 0 ) ​ 𝔼 A ^ ≥ 0 ​ ( A ^ ) = − P ⁡ ( A ^ < 0 ) ​ 𝔼 A ^ < 0 ​ ( A ^ ) \mu=P(\hat{A}\geq 0)\,\mathbb{E}_{\hat{A}\geq 0}(\hat{A})=-P(\hat{A}<0)\,\mathbb{E}_{\hat{A}<0}(\hat{A}) , which by definition is positive. And we substitute R θ = π θ , x ​ ( y t ) π old , x ​ ( y t ) R_{\theta}=\frac{\pi_{\theta,x}(y_{t})}{\pi_{\text{old},x}(y_{t})} in the conditions. Finally, we set the gradients on non-differentiable points to 0 and have 𝙱𝚒𝚊𝚜 ⁡ ( ∇ θ J ​ ( θ ) ) = μ ⋅ 𝔼 x , y ​ [ { ∇ θ R θ , if ​ π θ , x ​ ( y t ) < π old , x ​ ( y t ) ⋅ ( 1 − ϵ c ) , 0, if ​ π old , x ​ ( y t ) ⋅ ( 1 − ϵ c ) ≤ π θ , x ​ ( y t ) ≤ π old , x ​ ( y t ) ⋅ ( 1 + ϵ c ) , − ∇ θ R θ , if ​ π θ , x ​ ( y t ) > π old , x ​ ( y t ) ⋅ ( 1 + ϵ c ) . ] \displaystyle\small\mathtt{Bias}(\nabla_{\theta}J(\theta))=\mu\cdot\mathbb{E}_{x,y}\left[\begin{cases}\nabla_{\theta}R_{\theta},&\text{if }\pi_{\theta,x}(y_{t})<\pi_{\text{old},x}(y_{t})\cdot(1-\epsilon_{c}),\\ \hbox{\multirowsetup 0,}&\text{if }\pi_{\text{old},x}(y_{t})\cdot(1-\epsilon_{c})\leq\pi_{\theta,x}(y_{t})\\ &\hskip 9.24994pt\leq\pi_{\text{old},x}(y_{t})\cdot(1+\epsilon_{c}),\\ -\nabla_{\theta}R_{\theta},&\text{if }\pi_{\theta,x}(y_{t})>\pi_{\text{old},x}(y_{t})\cdot(1+\epsilon_{c}).\end{cases}\right]

Recall that R θ = ρ t ​ ( y , θ ) = π θ , x ​ ( y t ) π old , x ​ ( y t ) R_{\theta}=\rho_{t}(y;\theta)=\frac{\pi_{\theta,x}(y_{t})}{\pi_{\text{old},x}(y_{t})} and both π θ , x ​ ( y t ) \pi_{\theta,x}(y_{t}) and π old , x ​ ( y t ) \pi_{\text{old},x}(y_{t}) in range [ 0 , 1 ] [0,1] . Therefore, we observe that there is a positive gradient ( \big( gradient that increases OPEN π θ , x ​ ( y t ) ) \pi_{\theta,x}(y_{t})\big) if R θ < 1 − ϵ c R_{\theta}<1-\epsilon_{c} , and a negative gradient ( \big( gradient that decreases π θ , x ​ ( y t ) \pi_{\theta,x}(y_{t}) ) \big) if R θ > 1 + ϵ c R_{\theta}>1+\epsilon_{c} , which means the clipping bias discourages the model from leaving the clipping region. In the rest of the paper, for simplicity we refer to the positive gradient case as “positive gradient bias” and the other way as “negative gradient bias”.

#### B.1.3 Clipping Creates Asymmetric Updates Towards Model Prior Knowledge

In this section, we provide further evidence for our hypothesis that the clipping bias increases the likelihood of high-probability rollouts, similar to Yu et al. (2025) ’s analysis on GRPO with ground-truth labels. We showcase this trend with a simple example. Consider a case where a token has a high-probability π old , x ​ ( y t ) = 0.85 \pi_{\text{old},x}(y_{t})=0.85 and an ϵ c = 0.2 \epsilon_{c}=0.2 that we adopt in our experiments. Then, the upper threshold from the bias formula becomes π old , x ​ ( y t ) ⋅ ( 1 + ϵ c ) = 1.02 \pi_{\text{old},x}(y_{t})\cdot(1+\epsilon_{c})=1.02 . Since the probability output by the policy model cannot exceed 1, the upper clipping threshold of 1.02 1.02 is never reached. Thus, the gradient bias is nonnegative for this token, leading to a net positive gradient bias on the policy model, which leads to increase in probability on this token.

On the other hand, for a low-probability token where π old , x ​ ( y t ) = 0.02 \pi_{\text{old},x}(y_{t})=0.02 , the policy model receives a negative gradient bias when π θ , x ​ ( y t ) > π old , x ​ ( y t ) ⋅ ( 1 + ϵ c ) = 0.024 \pi_{\theta,x}(y_{t})>\pi_{\text{old},x}(y_{t})\cdot(1+\epsilon_{c})=0.024 ; and positive gradient bias when π θ , x ​ ( y t ) < π old , x ​ ( y t ) ⋅ ( 1 − ϵ c ) = 0.016 \pi_{\theta,x}(y_{t})<\pi_{\text{old},x}(y_{t})\cdot(1-\epsilon_{c})=0.016 . The low threshold for negative gradient bias makes penalties more likely than in the high-probability case.

The clipping range width scales linearly with the original probability: higher-probability tokens have wider ranges and face fewer penalties. This asymmetric treatment prevents low-probability samples from receiving substantial upweighting during training, causing the model to concentrate probability mass on its existing distribution.

##### Empirical validation.

We empirically validate our conjecture by disabling the clipping term in the GRPO loss, and observe differences in the training dynamics. Specifically, we focus on Token Probability π θ \pi_{\theta} and Code Frequency as defined below.

Recall that π old \pi_{\text{old}} is the old policy from the previous training step and the π θ \pi_{\theta} the current policy, let G G be the number of rollouts generated per prompt; let B B be the mini batch size, which is the number of prompts on which a single gradient update is performed; and let M M be the number of gradient updates from the old policy, π old \pi_{\text{old}} , to the new policy model, π θ \pi_{\theta} . Note that G ⋅ B ⋅ M G\cdot B\cdot M is the total number of rollouts generated by π old \pi_{\text{old}} . The average token probability of π θ \pi_{\theta} over all rollouts { y ( k ) } \{y^{(k)}\} at any step can be expressed as:

π θ , x ​ ( y ) = 1 M ​ ∑ i = 1 M 1 B ​ ∑ j = 1 B 1 G ​ ∑ k = 1 G π θ i , x j ​ ( y ( k ) ) . \pi_{\theta,x}(y)=\frac{1}{M}\sum_{i=1}^{M}\frac{1}{B}\sum_{j=1}^{B}\frac{1}{G}\sum_{k=1}^{G}\pi_{\theta_{i},x_{j}}(y^{(k)}).

Figure 9 (a) shows that the average token probability mass increases for the standard loss of GRPO with clipping (purple line), but stays relatively constant when clipping is disabled. In addition, we log the frequency of code reasoning throughout the RLVR training in Figure 9 (b). We hypothesize that the increase in average token probability correlates with the increase in code reasoning behavior, which, as we conjectured in Section 5 , enhances the performance of the model, as we empirically show in Figure 6 of Section 5.2 .

Random w/o Clipping Random w/ Clipping

##### Why only Qwen-Math models benefit.

The clipping bias mechanism operates on all models, but its effectiveness depends on what behaviors get amplified. The bias systematically favors a model’s pre-existing high-probability behaviors, but only benefits performance if those behaviors correlate with correctness. For Qwen2.5-Math models, code reasoning occurs in 65% of responses and correlates strongly with correctness (64% accuracy vs. 29% without code). We observe that, when RLVR with clipping bias increases the model’s code reasoning frequency to >90%, performance improves substantially. No-Code models (Llama, OLMo2-7B) generate no code reasoning, so clipping bias has no beneficial pattern to amplify. On the other hand, Bad-Code models (OLMo2-7B-SFT) generate code 98% of the time but with 21% accuracy vs. 40% for natural language, indicating that good performance does not rely on code reasoning. Overall, the clipping bias is universal, but its impact on performance depends entirely on whether the model’s dominant pre-existing behaviors happen to be effective reasoning strategies.

##### Implications.

Our findings reveal that GRPO’s clipping mechanism provides a meaningful training signal even from pure noisy rewards, by systematically favoring the model’s pre-existing behavioral patterns. This suggests that the apparent “reward signal” in random reward training is actually an artifact of the optimization algorithm’s bias toward exploiting learned priors rather than exploring new behaviors. This mechanism explains why random rewards work for Qwen2.5-Math models (which have strong code reasoning priors) but fail for other model families lacking these pre-existing capabilities. The training algorithm amplifies whatever reasoning patterns already correlate with correctness, regardless of the reward signal’s actual informativeness.

### B.2 High Stochasticity in RLVR Without Clipping

In this section, we empirically show that training without clipping has stochastic training dynamics and can sometimes obtain performance improvement despite the expected gradient being zero. Following the experimental setup in Section 4 , we disable the clipping effect through the following interventions: (i) removing clipping from the loss implementation, (ii) increasing mini-batch size to match rollout size, and (iii) decreasing the rollout size to match mini-batch size across multiple random seeds. We report the results in Figure 10 across different random seeds.

When clipping is disabled by adjusting batch size, model performance remains stable within a consistent range without converging toward either extreme of the accuracy spectrum. These runs involve 8 times fewer gradient updates due to the batch configuration.

In contrast, removing clipping from the implementation produces extreme stochasticity, occasionally resulting in high-performance convergence across multiple runs. While the exact mechanism remains unclear, we hypothesize that inherent randomness in RLVR training contributes to these observations. The group size G = 16 G=16 used in GRPO training creates high variance in gradient updates. Combined with the instability of unclipped training, this variance may accidentally reinforce certain reasoning patterns, leading to improved performance in some runs. We leave a systematic analysis of this behavior to future work.

## Appendix C Additional Results on AMC

We supplement additional AMC results for non-Qwen models in Figure 11 . The trends are consistent with the MATH-500 results that are shown in Section 3 .

Ground Truth Majority Vote Format Incorrect Random

## Appendix D Additional Results on AIME 2024 and AIME 2025

We present additional results on the AIME benchmarks, which are challenging math Olympiad tests containing significantly harder problems than those in MATH-500 or AMC. We evaluate average@8 accuracy on AIME24 and AIME25 ( Li et al., 2024 ) . AIME25 was created after the release date of all models considered in our study. Thus, evaluating performance on AIME25 allows us to control the risk that our models’ have seen similar problems during web pretraining. We evaluate the trained models from Section 2 and Section 3 . We show results on Qwen2.5-Math models in Figure 12 and on the 8 additional models from Section 3 in Figure 13 .

Ground Truth Majority Vote Incorrect Format Random

### D.1 Spurious Rewards Yield Significant RLVR Gains on Qwen2.5-Math

As shown in Figure 12 , spurious rewards can consistently yield performance gains on Qwen-Math models on AIME24. Intriguingly, we find that any AIME24 gains achievable from training Qwen models with spurious rewards largely vanish when evaluating on AIME 2025. We speculate that AIME25 contains questions that are more out-of-distribution to Qwen’s pretrained knowledge; spurious rewards—which largely serve to elicit existing knowledge—hence no longer provide benefit.

### D.2 (Lack of) Generalization to Other Models

Overall, results on other models are largely consistent with our earlier findings on AMC and MATH-500 (§ 3 ). Only Qwen2.5-7B, Qwen2.5-1.5B, and Llama3.1-8B-Instruct exhibit any notable gains from any reward signals on AIME. For Qwen2.5 models, weak and spurious rewards can yield gains; for example, format reward for Qwen2.5-1.5B and incorrect reward for Qwen2.5-7B. For Llama3.1-8B-Instruct, only standard rewards (e.g., ground truth and majority vote) yield gains. As observed above, performance and gains from RLVR training are lower across the board for all models on AIME25.

Ground Truth Majority Vote Incorrect Format Random

Compound

Original Qwen-Math-7B Qwen-Math-1.5B Qwen-7B Qwen-1.5B Olmo2-7B-SFT

## Appendix E Examination of Existing Methods on Models Beyond Qwen

In this section, we examine the existing methods—TTRL and 1-shot RL—that apply weak supervision during RL, as shown in Figure 15 . We find both methods are significantly effective on Qwen models, but not others.

Qwen2.5-Math-7B Qwen2.5-Math-1.5B Qwen2.5-7B Qwen2.5-1.5B OLMo2-7B-SFT OLMo2-7B Llama3.1-8B Llama3.2-3B Llama3.1-8B-Instruct Llama3.2-3B-Instruct

## Appendix F Compound Rewards that Inhibit Code Reasoning

Qwen-Math-7B Qwen-7B Olmo2-7B-SFT

Compound Reward

Original Reward

##### Inhibiting code reasoning during RLVR with spurious rewards can reduce gains on Qwen2.5-Math-7B, but increase gains on other models.