Spurious Rewards: Rethinking Training Signals in RLVR — part 3/6
Cleaned-text code points [31070, 46726). Bibliography omitted here; source.md includes it.
Denote the input prompt as x x and the corresponding model rollouts as y y . Denote the t t -th token in the i i -th rollout y y as y t y_{t} , where 1 ≤ t ≤ | y | 1\leq t\leq|y| . GRPO loss, as introduced by ( Shao et al., 2024 ) , contains the following components: For each prompt x x , we compute the group-wise mean r ¯ x \bar{r}_{x} and standard deviation σ x \sigma_{x} . The normalized group-relative advantage is A ^ ( x , y ) = r ( x , y ) − r ¯ x σ x . \hat{A}(x,y)=\frac{r(x,y)-\bar{r}_{x}}{\sigma_{x}}. Let π old \pi_{\text{old}} be the behavior policy that produced the trajectories and π ref \pi_{\text{ref}} be a frozen reference policy (for example, the initial supervised model). We denote the token-level importance ratio by ρ t ( y , θ ) = π θ ( y t | x , y < t ) π old ( y t | x , y < t ) \rho_{t}(y;\theta)=\frac{\pi_{\theta}(y_{t}|x,y_{<t})}{\pi_{\text{old}}(y_{t}|x,y_{<t})} , where t t is the token index. With PPO-style clipping threshold ϵ c \epsilon_{c} , KL-penalty weight λ \lambda , the surrogate objective maximized is
J ( θ ) = \displaystyle J(\theta)= 𝔼 x ∼ 𝒟 , y ∼ π old ( ⋅ | x ) [ ∑ t = 1 | y | min ( ρ t ( y ; θ ) A ^ ( x , y ) , clip ( ρ t ( y ; θ ) , 1 − ϵ c , 1 + ϵ c ) A ^ ( x , y ) ) ] \displaystyle\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\text{old}}(\cdot|x)}\!\left[\sum_{t=1}^{|y|}\min\Bigl(\rho_{t}(y;\theta)\,\hat{A}(x,y),\operatorname{clip}\bigl(\rho_{t}(y;\theta),1-\epsilon_{c},1+\epsilon_{c}\bigr)\,\hat{A}(x,y)\Bigr)\right] (3) − λ 𝔼 x ∼ 𝒟 [ KL ( π θ ∥ π ref ) ] . \displaystyle-\lambda\,\mathbb{E}_{x\sim\mathcal{D}}\Bigl[\mathrm{KL}\bigl(\pi_{\theta}\|\pi_{\text{ref}}\bigr)\Bigr].
The KL regularization is typically adopted to ensure our model does not deviate too far from the frozen reference model. However, since we focus primarily on verifiable rewards and investigate the signals produced by these rewards, we assume that distribution shift is of lesser concern following ( Liu et al., 2025 ) . Moreover, KL regularization adds confounding factors to our analysis. Recent work has also shown that removing the KL term leads to better performance ( Hu et al., 2025b ) . Therefore, we set λ = 0 \lambda=0 in our work.
### A.2 Datasets and Models
We conduct our experiments on three canonical mathematical reasoning benchmarks: • MATH-500 ( Hendrycks et al., 2021 ; Lightman et al., 2023 ) : A standardized subset of the MATH dataset focusing on advanced mathematical reasoning, including problems from algebra, calculus, and number theory.
• AMC ( Li et al., 2024 ) : The American Mathematics Competition dataset contains diverse mathematical problems ranging from algebra to combinatorics and geometry. For this benchmark, we report model’s average performance over 8 trials ( avg @ 8 \mathrm{avg@8} ).
• AIME ( Li et al., 2024 ) : The American Invitational Mathematics Examination dataset consists of challenging high-school level mathematical problems requiring multi-step reasoning. We include AIME questions from 2024 and 2025 for evaluation. For this benchmark, we report model’s average performance over 8 trials ( avg @ 8 \mathrm{avg@8} ).
For our initial experiments, we primarily utilize the Qwen2.5-Math-7B model, a 7 billion parameter language model specifically tuned for mathematical reasoning tasks. We select this model (1) for its strong baseline performance on mathematical problems while remaining computationally efficient for multiple experimental iterations and (2) because it is frequently used in prior work ( Zuo et al., 2025 ; Wang et al., 2025b ) .
### A.3 Training Configuration
Unless otherwise specified, we train each model on 8 GPUs with a constant learning rate of 5e-7, a mini batch size (number of rollouts seen before a gradient update) of 128, and a rollout batch size (number of prompts we rollout at the same time) of 64. For each prompt, we collect 16 rollouts to compute advantages for GRPO update. We use a sampling temperature τ = 1 \tau=1 . We do not apply KL divergence loss or entropy loss in our training.
### A.4 Decoding Configuration
We use temperature of 0.0 for pass at 1, and temperature of 0.6 for pass at k during decoding time. All other hyperparameters are the default value following the Zuo et al. (2025) codebase.
### A.5 Computation Resource
Each RLVR run in our experiment takes approximately 24 hours on 8 A100s.
### A.6 Visualization
In all plots, we smooth the metric of interest across ten training steps, and overlay the raw curves as transparent dashed lines. We do not smooth the step 0 performance, which corresponds to the performance of the base model before training. We report the smoothed value at the final training step as the final performance.
### A.7 Experimental Setups for TTRL and One-Shot RL
##### TTRL setup.
We follow all hyperparameters from the original TTRL paper ( Zuo et al., 2025 ) , and train all models using their publicly released implementation. Due to compute constraints, we do not extensively sweep hyperparameters on the new base models we consider. Note that the TTRL accuracies are not directly comparable to our majority-vote reward results, because (a) TTRL trains on (unlabeled) test prompts while we train on a much larger set of distinct train prompts, and (b) TTRL updates labels during training based on the online policy’s majority vote, while we assign labels once offline.
##### Differences in our One-Shot RL setup from ( Wang et al., 2025b ) .
We note that for the one-shot RL settings in our paper, we do not apply entropy loss for more consistent setups with other experiments and better stability. However, ( Wang et al., 2025b ) shows that the entropy loss may further improve the performance of one-shot RL and post-saturation generalization. We use the same training example π 1 \pi_{1} (defined in ( Wang et al., 2025b ) ) for one-shot RLVR experiments on all models as in ( Wang et al., 2025b ) . Their work discusses that different models should possibly select different examples for more performant one-shot RLVR training, but use π 1 \pi_{1} in all experiments due to high cost of trying different possible 1-shot examples. In particular, we note that this setup difference may cause a noticeable performance difference for Qwen2.5-7B model compared to the entropy loss setting used in ( Wang et al., 2025b ) (71.2% with entropy loss vs. 54.8% without).
## Appendix B The Curious Case of Random Rewards
In this section, we study a special case of our spurious rewards, random rewards, where the reward is randomly assigned independently of the model rollouts. We first show that random rewards work across several non-zero reward probabilities (Figure 8 ), confirming that this is a stable observation. Next, we provide detailed gradient derivation in this special case and discuss our hypothesis on one of the potential sources of training signals with random rewards—the clipping bias in GRPO update.
##### Random rewards with varying probabilities consistently improve performance.
We train Qwen2.5-Math-7B using GRPO with random rewards assigned by 𝙱𝚎𝚛𝚗𝚘𝚞𝚕𝚕𝚒 ( γ ) \mathtt{Bernoulli}(\gamma) variables, where γ ∈ { 0.7 , 0.5 , 0.3 , 0.001 , 0 } \gamma\in\{0.7,0.5,0.3,0.001,0\} . Each response receives reward 1 with probability γ \gamma and 0 otherwise. We find that all non-zero probability configurations successfully lead to significant performance gains after an initial period of exploration, yielding comparable performance to ground truth rewards (Figure 8 ). γ = 0 \gamma=0 yields no improvement as expected, since constant rewards provide no learning signal. The convergence speed varies with γ \gamma , but all configurations eventually reach similar high-performance regimes, with accuracy improvements of 15-20 percentage points on MATH-500.
##### Recap of notations.
Recall that we denote the input prompt as x x and the corresponding model rollouts as { y ( 1 ) , ⋯ , y ( G ) } \{y^{(1)},\cdots,y^{(G)}\} , where G ∈ ℕ + G\in\mathbb{N^{+}} is the rollout size. In addition, we denote the t t -th token in the i i -th rollout y ( i ) y^{(i)} as y t ( i ) y^{(i)}_{t} , where t ∈ ℕ + t\in\mathbb{N^{+}} and 1 ≤ t ≤ | y ( i ) | 1\leq t\leq|y^{(i)}| . The normalized group-relative advantage in GRPO is A ^ ( x , y ) = r ( x , y ) − r ¯ x σ x \hat{A}(x,y)=\frac{r(x,y)-\bar{r}_{x}}{\sigma_{x}} , where r ¯ x \bar{r}_{x} is the group-wise mean and σ x \sigma_{x} is the group-wise standard deviation. The token-level importance ratio 4 4 4 For simplicity, we denote π ( y t | x , y < t ) \pi(y_{t}|x,y_{<t}) as π x ( y t ) \pi_{x}(y_{t}) , and similarly π θ ( y t | x , y < t ) \pi_{\theta}(y_{t}|x,y_{<t}) as π θ , x ( y t ) \pi_{\theta,x}(y_{t}) , π old ( y t | x , y < t ) \pi_{\text{old}}(y_{t}|x,y_{<t}) as π old , x ( y t ) \pi_{\text{old},x}(y_{t}) . is ρ t ( y , θ ) = π θ , x ( y t ) π old , x ( y t ) \rho_{t}(y;\theta)=\frac{\pi_{\theta,x}(y_{t})}{\pi_{\text{old},x}(y_{t})} , where t t is the token index.
### B.1 Conjecture: Clipping Bias Brings Training Signals under Random Rewards
#### B.1.1 Expected Advantage of Random Rewards
We investigate the expected advantage when the reward signal is assigned by a random 𝙱𝚎𝚛𝚗𝚘𝚞𝚕𝚕𝚒 ( γ ) \mathtt{Bernoulli(\gamma)} variable. For a prompt x x and rollouts { y ( i ) } i = 1 G \{y^{(i)}\}_{i=1}^{G} , the reward of each y ( i ) y^{(i)} is r ( x , y ( i ) ) ∼ 𝙱𝚎𝚛𝚗𝚘𝚞𝚕𝚕𝚒 ( γ ) r(x,y^{(i)})\sim\mathtt{Bernoulli(\gamma)} , so the expected average reward of x x is γ \gamma . Assuming A ^ i = 0 \hat{A}_{i}=0 when σ x = 0 \sigma_{x}=0 , the sum of the normalized advantages over G G rollouts ∑ i = 1 G A ^ ( x , y ( i ) ) := ∑ i = 1 G r ( x , y ( i ) ) − r ¯ x σ x = ∑ i = 1 G r ( x , y ( i ) ) − G ⋅ r ¯ x σ x = 0 \sum_{i=1}^{G}\hat{A}(x,y^{(i)}):=\sum_{i=1}^{G}\frac{r(x,y^{(i)})-\bar{r}_{x}}{\sigma_{x}}=\frac{\sum_{i=1}^{G}r(x,y^{(i)})-G\cdot\bar{r}_{x}}{\sigma_{x}}=0 by construction. Furthermore, for i.i.d. rewards that are independent of the provided samples, such as the random rewards used in our experiments, 𝔼 ( ∑ i = 1 G A ^ ( x , y ( i ) ) ) = G ⋅ 𝔼 ( A ^ ) = 0 ⟹ 𝔼 ( A ^ ) = 0 \mathbb{E}(\sum_{i=1}^{G}\hat{A}(x,y^{(i)}))=G\cdot\mathbb{E}(\hat{A})=0\implies\mathbb{E}(\hat{A})=0 .
The clipping term in GRPO loss (Equation 3 ) prevents excessive deviation from the previous policy, stabilizing training. Recent work suggests this term, which operates on the ratio ρ t ( y , θ ) = π θ , x ( y t ) / π old , x ( y t ) \rho_{t}(y;\theta)=\pi_{\theta,x}(y_{t})/\pi_{\text{old},x}(y_{t}) , introduces bias toward exploitation in the case of ground truth reward ( Yu et al., 2025 ) . Here, we analyze the bias in the settings of the random rewards, where each rollout receives an advantage that is independent of the rollout.
We note that the loss is no longer differentiable everywhere with clipping. In practical implementations, e.g., PyTorch, the gradients will be automatically set to 0 when the value is clipped. For simplicity, we discuss the gradient of the loss function by assuming 0 gradient at the non-differentiable points.
#### B.1.2 Gradient Derivation of Clipping Bias under Random Rewards
In this section, we derive the training signals induced by the clipping factor, which we name as a ‘‘clipping bias’’. Specifically, we define the clipping bias as the difference in the expected gradients after adding clipping to the GRPO objective 5 5 5 For simplicity, we denote 𝔼 x ∼ 𝒟 , y ∼ π old ( ⋅ | x ) \mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\text{old}(\cdot|x)}} as 𝔼 x , y \mathbb{E}_{x,y} : 𝙱𝚒𝚊𝚜 ( ∇ θ J ( θ ) ) = 𝔼 x , y [ ∇ θ J ( θ ) ] − 𝔼 x , y [ ∇ θ L unclipped ( θ ) ] . \mathtt{Bias}(\nabla_{\theta}J(\theta))=\mathbb{E}_{x,y}[\nabla_{\theta}J(\theta)]-\mathbb{E}_{x,y}[\nabla_{\theta}L^{\text{unclipped}}(\theta)].
As we derived above, 𝔼 ( A ^ ( x , y j ) ) = 0 \mathbb{E}(\hat{A}(x,y_{j}))=0 . Assuming a simple loss with no clipping, the expectation of the policy gradient without the clipping term is also trivially zero given that A ^ \hat{A} is independent of other variables: 𝔼 x , y [ ∇ θ L unclipped ( θ ) ] \displaystyle\mathbb{E}_{x,y}[\nabla_{\theta}L^{\text{unclipped}}(\theta)] = \displaystyle= 𝔼 x , y [ 1 | y | ∑ t = 1 | y | ∇ θ ρ t ( y , θ ) A ^ ( x , y ) ] \displaystyle\mathbb{E}_{x,y}\!\left[\frac{1}{|y|}\sum_{t=1}^{|y|}\nabla_{\theta}\rho_{t}(y;\theta)\,\hat{A}(x,y)\right] = \displaystyle= 𝔼 ( A ^ ) 𝔼 x , y [ 1 | y | ∑ t = 1 | y | ∇ θ ρ t ( y , θ ) ] \displaystyle\mathbb{E}(\hat{A})\mathbb{E}_{x,y}\!\left[\frac{1}{|y|}\sum_{t=1}^{|y|}\nabla_{\theta}\rho_{t}(y;\theta)\right] = \displaystyle= 0 . \displaystyle 0. Therefore, the clipping bias under random reward is 𝙱𝚒𝚊𝚜 ( ∇ θ J ( θ ) ) = 𝔼 [ ∇ θ J ( θ ) ] − 𝔼 [ ∇ θ L unclipped ( θ ) ] = 𝔼 [ ∇ θ J ( θ ) ] . \mathtt{Bias}(\nabla_{\theta}J(\theta))=\mathbb{E}[\nabla_{\theta}J(\theta)]-\mathbb{E}[\nabla_{\theta}L^{\text{unclipped}}(\theta)]=\mathbb{E}[\nabla_{\theta}J(\theta)].
Next, we compute the exact form of this clipping bias, i.e., the gradient in GRPO with random rewards. Recall from Eq. 3 , the surrogate objective that we aim to maximize is: J ( θ ) = min ( ρ t ( y , θ ) A ^ ( x , y ) , clip ( ρ t ( y , θ ) , 1 − ϵ c , 1 + ϵ c ) A ^ ( x , y ) ) . J(\theta)=\min\Bigl(\rho_{t}(y;\theta)\,\hat{A}(x,y),\operatorname{clip}\bigl(\rho_{t}(y;\theta),1-\epsilon_{c},1+\epsilon_{c}\bigr)\,\hat{A}(x,y)\Bigr).
For simplicity, we further denote R θ = ρ t ( y , θ ) = π θ , x ( y t ) π old , x ( y t ) R_{\theta}=\rho_{t}(y;\theta)=\frac{\pi_{\theta,x}(y_{t})}{\pi_{\text{old},x}(y_{t})} to be the token-level importance ratio, C = clip ( R θ , 1 − ϵ c , 1 + ϵ c ) C=\operatorname{clip}(R_{\theta},1-\epsilon_{c},1+\epsilon_{c}) to be the clipping term, and A ^ ( x , y ) \hat{A}(x,y) as A ^ \hat{A} . So we have J ( θ ) = min ( R θ ⋅ A ^ , C ⋅ A ^ ) J(\theta)=\min\bigl(R_{\theta}\cdot\hat{A},C\cdot\hat{A}\bigr) . We now analyze the gradient of GRPO loss with respect to the policy model parameters θ \theta based on the sign of A ^ \hat{A} .
As discussed in Section B.1 , the loss function is not differentiable everywhere. Therefore, we discuss the gradients with respect to the differentiable regions below and set the gradients to 0 at the non-differentiable points at the end.
##### Case 1: A ^ ≥ 0 \hat{A}\geq 0 .
When A ^ ≥ 0 \hat{A}\geq 0 , we can directly take A ^ \hat{A} out of the min \min function. Thus, J ( θ ) = min ( R θ ⋅ A ^ , C ⋅ A ^ ) = A ^ ⋅ min ( R θ , C ) . J(\theta)=\min\bigl(R_{\theta}\cdot\hat{A},C\cdot\hat{A}\bigr)=\hat{A}\cdot\min(R_{\theta},C). Since C C is defined as C = clip ( R θ , 1 − ϵ c , 1 + ϵ c ) C=\operatorname{clip}(R_{\theta},1-\epsilon_{c},1+\epsilon_{c}) , we have: • If R θ < 1 − ϵ c R_{\theta}<1-\epsilon_{c} , then C = 1 − ϵ c C=1-\epsilon_{c} . Thus, min ( R θ , C ) = R θ \min(R_{\theta},C)=R_{\theta} .
• If 1 − ϵ c ≤ R θ ≤ 1 + ϵ c 1-\epsilon_{c}\leq R_{\theta}\leq 1+\epsilon_{c} , then C = R θ C=R_{\theta} . Thus, min ( R θ , C ) = R θ \min(R_{\theta},C)=R_{\theta} .
• If R θ > 1 + ϵ c R_{\theta}>1+\epsilon_{c} , then C = 1 + ϵ c C=1+\epsilon_{c} . Thus, min ( R θ , C ) = 1 + ϵ c \min(R_{\theta},C)=1+\epsilon_{c} .
Combining these, when A ^ ≥ 0 \hat{A}\geq 0 , the objective is:
The gradient with respect to θ \theta for this case is:
##### Case 2: A ^ < 0 \hat{A}<0 .
When A ^ < 0 \hat{A}<0 , multiplying by A ^ \hat{A} flips the min \min function. So, L − ( θ ) = min ( R θ ⋅ A ^ , C ⋅ A ^ ) = A ^ ⋅ max ( R θ , C ) . L^{-}(\theta)=\min(R_{\theta}\cdot\hat{A},C\cdot\hat{A})=\hat{A}\cdot\max(R_{\theta},C).