Spurious Rewards: Rethinking Training Signals in RLVR — part 5/6

Cleaned-text code points [62555, 78531). Bibliography omitted here; source.md includes it.

We hypothesized that code reasoning is part of the source of weak and spurious rewards gains. Contrapositively, penalizing code frequency could potentially reduce the gains of these rewards in Qwen2.5-Math. To test this, we designed compound rewards that intersect each weak or spurious reward with a no Python reward , so that a response is rewarded if and only if (1) it satisfies the original spurious reward condition and (2) it does not contain the string “ python ”. Consistent with our hypothesis, format rewards cease to improve Qwen2.5-Math-7B when paired with the no code reward (Figure 16(c) ). The spurious incorrect compound reward matches the original incorrect reward on MATH-500; on more difficult benchmarks such as AMC and AIME, gains also persist but are reduced (Figures 16(b) , 14 ). Thus, while ablating code reasoning does hurt the incorrect reward as predicted, we posit that other beneficial behaviors (e.g., reduced repetition, see Appendix G ) may still be superficially elicited. In addition, ground - truth rewards still yield gains, suggesting benefits beyond eliciting code reasoning (consistent with the code - frequency trends in Figure 6 ).

Intriguingly, for Bad-Code models Qwen2.5-7B and OLMo2-7B-SFT, compound rewards often outperform the originals. Most strikingly, OLMo2-7B-SFT—which degrades under standalone format or incorrect rewards—gains +8.9 and +5.5 points, respectively, once the no-code reward is added. We hypothesize that this is because Qwen2.5-7B and OLMo2-7B-SFT exhibit weak code reasoning before RLVR training, so compound rewards explicitly downweight a behavior that is suboptimal for these models.

In Figure 14 , we present additional results for our compound rewards on (1) more models (Qwen2.5-1.5B, Qwen2.5-Math-1.5B) and (2) more benchmarks (AIME24, AIME25). Overall, results corroborate our analysis in Section 5.3 ; Qwen2.5-Math-1.5B follows the same overall trends as Qwen2.5-Math-7B. Compound rewards also continue to benefit Qwen2.5-1.5B, where gains are comparable to the gains from using the original reward. This slightly contrasts our observations in Qwen2.5-7B, where compound rewards often yielded stronger gains. We conjecture that Qwen2.5-1.5B is stronger at code reasoning, which is consistent with our finding in Section 5.3 that inducing code reasoning via prompting improves performance in in Qwen2.5-1.5B but not Qwen2.5-7B. We are unsure of the exact reason for this discrepancy between these two models (which have the same pretraining data) and leave it for future work.

## Appendix G Beyond Code Reasoning: Another Beneficial Pattern that RLVR Can Easily Elicit

In the main paper, we show that RLVR with spurious rewards can improve Qwen2.5-Math’s performance by surfacing useful reasoning patterns learned during pretraining, and we use code reasoning as a standout example. We note that code reasoning is one distinctive reasoning representation, but not the only one. In this section, we briefly discuss another pattern—no repetition—that can also be easily elicited by RLVR in addition to code reasoning.

We observe that Qwen2.5-Math models have a relatively higher tendency to produce repetitive outputs compared to Llama3 and Olmo2 models. We find that answers with code reasoning often do not have this issue. Therefore, we further study the effect of purely discouraging repetition in answers. To study this, we design a new repetition reward that returns a score of 0 when the answer contains any string repeated more than 10 times. Otherwise, it rewards the model with a full score of 1.

As shown in Figure 17 , the RLVR no-repetition reward improves the performance of Qwen2.5-Math-7B and Qwen2.5-Math-1.5B on MATH and AMC. We observe minimal or even negative improvement on other models. Based on our findings, we hypothesize that various patterns exist whose presence correlates with answer correctness. These patterns, including code reasoning and no repetition, can be easily elicited by RLVR even when the rewards provide no information about the ground-truth answers. Still, the effectiveness of eliciting these patterns is heavily model-dependent.

Qwen2.5-Math-7B Qwen2.5-Math-1.5B Qwen2.5-7B Qwen2.5-1.5B OLMo2-7B-SFT OLMo2-7B Llama3.1-8B Llama3.2-3B Llama3.1-8B-Instruct Llama3.2-3B-Instruct

## Appendix H Switch of Reasoning Strategies during RLVR

In this section, we present further analysis on the change of reasoning strategies during RLVR for Qwen2.5-Math and Qwen2.5 models.

##### Reasoning strategy switches during RLVR.

Qwen2.5-Math-7B’s accuracy increases by an average of 23.5 23.5 absolute points across different training signals. To further break down this gain, we track the performance of models trained on each training signal across four disjoint subsets of the test prompts: (1) Code → \rightarrow Code : the model uses code reasoning both before and after RLVR; (2) Code → \rightarrow Lang : the model initially uses code reasoning but switches to natural language reasoning; (3) Lang → \rightarrow Code : the model initially uses natural language reasoning but switches to code reasoning; and (4) Lang → \rightarrow Lang : the model uses natural language reasoning both before and after RLVR. Specifically, we focus on two interconnected metrics: frequency and accuracy of each subset. To systematically quantify the contribution of each subset to the performance gain, we define a Partial Contribution Score , C d C_{d} , for any subset d ⊆ 𝒟 d\subseteq\mathcal{D} of the entire test set 𝒟 \mathcal{D} , such that C d C_{d} is the ratio between the net increase in the number of correctly answered problems in d d divided by the net increase in the number of correctly answered problems in 𝒟 \mathcal{D} : C d = ∑ x ∈ d 𝕀 ⁡ [ correct ​ ( x t ) ] − 𝕀 ⁡ [ correct ​ ( x 0 ) ] ∑ x ∈ 𝒟 𝕀 ⁡ [ correct ​ ( x t ) ] − 𝕀 ⁡ [ correct ​ ( x 0 ) ] , where ​ x 0 ​ and ​ x t ​ are the initial and final answers. C_{d}=\frac{\sum_{x\in d}\mathbb{I}[\text{correct}(x_{t})]-\mathbb{I}[\text{correct}(x_{0})]}{\sum_{x\in\mathcal{D}}\mathbb{I}[\text{correct}(x_{t})]-\mathbb{I}[\text{correct}(x_{0})]},\hskip 5.69054pt\text{where }x_{0}\text{ and }x_{t}\text{ are the initial and final answers.}

##### Frequency:

Figure 18 shows Qwen2.5-Math-7B’s reasoning strategy switching pattern. For all weak and spurious rewards, the model uses more code reasoning after RLVR. While few originally-code-reasoning problems switch to language reasoning (Code → \rightarrow Lang), most originally-language-reasoning problems convert to code reasoning (Lang → \rightarrow Code). Ground truth reward does not follow this pattern. For Bad-Code models (Qwen2.5-7B and OLMo2-7B-SFT), meaningful rewards steer models away from bad code reasoning. Code reasoning decreases with ground truth, majority vote, and incorrect rewards for Qwen2.5-7B, but only with ground truth and majority vote for OLMo2-7B-SFT (Figures 6(b) and 6(c) ). For No-Code models, RLVR fails to elicit meaningful changes in reasoning strategy, as this capability is likely not learned during pre-training.

Accuracy: From Figure 18 , there is a drastic increase in accuracy in the Lang → \rightarrow Code subset after RLVR across all training signals. This is reflected in Table 3 , which shows that 58.3% of the performance gain of Qwen2.5-Math-7B is from this subset. Similarly in Qwen2.5-Math-1.5B, switching from natural language reasoning to code reasoning contribute to 78.7% of the performance gain. For the Bad-Code models, Code → \rightarrow Lang contributes to 93.9% of the performance gain of Qwen2.5-7B. This is intuitive, as the model has a higher langauge reasoning accuracy than code reasoning accuracy, RLVR training essentially encourages the model to use the reasoning strategy that it is better at. For No-Code models, since there is no code reasoning before or after RLVR, all performance gains (or losses) are from the Lang → \rightarrow Lang subset. These results suggest that much of the accuracy gain from RLVR on these spurious rewards in Qwen2.5-Math and Qwen2.5 is simply from eliciting the right reasoning strategy from the model.

## Appendix I Spurious Prompts: Qwen2.5-Math-7B’s Unreasonably High Sensitivity to Prompts

Ground Truth Format Random

As we show in Section 5.3 , prompting the model to use code reasoning brings a 15.0% gain on Qwen2.5-Math-7B. This suggests that prompt engineering is a valid approach to elicit desired behaviors, which does not require parameter updates. In this section, we conduct an additional analysis on how choosing different existing prompts can impact model behavior and how such a technique interacts with RLVR. In addition, we show that model performance can be improved even when using some task-unrelated, information-less prompts, which we name “spurious prompts.”

### I.1 Existing Prompts

We collect different prompts from popular RL training frameworks and recent evaluation works: Qwen Default ( Yang et al., 2024a ) , SimpleRL-Zoo ( Zeng et al., 2025 ) , Sober Reasoning ( Hochlehnert et al., 2025 ) , and our hand-constructed prompts (detailed in Table 4 ). Specifically, we introduce two new prompts—the first prompt, Math Prompt, indicates the evaluation domain without extra information about the format; the second prompt, Spurious Prompt, is a randomly picked LaTeX placeholder text generated by lipsum . Although it has been known that LLM evaluation can be sensitive to evaluation prompts ( Sclar et al., 2024 ) , our motivation to introduce these two particular prompts is to study how concise domain knowledge or even random perturbation in context can impact evaluation performance. We perform RL training using these prompts for both trajectory rollouts and evaluation.

As shown in Table 5 , Sober prompt brings the highest parsable rate of the answers, while our spurious prompt leads to the highest accuracy on MATH-500. The results indicate that the model is very sensitive to prompts and the best performance does not necessarily require the highest parsable rate nor task-relevant information in context.

We further show the training trajectories using different prompts in Figure 19 . We find that models trained with the Qwen default, math problem: , and Simplerl-zoo converge to similar performance after RLVR with the same training setting. We conjecture that the behavior that can be elicited by prompting tends to be a subset of the easy-to-elicit behaviors in RLVR, while RLVR can elicit additional behaviors that are not predefined in prompts.

### I.2 Spurious Prompts

As Qwen2.5-Math-7B shows high sensitivity to the prompts, we are curious about how much it could be impacted by spuriously unrelated prompts, e.g., the placeholder text in LaTex generated by lipsum . Specifically, we construct a spurious prompt using lipsum with a similar prompt length as the Sober prompt in Table 4 . Surprisingly, the spurious prompt gives the highest initial performance compared with the other commonly used task-specific prompts, as shown in Figure 19(e) . Although we find that the improvement does not always happen for randomly picked prompts, the high performance with our spurious prompt indicates that the model can be very sensitive to in-context perturbations, where the benefit may sometimes originate from this sensitivity rather than from the task-relevant content in the prompt.

## Appendix J Additional Results on Models That Have Undergone RL Training

In this section, we provide additional results for models that have been trained with reinforcement learning, including two Qwen Instruct models (Qwen2.5-Math-7B-Instruct ( Yang et al., 2024a ) and Qwen2.5-7B-Instruct ( Yang et al., 2024b ) ) and one Tulu3 model (Llama-3.1-Tulu-3-8B ( Lambert et al., 2024 ) ). In this section, we show that the post-RL models may behave differently from other base or instruction-tuned models in RLVR.

As shown in Figure 20 , Qwen Instruct models show minimal improvement from RLVR training across different reward types in our setup, even when using ground-truth rewards. We observe this same pattern in other models: their performance plateaus after sufficient RLVR training steps. Therefore, we conjecture that Qwen2.5-Math-7B-Instruct benefits less from our RLVR experiments because it has already reached saturation from its previous RL training. While improved RLVR algorithms or higher-quality training data could still yield gains on the currently saturated models, we leave this exploration for future work.

Furthermore, we examine the RLVR behavior of Llama-3.1-Tulu-3-8B (Figure 20(c) ) and find that, unlike the Qwen2.5 Instruct models, training with ground truth rewards provides only marginal gains on MATH and limited gains on ACM and AIME. The trends we observe for Llama-3.1-Tulu-3-8B are similar to those we observe for its base model Llama-3.1-8B. We conjecture that this might be because these models share similar prior knowledge before RL training, which leads to similar RLVR behaviors. However, unlike the Qwen Instruct models, Llama-3.1-Tulu-3-8B still shows gains on MATH despite having been trained with RLVR on the Tulu3 dataset.

Ground Truth Format Random

## Appendix K Qualitative Analysis on Qwen2.5-Math-7B’s Coding Reasoning Behaviors

In this section, we show several qualitative examples on how Qwen2.5-Math-7B can reason in code. In addition, we show its code reasoning behavior is robust to numerical perturbations—the model generates similar code to solve the numerical perturbed question. However, we find Qwen2.5-Math-7B only uses natural language solutions when we rephrased the question using an alternative narrative.

##### Qwen2.5-Math-7B can reason in code.

In Figure 22 – 23 , we show 3 qualitative examples of Qwen2.5-Math-7B outputs on 3 randomly picked questions from MATH-500. We find Qwen2.5-Math-7B is able to conduct coding reasoning, i.e., solving a problem by writing code, and predict the code execution outputs. Surprisingly, the model can predict the code execution outputs with a relatively high accuracy—in the examples shown in Figure 22 and Figure 21 , Qwen2.5-Math-7B is able to compute the execution answers with 16-float precision without access to any code interpreter. In the example shown in Figure 23 , Qwen2.5-Math-7B gives a wrong answer 4323. However, it is still very close to the ground-truth answer 4343.

##### Qwen2.5-Math-7B’s code reasoning behavior is robust to numerical perturbations.

We further perturbed the number 999 in the question shown in Figure 23 to a few random integer numbers between the range of 100 and 1000, as shown in Figure 24 – 26 . We find that Qwen2.5-Math-7B is capable of solving our perturbed questions with similar code snippets in their answers. For example, in Figure 24 , we show that Qwen2.5-Math-7B is able to derive the correct answer 4344 in the numerically perturbed version of the question in Figure 23 —we changed the input number from 999 to 1000, and the corresponding ground-truth answer changed from 4343 6 4343_{6} to 4344 6 4344_{6} . We find Qwen2.5-Math-7B used the same code function to solve the question, and interestingly, it predicts the correct answer for the perturbed question, although it predicted an inaccurate answer for the original question in Figure 23 .

##### Qwen2.5-Math-7B’s code reasoning behavior may not directly generalize to rephrased questions.

We also rephrased the question in Figure 23 , Figure 25 , and Figure 26 , respectively, with an alternative narrative. As shown in Figure 27 – 29 , Qwen2.5-Math-7B first identifies and concludes the same core question in the first sentence of its answer. However, it does not use any code to answer this rephrased question. We hypothesize that whether Qwen2.5-Math-7B uses code reasoning to answer the question depends on the specific prompt distribution, i.e., whether the question is close to any of its training data that is augmented with code solutions, rather than the problem itself.