Reinforcement Learning for Reasoning in Large Language Models with One Training Example — part 4/5

Cleaned-text code points [46485, 61937). Bibliography omitted here; source.md includes it.

In Tab. 9 , we present the detailed 1(few)-shot RLVR results for Qwen2.5-Math-1.5B. Here, we record the model’s best performance on each benchmark individually, so their average can be higher than the best overall average performance (“Avg.”). We include these results to estimate the upper limit of what the model can achieve on each benchmark. Additionally, we include several examples that, while not performing as well as π 1 \pi_{1} or π 13 \pi_{13} , still demonstrate significant improvements, such as π 2 \pi_{2} , π 1201 \pi_{1201} , and π 1209 \pi_{1209} . We observe that, in general, better results correspond to a larger checkpoint step for best average performance, which may correspond to a longer post-saturation generalization process. Similarly, in Tab. 11 , we also include the best per-benchmark results for Qwen2.5-Math-7B, Llama-3.2-3B-Instruct, respectively, together with Qwen2.5-Math-1.5B with PPO training.

#### C.1.4 Detailed Test curves on MATH500 for 1-shot RLVR on Qwen2.5-Math-1.5B.

We plot the performance curves for each subject in MATH500 under 1-shot RLVR using different mathematical examples. As shown in Fig. 6 , the choice of example leads to markedly different improvements and training dynamics in 1-shot RLVR, highlighting the critical importance of data selection for future few-shot RLVR methods.

#### C.1.5 Detailed RLVR results on eacn benchmark over training process.

To better visualize the training process of RLVR and compare few-shot RLVR with full-set RLVR, we show the performance curves for each benchamrk on each model in Fig. 7 , 8 , 9 . It will be interesting to see that if applying 1(few)-shot RLVR for more stable GRPO variants [ 13 , 11 , 12 , 16 ] can alleviate this phenomenon. In addition to the conclusions discussed in Sec. 3.3 , we also note that Llama3.2-3B-Instruct is more unstable during training, as almost all setups start having performance degradation before 200 steps.

In Appendix C.1.2 , we also test the base model and instruction version models in Qwen family. Their test curves are also shown in Fig. 10 and Fig. 11 .

#### C.1.6 More Evaluation on DeepSeek-R1-Distill-Qwen-1.5B

In Tab. 12 we show the DeepSeek-R1-Distill-Qwen-1.5B results at 8k and 32k evaluation lengths. The experimental setup is illustrated in Appendix B.3 .

### C.2 Analysis

#### C.2.1 Test Curves for Ablation Study

In Fig. 12 , we can see the test curves for ablation study (Sec. 4.1 ). We can see that policy gradient loss is the main contributor of 1-shot RLVR. More discussions about format fixing are in Appendix C.2.3 .

#### C.2.2 Entropy loss

##### Detailed results of entropy-loss-only training.

As in Sec. 4.2 , we show the full results of entropy-loss-only training in Tab. 13 . Training with only entropy loss for a few steps can improve model performance on all math benchmarks except AIME2025. The test curves are in Fig. 12 . Notice that the improvement of entropy-loss-only training on Qwen2.5-Math-1.5B is similar to that of RLVR with format reward (Appendix C.2.3 , Tab. 14 ), thus we doubt that the effectiveness of entropy-loss-only training may come from format fixing, and we leave the rigorous analysis of this phenomenon for future works.

##### Discussion of entropy loss and its function in 1-shot RLVR.

Notably, we observe that the benefit of adding entropy loss for 1-shot RLVR is consistent with conclusions from previous work [ 60 ] on the full RLVR dataset, which shows that appropriate entropy regularization can enhance generalization, although it remains sensitive to the choice of coefficient. We conjecture the success of 1-shot RLVR is that the policy gradient loss on the learned example (e.g., π ⁡ ( 1 ) \pi(1) ) actually acts as an implicit regularization by ensuring the correctness of learned training examples when the model tries to explore more diverse responses or strategies, as shown in Fig. 3 (Step 1300). And because of this, both policy loss and entropy loss can contribute to the improvement of 1-shot RLVR. We leave the rigorous analysis to future works.

#### C.2.3 (Only) Format Correction?

As discussed in Dr. GRPO [ 13 ] , changing the template of Qwen2.5-Math models can significantly affect their math performance. In this section, we investigate some critical problems: is (1-shot) RLVR doing format fixing? And if the answer is true, is this the only thing 1-shot RLVR does?

To investigate it, we consider three methods:

##### (a). Applying format reward in RLVR.

We first try to apply only format reward for RLVR (i.e., if the verifier can parse the final answer from model output, then it gets 1 reward no matter if the answer is correct or not, otherwise it gets 0 reward), considering both 1-shot and full-set. The results are shown in Tab. 14 , and the test curves are shown in Fig. 14 and Fig. 13 , respectively.

Notably, we can find that (1) Applying format reward to full-set RLVR and 1-shot RLVR behave very similarly. (2) applying only format reward is already capable of improving model performance significantly (e.g., about 29% improvement on MATH500 and about 11% gain on average). (3) There is still significant gap between the performance of 1-shot RLVR with outcome reward using π 1 \pi_{1} and that of format-reward RLVR (e.g., +7.4% on MATH500 and +5.8% on average), although they may have similar ratios of responses that contain “ \boxed{} ” in evaluation (More discussions are in (b) part). (4) In particular, format-reward RLVR is more sensitive to entropy loss based on Fig. 14 and Fig. 13 .

Interestingly, we also note that the best performance of format-reward RLVR on MATH500 and AIME24 are close to that for 1-shot RLVR with relatively worse examples, for example, π 7 \pi_{7} and π 11 \pi_{11} in Tab. 3 . This may imply that 1-shot RLVR with outcome reward can at least work as well as format-reward RLVR, but with proper examples that can better incentivize the reasoning capability of the model, 1-shot RLVR with outcome reward can bring additional non-trivial improvement . Appendix C.2.5 provides a prompt π 1 ′ \pi_{1}^{\prime} , which uses a sub-question of π 1 \pi_{1} , as an example to support our claim here.

##### (b) Observe the change of format in 1-shot RLVR.

We then investigate how the output format of the model, for example, the number of \boxed{} , changes in the 1-shot RLVR progress. The results are shown in Fig. 15 . We can see that (1) the test accuracy is strongly positively correlated to the number of \boxed{} , which matches our claim that format fixing contributes a lot to model improvement in (a), but (2) for some benchmarks like MATH500, Minerva Math and OlympiadBench, when the number of \boxed{} keeps a relatively high ratio, the test accuracy on these benchmarks is still improving, which may imply independent improvement of reasoning capability.

In particular, to prevent the case that the model outputs the correct answer but not in \boxed{} , we also use LLM-as-a-judge [ 61 ] with QwQ-32B [ 62 ] to judge if the model contains the correct answer in the response. The results are shown in Tab. 15 . We can see that the accuracy judged by rule-based Qwen-Eval pipeline and LLM judger QwQ-32B are very close, and as the ratio of \boxed{} increases, the test accuracy also increases, which implies that the number of correct answers exhibited in the response also increases, rather than just putting correct answer into \boxed{} .

Notably, we also observe that Qwen2.5-Math models contain lots of repetition at the end of model responses, which may result in failure of obtaining final results. The ratio of repetition when evaluating MATH500 can be as high as about 40% and 20% for Qwen2.5-Math-1.5B and Qwen2.5-Math-7B, respectively, which is only about 2% for Llama3.2-3B-Instruct. This may result in the large improvement of format fixing (e.g., format-reward RLVR) mentioned in (a).

(c) In-context learning with one-shot example. In-context learning [ 63 ] is a widely-used baseline for instruction following (although it may still improve model’s reasoning capability). In this section, we try to see if 1-shot RLVR can behave better than in-context learning. Especially, we consider the official 4 examples chosen by Qwen-Eval [ 25 ] for in-context learning, and also the single training example π 1 \pi_{1} . The results are shown in Tab. 16 .

We can find that (1) surprisingly, π 1 \pi_{1} with self-generated response can behave much better than Qwen’s official examples , both for 1.5B and 7B models. In particular on Qwen2.5-Math-7B, in-context learning with π 1 \pi_{1} can improve MATH500 from 51.0% to 75.4% and on average from 22.4% to 37.4%. (2) Although in-context learning also improves the base models, 1-shot RLVR still performs better than all in-context results, showing the advantage of RLVR.

In short, we use these three methods to confirm that 1-shot RLVR indeed does format fixing and obtains a lot of gain from it, but it still has additional improvement that cannot be easily obtained from format reward or in-context learning.

#### C.2.4 Influence of Random Wrong Labels

In this section, we want to investigate the label robustness of RLVR. It’s well-known that general deep learning is robust to label noise [ 64 ] , and we want to see if this holds for RLVR. We try to randomly flip the labels of final answers in DSR-sub and see their performance. Here we randomly add or subtract numbers within 10 and randomly change the sign. If it is a fraction, we similarly randomly add or subtract the numerator and denominator.

The results are in Tab. 17 . We can see that (1) changing 60% of the data with wrong labels can still achieve good RLVR results. (2) if 90% of the data in the dataset contains wrong labels (i.e., only about 120 data contain correct labels, and all other 1.1k data have wrong labels), the model performance will be worse than that for 1-shot RLVR with π 1 \pi_{1} (which only contains 1 correct label!). This may show that RLVR is partially robust to label noise, but if there are too many data with random wrong labels, they may hurt the improvement brought by data with correct labels.

#### C.2.5 Change the Prompt of π 1 \pi_{1}

As discussed in Sec. 3.2.1 , we show that the model can almost solve π 1 \pi_{1} but sometimes fails in solving its last step: “Calculate 2048 3 \sqrt[3]{2048} ”. We use this step itself as a problem ( π 1 ′ \pi_{1}^{\prime} ), and see how it behaves in 1-shot RLVR. The results are in Tab. 18 . Interestingly, we find that π 1 ′ \pi_{1}^{\prime} significantly underperforms π 1 \pi_{1} and has only 1.3% average improvement compared with format reward (as illustrated in Appendix C.2.3 (a)). We think the reason should be that although solving 2048 3 \sqrt[3]{2048} is one of the most difficult parts of π 1 \pi_{1} , π 1 \pi_{1} still needs other key steps to solve (e.g., calculating k k from P = k ​ A ​ V 3 P=kAV^{3} given some values) that may generate different patterns of CoT (rather than just calculating), which may allow more exploration space at the post-saturation generalization stage and maybe better incentivize the model’s reasoning capability.

### C.3 Response Length

In Tab. 19 , we report the average response length on the evaluation tasks. The response length on the test tasks remains relatively stable compared to that on the training data.

### C.4 Pass@8 Results

In Tab. 20 , we report the pass@8 results on the evaluation tasks. Interestingly, we find that (1) 1-shot RLVR achieves comparable or even slightly better pass@8 performance (51.7(2) full-set RLVR (with 1.2k DSR-sub) exhibits a noticeable downward trend in pass@8 performance after 200 steps, which is consistent with recent findings that RLVR may sometimes degrade the pass@n performance [ 20 ] .

## Appendix D Discussions

### D.1 Limitations of Our Work

Due to the limit of computational resources, we haven’t tried larger models like Qwen2.5-32B training currently. But in general, a lot of RLVR works are conducted on 1.5B and 7B models, and they already achieve impressive improvement on some challenging math benchmarks like OlympiadBench, so our experiments are still insightful for RLVR topics. Another limitation of our work is that we mainly focus on the math domain, but haven’t tried 1(few)-shot RLVR on other verifiable domains like coding. But we also emphasize that all math-related experiments and conclusions in our paper are logically self-contained and clearly recorded, to ensure clarity and avoid confusion for readers. And we mainly focus on analyzing this new phenomenon itself, which already brings a lot of novel observations (e.g., cross-category generalization, post-saturation generalization, and more frequent self-reflection in 1-shot RLVR, etc.). We leave the few-shot RLVR on other scenarios for future work.

In particular, we note that our main focus is to propose a new observation rather than propose a new better method, noting that 1-shot RLVR doesn’t save (and maybe requires more) RL computation. Besides, π 1 \pi_{1} is not necessarily the best choice for 1-shot RLVR on other models, since it’s selected based on the historical variance score of Qwen2.5-Math-1.5B. In general, using few-shot RLVR may be more stable for training, as we have seen that on DeepSeek-R1-Distill-Qwen-1.5B (Tab. 4 ), Qwen2.5-Math-7B (Tab. 4 , 10 ) and Qwen2.5-1.5B (Tab. 10 ), RLVR with 16 examples ( { π 1 , … , π 16 } \{\pi_{1},\ldots,\pi_{16}\} ) works as well as RLVR with 1.2k dataset DSR-sub and outperforms 1-shot RL with π 1 \pi_{1} .

### D.2 Reasoning Capability of Base Models

The effectiveness of 1(few)-shot RLVR provides strong evidence for an assumption people proposed recently, that is, base models already have strong reasoning capability [ 13 , 6 , 20 , 21 ] . For example, Dr. GRPO [ 13 ] has demonstrated that when no template is used, base models can achieve significantly better downstream performance. Recent work further supports this observation by showing that, with respect to the pass@k metrics, models trained via RLVR gradually perform worse than the base model as k k increases [ 20 ] . Our work corroborates this claim from another perspective, as a single example provides almost no additional knowledge. Moreover, our experiments reveal that using very few examples with RLVR is already sufficient to achieve significant improvement on mathematical reasoning tasks. Thus, it is worth investigating how to select appropriate data to better activate the model during the RL stage while maintaining data efficiency .

### D.3 Why Model Continues Improving After the Training Accuracy Reaches Near 100%?

A natural concern of 1-shot RLVR is that if training accuracy reaches near 100% (which may occur when over-training on one example), the GRPO advantage (Eqn. 6 ) should be zero, eliminating policy gradient signal. However, entropy loss encourages diverse outputs, causing occasional errors ( 99.x% training accuracy) and non-zero gradients (advantage becomes large for batches with wrong responses due to small variance). This shows the importance of entropy loss to the post-saturation generalization (Fig. 5 ). Supporting this, Fig. 16 shows that for 1-shot RLVR training ( π 1 \pi_{1} ) on Qwen2.5-Math-1.5B, policy gradient loss remains non-zero after 100 steps.

### D.4 Future Works

We believe our findings can provide some insights for the following topics:

##### Data Selection and Curation.