Reinforcement Learning for Reasoning in Large Language Models with One Training Example — part 2/5

Cleaned-text code points [14979, 30692). Bibliography omitted here; source.md includes it.

First, we inspect the examples that produce such strong results. Tab. 2 lists the instances of π 1 \pi_{1} , which is defined by Eqn. 2 . We can see that it’s actually an algebra problem with a physics background. The key steps for it are obtaining k = 1 / 256 k=1/256 for formula P = k ​ A ​ V 3 P=kAV^{3} , and calculating V = ( 2048 ) 1 / 3 ≈ 12.699 V=(2048)^{1/3}\approx 12.699 . Interestingly, we note that base model already almost solves π 1 \pi_{1} . In Fig. 3 , the base model without any training already solves all the key steps before calculating ( 2048 ) 1 / 3 (2048)^{1/3} with high probability 3 3 3 A more precise answer for π 1 \pi_{1} should be 12.7 12.7 rather than 12.8 12.8 , but this slight deviation does not affect the experimental results. We show that both values yield strong performance in Tab. 5 in Sec. 4.1 . . Just for the last step to calculate the cube root, the model has diverse outputs, including 4, 10.95, 12.6992, 8 ​ 4 3 8\sqrt[3]{4} , 12.70, 12.8, 13, etc. Specifically, for 128 samplings from the base model, 57.8% of outputs are “12.7” or “12.70”, 6.3% of outputs are “12.8”, and 6.3% are “13”. More examples used in this paper are shown in Appendix E . In Appendix C.2.5 , we show that interestingly, even though the key step in solving π 1 \pi_{1} is computing 2048 3 \sqrt[3]{2048} , including only this question in the training example leads to significantly worse performance compared to using full π 1 \pi_{1} .

#### 3.2.2 Post-saturation Generalization: Generalization After Training Accuracy Saturation

Then, we show an interesting phenomenon in 1-shot RLVR. As shown in Fig. 2 , since we only have one training example, it’s foreseeable that the training accuracy for π 1 \pi_{1} and π 13 \pi_{13} quickly saturates before the 100th step. However, the performance on the test set still continues improving: 1-shot RLVR with π 1 \pi_{1} gets 3.4% average improvement from step 100 to step 1540, while using π 13 \pi_{13} yields a 9.9% average improvement from step 500 to step 2000 4 4 4 This behavior looks similar to “grokking”, but we do not emphasize the sudden onset of generalization after training saturates. In Sec. 4.1 , we show that post-saturation generalization is distinct from grokking. . Besides, this phenomenon cannot be observed when using full-set RLVR with DSR-sub currently, as the test performance has started to drop before training accuracy converges.

Moreover, we compare the training and evaluation responses in Fig. 3 . Surprisingly, we find that at the final stage of 1-shot RLVR, the model overfits the single training example by mixing the correct calculation process into long unintelligible multilingual outputs in its outputted reasoning. Nonetheless, the test responses still remain normally and maintain high accuracy, indicating that post-saturation generalization still holds even after overfitting the training example . In particular, overfitting in RLVR occurs quite late ( π 1 \pi_{1} after 1400 steps and π 13 \pi_{13} after 1800 steps). Considering that each example is sampled 1024 times per step, the single training example is not overfitted until after millions of rollouts. Further analysis is provided in Sec. 4.1 .

#### 3.2.3 1-shot RLVR is Effective for Many Examples & Brings Improvements across Categories

In this section, we investigate whether different data behave differently in 1-shot RL, and whether 1-shot RLVR with one training example from a specific category can help the model better generalize to other categories. We select data with high ( π 1 , … , π 17 \pi_{1},\ldots,\pi_{17} ), medium ( π 605 , π 606 \pi_{605},\pi_{606} ), and low ( π 1201 , … ​ π 1209 \pi_{1201},\ldots\pi_{1209} ) historical variance (Eqn. 1 ) and from different topics. We determine the categories of the questions based on their characteristics. We show their detailed MATH500 performance for both overall and subclasses in Tab. 3 . More performance curves are in Appendix C.1 .

We observe that (1) 1-shot RLVR improves performance across all categories in MATH500. Almost all examples yield a ≥ 30 % \geq 30\% improvement over the base model, except for the incorrect example π 1207 \pi_{1207} and the extremely difficult example π 1208 \pi_{1208} , which cause the model to fail to generate any correct solutions. (2) 1-shot RLVR can perform at least as well as the format-reward baseline (except π 1207 \pi_{1207} and π 1208 \pi_{1208} ), and with appropriate examples, 1-shot RLVR with outcome reward can achieve additional non-trivial improvements. From Tab. 3 , we observe that the improvements of some examples (e.g., π 7 \pi_{7} , π 11 \pi_{11} , and π 606 \pi_{606} ) mainly come from format correction. However, many other examples (e.g., π 1 \pi_{1} , π 13 \pi_{13} , and π 1209 \pi_{1209} ) still exhibit non-trivial improvements beyond format fixing. Further discussion is provided in Appendix C.2.3 . (3) Counterintuitively, test data belonging to the same category as the single training example does not necessarily exhibit better improvement. For instance, π 11 \pi_{11} belongs to Number Theory, but RLVR trained with π 11 \pi_{11} achieves a relatively low Number Theory score compared to using other examples (e.g., π 605 \pi_{605} from Precalculus). This may indicate that the reasoning capability stimulated by an instance cannot be simply predicted by superficial features such as categories [ 35 ] . Additional analysis on prompt complexity is provided in Appendix C.2.5 .

#### 3.2.4 More Frequent Self-Reflection on Test Data

In this section, we show another empirical observation of 1-shot RLVR: it can increase the frequency of self-reflection [ 6 ] in the model responses as training progresses. To study this, we check the output patterns of different checkpoints from the RLVR training on Qwen2.5-Math-1.5B. We find that their self-reflection process often appears with words “rethink” , “recheck” and “recalculate” . Therefore, we count the number of responses that contain these three words when evaluating 6 mathematical reasoning tasks. The results are in Fig. 4 . First , after around 1.3k steps, the response length and entropy loss increase significantly, which may imply the attempt of diverse output patterns or overfitting (Fig. 3 ). Second , for the evaluation task, the base model itself already exhibits self-reflection processes, which supports the observation in recent works [ 13 , 21 ] . Third , the number of self-recheck processes increases at the later stages of 1-shot RL training, which again confirms that the model generalizes well on test data and shows more complex reasoning processes even after it overfits the training data. Interestingly, for the 1.2k DeepScaleR subset, the frequency of reflection slightly decreases as the training progresses, matching the decreasing response length.

### 3.3 1/Few-shot RLVR on Other Models/Algorithms

We further investigate whether 1(few)-shot RLVR is feasible for other models and RL algorithms. We consider setup mentioned in Sec. 3.1 , and the results are shown in Tab. 4 (Detailed results on each benchmark are in Appendix C.1 ). We can see (1) for Qwen2.5-Math-7B, 1-shot RLVR with π 1 \pi_{1} improves average performance by 17.8% (5.9% higher than format-reward baseline), and 4-shot RLVR performs as well as RLVR with DSR-sub. Moreover, { π 1 , … , π 16 \pi_{1},\ldots,\pi_{16} } performs better than the subset consisting of 16 randomly sampled examples. (2) For Llama-3.2-3B-Instruct, the absolute gain from RLVR is smaller, but 1(few)-shot RLVR still matches or surpasses (e.g., { π 1 , π 13 } \{\pi_{1},\pi_{13}\} ) the performance of full-set RLVR. We also show the instability of the RLVR process on Llama-3.2-3B-Instruct in Appendix C.1 . (3) RLVR with π 1 \pi_{1} using PPO also works for Qwen2.5-Math-1.5B with PPO. (4) For DeepSeek-R1-Distill-Qwen-1.5B, the performance gap between few-shot and full-set RLVR is larger. Nevertheless, few-shot RLVE still yield improvement. More results are in Appendix C .

## 4 Analysis

In this section, we concentrate on exploring the potential mechanisms that allow RLVR to work with only one or a few examples. We hope the following analyses can provide some insight for future works. Additional experiments and discussions about the format correction (Appendix C.2.3 ), prompt modification (Appendix C.2.5 ) and the reasoning capabilities of base models (Appendix D ) are included in supplementary materials.

### 4.1 Ablation Study: Policy Gradient Loss is the Main Contributor, and Entropy Loss Further Improve Post-Saturation Generalization

As discussed in Sec. 3.2.2 , 1-shot RLVR shows the property of post-saturation generalization. This phenomenon is similar to “grokking” [ 36 , 37 ] , which shows that neural networks first memorize/overfit the training data but still perform poorly on the test set, while suddenly improve generalization after many training steps. A natural question is raised: Is the performance gain from 1-shot RLVR related to the “grokking” phenomenon? To answer this question, noting “grokking” is strongly affected by regularization [ 36 , 38 , 39 , 40 , 41 ] like weight decay, we conduct an ablation study by removing or changing the components of the loss function one by one to see how each of them contributes to the improvement.

The results are shown in Tab. 5 (Test curves are in Appendix C.2.1 ). We see that if we only add policy gradient loss (Row 2) with π 1 \pi_{1} , we already get results close to that of the full loss training (Row 5). In addition, further adding weight decay (Row 3) and KL divergence loss (Row 4) has no significant impact on model performance, while adding entropy loss (Row 5) can further bring 4.0% improvement for MATH500 and 2.5% for AIME24. Here we need to be careful about the weight of the entropy loss, as a too large coefficient (Row 6) might make the training more unstable. These observations support that the feasibility of 1(few)-shot RLVR is mainly attributed to policy gradient loss, rather than weight decay, distinguishing it from “grokking” , which should be significantly affected by weight decay. To double check this, we show that only adding weight decay and KL divergence (Row 8) has little influence on model performance, while using only policy gradient loss and entropy loss (Row 7) behaves almost the same as the full GRPO loss.

Moreover, we also argue that encouraging greater diversity in model outputs—for instance, adding proper entropy loss — can enhance post-saturation generalization in 1-shot RLVR. As shown in Fig. 5 , without entropy loss, model performance under 1-shot RLVR shows limited improvement beyond step 150, coinciding with the point at which training accuracy saturates (Fig. 2 , Left). By adding entropy loss, the model achieves an average improvement of 2.3%, and further increasing the temperature to t = 1.0 t=1.0 yields an additional 0.8% gain. More discussions about entropy loss and post-saturation generalization are in Appendix C.2.2 .

### 4.2 Entropy-Loss-Only Training & Label Correctness

In Tab. 3 , we find that when using π 1207 \pi_{1207} and π 1208 \pi_{1208} , it is difficult for model to output the ground truth label and receive rewards during 1-shot RLVR training, resulting in a very sparse policy gradient signal. Nevertheless, they still outperform the base model, although their performance remains lower than that of the format-reward baseline. To investigate this, we remove the policy loss from the full GRPO loss (Tab. 5 , Row 9) or even retain only the entropy loss (Row 10), and again observe similar improvement. Furthermore, this phenomenon also happens on Qwen2.5-Math-7B and Llama-3.2-3B-Instruct, although only improve at the first several steps. These results implies entropy loss may independently contribute to performance gains from format correction, which, although much smaller than those from policy loss, are still nontrivial.

Moreover, we conduct an experiment by altering the label to (1) the correct one (“12.7,” Row 11), (2) an incorrect one that model can still overfit (“4,” Row 12), and (3) an incorrect one that the model can neither guess nor overfit (“9292725,” Row 13). We compare them with (4) the original label (“12.8,” Row 5). Interestingly, we find the performance rankings are (1) ≈ \approx (4) > > (3) > > (2). This suggests that slight inaccuracies in the label do not significantly impair 1-shot RLVR performance. However, if the incorrect label deviates substantially while remaining guessable and overfittable, the resulting performance can be even worse than using a completely incorrect and unguessable label, which behaves similarly to training with entropy loss alone (Row 10). In Appendix C.2.4 , we also discuss label robustness on full-set RLVR by showing that if too many data in the dataset are assigned random wrong labels, full-set RLVR can perform worse than 1-shot RLVR.

## 5 Conclusion

In this work, we show that 1-shot RLVR is sufficient to trigger substantial improvements in reasoning tasks, even matching the performance of RLVR with thousands of examples. The empirical results reveal not only improved task performance but also additional observations such as post-saturation generalization, cross-category generalization, more frequent self-reflection and also additional analysis. These findings suggest that the reasoning capability of the model is already buried in some base models, and encouraging exploration on a very small amount of data is capable of generating useful RL training signals for igniting these LLM’s reasoning capability. It also demonstrates the anti-overfitting property of the RLVR algorithm with zero-mean advantage, as we can train on a single example millions of times without performance degradation. Our work also emphasizes the importance of better selection and collection of data for RLVR. We discuss directions for future work in Appendix D.4 , and also discuss limitations in Appendix D.1 .

## 6 Acknoledgements

We thank Lifan Yuan, Hamish Ivison, Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Pang Wei Koh, Kaixuan Huang, Mickel Liu, Jacqueline He, Noah Smith, Jiachen T. Wang, Yifang Chen, and Weijia Shi for very constructive discussions. YW and ZZ acknowledge the support of Amazon AI Ph.D. Fellowship. SSD acknowledges the support of NSF IIS-2110170, NSF DMS-2134106, NSF CCF-2212261, NSF IIS-2143493, NSF CCF-2019844, NSF IIS-2229881, and the Sloan Research Fellowship.

## Appendix A Related Work

Reinforcement Learning with Verifiable Reward (RLVR). RLVR, where the reward is computed by a rule-based verification function, has been shown to be effective in improving the reasoning capabilities of LLMs. The most common practice of RLVR when applying reinforcement learning to LLMs on mathematical reasoning datasets is to use answer matching: the reward function outputs a binary signal based on if the model’s answer matches the gold reference answer [ 4 , 5 , 2 , 3 , 42 , 43 , 44 ] . This reward design avoids the need for outcome-based or process-based reward models, offering a simple yet effective approach. The success of RLVR is also supported by advancements in RL algorithms, including value function optimization or detail optimization in PPO [ 7 ] (e.g., VinePPO [ 9 ] , VC-PPO [ 10 ] , VAPO [ 12 ] ), stabilization and acceleration of GRPO [ 2 ] (e.g., DAPO [ 11 ] , Dr. GRPO [ 13 ] , GRPO+ [ 14 ] , SRPO [ 16 ] ), and integration of various components (e.g., REINFORCE++ [ 15 ] ). There are also some recent works that focus on RLVR with minimal human supervision (without using labeled data or even problems), such as Absolute-Zero [ 45 ] , EMPO [ 46 ] , and TTRL [ 47 ] .