Reinforcement Learning for Reasoning in Large Language Models with One Training Example — part 5/5

Cleaned-text code points [61937, 67126). Bibliography omitted here; source.md includes it.

Currently, there are no specific data selection methods for RLVR except LIMR [ 19 ] . Note that 1-shot RLVR allows for evaluating each example individually, it will be helpful for assessing the data value, and thus help to design better data selection strategy. What’s more, noting that different examples can have large differences in stimulating LLM reasoning capability (Tab. 3 ), it may be necessary to find out what kind of data is more useful for RLVR, which is critical for the RLVR data collection stage. It’s worth mentioning that our work does not mean scaling RLVR datasets is useless , but it emphasizes the importance of better selection and collection of data for RLVR.

##### Understanding 1-shot RLVR and Post-saturation Generalization

A rigorous understanding for the feasibility of 1-shot LLM RLVR and post-saturation generalization is still unclear. We think that one possible hypothesis is that the policy loss on the learned examples plays a role as “implicit regularization” of RLVR when the model tries to explore more diverse output strategies under the encouragement of entropy loss or larger rollout temperature. It will punish the exploration patterns that make the model fail to answer the learned data, and thus provide a verification for exploration. It’s interesting to explore if the phenomenon has relevance to Double Descent [ 65 ] or the implicit regularization from SGD [ 66 , 67 ] , as 1-shot RLVR on π 13 \pi_{13} (Fig. 2 , middle) shows a test curve similar to Double Descent. We leave the rigorous analysis of this phenomenon for future works, and we believe that can help us to comprehend what happens in the RLVR process.

##### Importance of Exploration.

In Sec. 4.1 , we also highlight the importance of entropy loss in 1-shot RLVR, and note that a more thorough explanation of why training with only entropy loss can enhance model performance remains an interesting direction for future work (Sec. 4.2 ). Relatedly, entropy loss has also received increasing attention from the community, with recent works discussing its dynamics [ 68 , 47 , 60 ] or proposing improved algorithms from the perspective of entropy [ 46 ] . Moreover, we believe a broader and more important insight for these is that encouraging the model to explore more diverse outputs within the solution space is critical, as it may significantly impact the model’s generalization to downstream tasks [ 69 ] . Adding entropy loss is merely one possible approach to achieve this goal and may not necessarily be the optimal solution. As shown in our paper and previous work [ 60 ] , the effectiveness of entropy loss is sensitive to the choice of coefficient, which could limit its applicability in larger-scale experiments. We believe that discovering better strategies to promote exploration could further enhance the effectiveness of RLVR.

##### Other Applications.

In this paper, we focus primarily on mathematical reasoning data; however, it is also important to evaluate the efficacy of 1-shot RLVR in other domains, such as code generation or tasks without verifiable rewards. Moreover, investigating methodologies to further improve few-shot RLVR performance under diverse data-constrained scenarios represents a valuable direction. Examining the label robustness of RLVR, as discussed in Sec. 4.2 , likewise merits further exploration. Finally, these observations may motivate the development of additional evaluation sets to better assess differences between 1-shot and full-set RLVR on mathematical or other reasoning tasks.

## Appendix E Example Details

In the main paper, we show the details of π 1 \pi_{1} . Another useful example π 13 \pi_{13} is shown in Tab. 21 . Here we mention that π 13 \pi_{13} is a geometry problem and its answer is precise. And similar to π 1 \pi_{1} , the initial base model still has 21.9% of outputs successfully obtaining 4 3 \frac{4}{3} in 128 samplings.

Besides, Tab. 22 through 42 in the supplementary material provide detailed information for each example used in our experiments and for all other examples in { π 1 , … , π 17 \pi_{1},\ldots,\pi_{17} }. Each table contains the specific prompt and corresponding ground truth label for an individual example.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .