Reinforcement Learning for Reasoning in Large Language Models with One Training Example — part 1/5

Cleaned-text code points [0, 14979). Bibliography omitted here; source.md includes it.

# Reinforcement Learning for Reasoning in Large Language Models with One Training Example

###### Abstract

We show that reinforcement learning with verifiable reward using one training example ( 1-shot RLVR ) is effective in incentivizing the mathematical reasoning capabilities of large language models (LLMs). Applying RLVR to the base model Qwen2.5-Math-1.5B, we identify a single example that elevates model performance on MATH500 from 36.0% to 73.6% (8.6% improvement beyond format correction), and improves the average performance across six common mathematical reasoning benchmarks from 17.6% to 35.7% (7.0% non-format gain). This result matches the performance obtained using the 1.2k DeepScaleR subset (MATH500: 73.6%, average: 35.9%), which contains the aforementioned example. Furthermore, RLVR with only two examples even slightly exceeds these results (MATH500: 74.8%, average: 36.6%). Similar substantial improvements are observed across various models (Qwen2.5-Math-7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and PPO), and different math examples. In addition, we identify some interesting phenomena during 1-shot RLVR, including cross-category generalization, increased frequency of self-reflection, and sustained test performance improvement even after the training accuracy has saturated, a phenomenon we term post-saturation generalization . Moreover, we verify that the effectiveness of 1-shot RLVR primarily arises from the policy gradient loss, distinguishing it from the "grokking" phenomenon. We also show the critical role of promoting exploration (e.g., by incorporating entropy loss with an appropriate coefficient) in 1-shot RLVR training. We also further discuss related observations about format correction, label robustness and prompt modification. These findings can inspire future work on RLVR efficiency and encourage a re-examination of recent progress and the underlying mechanisms in RLVR. Our code, models, and data are open source at https://github.com/ypwang61/One-Shot-RLVR .

## 1 Introduction

Recently, significant progress has been achieved in enhancing the reasoning capabilities of large language models (LLMs), including OpenAI-o1 [ 1 ] , DeepSeek-R1 [ 2 ] , and Kimi-1.5 [ 3 ] , particularly for complex mathematical tasks. A key method contributing to these advancements is Reinforcement Learning with Verifiable Reward (RLVR) [ 4 , 5 , 2 , 3 ] , which commonly employs reinforcement learning on an LLM with a rule-based outcome reward, such as a binary reward indicating the correctness of the model’s final answer to a math problem. Several intriguing empirical phenomena have been observed in RLVR, such as the stimulation or enhancement of specific cognitive behaviors [ 6 ] (e.g., self-reflection) and improved generalization across various downstream tasks [ 5 , 2 , 3 ] .

Currently, substantial efforts are directed toward refining RL algorithms (e.g., PPO [ 7 ] and GRPO [ 8 ] ) to further enhance RLVR’s performance and stability [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] . Conversely, data-centric aspects of RLVR remain relatively underexplored. Although several studies attempt to curate high-quality mathematical reasoning datasets [ 17 , 18 , 11 ] , there is relatively limited exploration into the specific role of data in RLVR. Thus, critical questions remain open: How much data is truly necessary? What data is most effective? How do the quality and quantity of the training data relate to observed empirical phenomena (e.g., self-reflection and robust generalization)? The most relevant study to these problems is LIMR [ 19 ] , which proposed a metric called learning impact measurement (LIM) to evaluate the effectiveness of training examples. Using the LIM score, they maintain model performance while reducing the number of training examples by sixfold. However, this study does not explore how aggressively the RLVR training dataset can be reduced. Motivated by these considerations, in this paper, we specifically investigate the following research question:

"To what extent can we reduce the training dataset for RLVR while maintaining comparable performance compared to using the full dataset?"

We empirically demonstrate that, surprisingly, the training dataset for RLVR can be reduced to as little as ONE example! This finding supports recent claims that base models already possess significant reasoning capabilities [ 13 , 20 , 6 , 21 ] , and further shows that a single example is sufficient to substantially enhance the base model’s mathematical performance. We refer to this setup as 1-shot RLVR . We summarize our contributions and findings below:

• We find that selecting one specific example as the training dataset can achieve similar downstream performance to that of the 1.2k DeepScaleR subset (DSR-sub) containing that example. Specifically, this improves the Qwen2.5-Math-1.5B model from 36.0% to 73.6% on MATH500, and from 17.6% to 35.7% on average across 6 mathematical reasoning benchmarks, including non-trivial improvements beyond format correction (Fig. 1 ). Notably, these two examples are relatively easy for the base model, which can solve them with high probability without any training (Sec. 3.2.1 ). Additionally, 1-shot RLVR on math examples can improve model performance on non-mathematical reasoning tasks, even outperforming full-set RLVR (Tab. 1 ).

• We confirm the effectiveness of 1(few)-shot RLVR across different base models (Qwen2.5-Math-1.5/7B, Llama3.2-3B-Instruct), models distilled from long Chain-of-Thought (CoT) data (DeepSeek-R1-Distill-Qwen-1.5B), and different RL algorithms (GRPO, PPO).

• We highlight an intriguing phenomenon in 1-shot RLVR: post-saturation generalization. Specifically, the training accuracy on the single example rapidly approaches 100%, yet the model’s test accuracy continues to improve. Moreover, despite using only one training example, overfitting does not occur until after approximately 1.4k training steps. Even post-overfitting, while the model’s reasoning outputs for the training example become incomprehensible multilingual gibberish mixed with correct solutions, its test performance remains strong, and the reasoning outputs for the test examples remain human-interpretable.

• In addition, we demonstrate the following phenomena: (1) 1-shot RLVR is viable for many examples in the full dataset when each example is individually used for training. We also discuss its connection with format correction in Appendix C.2.3 . (2) 1-shot RLVR enables cross-category generalization: training on a single example from one category (e.g., Geometry) often enhances performance in other categories (e.g., Algebra, Number Theory). (3) As 1-shot RLVR training progresses, both the response length for the training example and the frequency of self-reflective terms in downstream tasks increase.

• Through ablation studies, we show that policy gradient loss primarily drives the improvements observed in 1-shot RLVR, distinguishing it from “grokking”, which heavily depends on regularization methods like weight decay. Additionally, we emphasize the importance of promoting diverse exploration in model outputs, showing that adding an entropy loss with an appropriate coefficient further enhances performance.

• Lastly, we find that employing entropy loss alone, even without any outcome reward, yields a performance boost, although it remains weaker than the format-reward baseline. Similar improvements are observed for Qwen2.5-Math-7B and Llama-3.2-3B-Instruct. We also discuss label robustness and prompt modification in RLVR (Appendix C.2 ).

## 2 Preliminary

##### RL Loss Function.

In this paper, we adopt GRPO [ 8 , 2 ] as the RL algorithm for LLMs unless stated otherwise. We briefly introduce three main components in the loss function as below and provide more details in Appendix B.1 .

(1) Policy gradient loss : it encourages the model to produce responses with higher rewards, assigning weights according to their group-normalized advantages. Thus, better-than-average solutions are reinforced, whereas inferior ones are penalized. Since we focus on mathematical problems, the reward is defined as binary (0-1), where a reward of 1 is granted only when the outcome of the model’s response correctly matches the ground truth. We do not include the format reward when using the outcome reward , but format-reward RLVR is used as a baseline for Qwen models. Further discussion can be found in Appendix C.2.3 .

(2) KL loss : it helps to maintain general language quality by measuring the divergence between current model’s responses and those from reference model.

(3) Entropy loss [ 22 ] : applied with a negative coefficient, it incentivizes higher per-token entropy to encourage exploration and generate more diverse reasoning paths. We note that entropy loss is not strictly necessary for GRPO training, but it is included by default in verl [ 22 ] used in our experiments. Its effect on 1-shot RLVR is discussed in Sec. 4.1 .

##### Data Selection: Historical Variance Score.

To explore how extensively we can reduce the RLVR training dataset, we propose a simple data selection approach for ranking training examples. We first train the model for E E epochs on the full dataset using RLVR. Then for each example i ∈ [ N ] = { 1 , … , N } i\in[N]=\{1,\ldots,N\} , we can obtain a list of historical training accuracy L i = [ s i , 1 , … , s i , E ] L_{i}=[s_{i,1},\ldots,s_{i,E}] , which records its average training accuracy for every epoch. Note that some previous work has shown that the variance of the reward signal [ 23 ] is critical for RL training, we simply rank the data by their historical variance of training accuracy, which is directly related to the reward: v i := var ​ ( s i , 1 , … , s i , E ) v_{i}:=\text{var}(s_{i,1},\ldots,s_{i,E}) (1) Next, we define a permutation π : [ N ] → [ N ] \pi:[N]\to[N] such that v π ⁡ ( 1 ) ≥ ⋯ ≥ v π ⁡ ( N ) . v_{\pi(1)}\geq\cdots\geq v_{\pi(N)}. Under this ordering, π ⁡ ( j ) \pi(j) (denoted as π j \pi_{j} for convenience) corresponds to the example with the j j -th largest variance v i v_{i} : π j := π ⁡ ( j ) = arg ​ sort j ⁡ { v l : l ∈ [ N ] } \pi_{j}:=\pi(j)=\operatorname*{arg\,sort}_{j}\{v_{l}:l\in[N]\} (2) We then select examples according to this straightforward ranking criterion. For instance, π 1 \pi_{1} , identified by the historical variance score on Qwen2.5-Math-1.5B, performs well in 1-shot RLVR (Sec. 3.2.3 , 3.3 ). We also choose additional examples from diverse categories among { π 1 , … , π 17 } \{\pi_{1},\ldots,\pi_{17}\} and evaluate them under 1-shot RLVR (Tab. 3 ), finding that π 13 \pi_{13} likewise achieves strong performance. Importantly, we emphasize that this criterion is not necessarily optimal for selecting single examples for 1-shot RLVR 1 1 1 Nevertheless, as shown in Tab. 4 (Sec. 3.3 ), selection based on historical variance scores outperforms random selection in RLVR on Qwen2.5-Math-7B. . In fact, Tab. 3 shows that many examples, including those with moderate or low historical variance, can individually produce improvements on MATH500 when used as a single training example in RLVR. This suggests a potentially general phenomenon that is independent of the specific data selection method.

## 3 Experiments

### 3.1 Setup

Models. We by default run our experiments on Qwen2.5-Math-1.5B [ 24 , 25 ] , and also verify the effectiveness of Qwen2.5-Math-7B [ 25 ] , Llama-3.2-3B-Instruct [ 26 ] , and DeepSeek-R1-Distill-Qwen-1.5B [ 2 ] for 1-shot RLVR in Sec. 3.3 . We also include the results of Qwen2.5-1.5B and Qwen2.5-Math-1.5B-Instruct in Appendix C.1.2 .

Dataset. Due to resource limitations, we randomly select a subset consisting of 1209 examples from DeepScaleR-Preview-Dataset [ 18 ] as our instance pool (“DSR-sub”). For data selection (Sec. 2 ), as described in Sec. 2 , we first train Qwen2.5-Math-1.5B for 500 steps, and then obtain its historical variance score (Eqn. 1 ) and the corresponding ranking (Eqn. 2 ) on the examples. To avoid ambiguity, we do not change the correspondence between { π i } i = 1 1209 \{\pi_{i}\}_{i=1}^{1209} and examples for all the experiments, i.e., they are all ranked by the historical variance score of Qwen2.5-Math-1.5B. We also use the MATH [ 27 ] training set (consisting of 7500 instances) as another dataset in full RLVR to provide a comparison. More details are in Appendix B.2 .

Training. As described in Sec. 2 , we follow the verl [ 22 ] pipeline, and by default, the coefficients for KL divergence and entropy loss are β = 0.001 \beta=0.001 and α = − 0.001 \alpha=-0.001 , respectively. The training rollout temperature is set to 0.6 for vLLM [ 28 ] . The training batch size and mini-batch size are 128 2 2 2 Note that verl sets drop_last =True for training dataloader, so the dataset must be at least as large as the training batch size. To enable RLVR with very few examples, we duplicate the selected example until reaching 128 samples and store them as a new dataset. , and we sample 8 responses for each prompt. Therefore, we have 8 gradient updates for each rollout step. By default, the maximum prompt length is 1024, and the maximum response length is 3072, considering that Qwen2.5-Math-1.5B/7B’s context length are 4096. For a fairer comparison on Qwen models, we include the format-reward baseline, which assigns a reward of 1 if and only if the final answer can be parsed from the model output (see Appendix C.2.3 for details). More details are in Appendix B.4 .

Evaluation. We use the official Qwen2.5-Math evaluation pipeline [ 25 ] for our evaluation. Six widely used complex mathematical reasoning benchmarks are used in our paper: MATH500 [ 27 , 29 ] , AIME 2024 [ 30 ] , AMC 2023 [ 31 ] , Minerva Math [ 32 ] , OlympiadBench [ 33 ] , and AIME 2025 [ 30 ] . We also consider non-mathematical reasoning tasks ARC-Easy and ARC-Challenge [ 34 ] . More details about benchmarks are in Appendix B.3 . For AIME 2024, AIME 2025, and AMC 2023, which contain only 30 or 40 questions, we repeat the test set 8 times for evaluation stability and evaluate the model with temperature = 0.6, and finally report the average pass@1 ( avg@8 ) performance. And for other 3 mathematical benchmarks, we let temperature be 0. The evaluation setup for DeepSeek-R1-Distill-Qwen-1.5B and other evaluation details are provided in Appendix B.5 .

### 3.2 Observation of 1/Few-Shot RLVR

In Fig. 1 , we have found that RLVR with 1 or 2 examples can perform as well as RLVR with thousands of examples, yielding significant improvements in both format and non-format aspects. Tab. 1 further shows that 1(few)-shot RLVR with these math examples enable better generalization on non-mathematical reasoning tasks (More details are in Appendix C.1 ). To better understand this phenomenon, we provide a detailed analysis of 1-shot RLVR in this section.

#### 3.2.1 Dissection of π 1 \pi_{1} : A Not-So-Difficult Problem