Reinforcement Learning for Reasoning in Large Language Models with One Training Example — part 3/5

Cleaned-text code points [30692, 46485). Bibliography omitted here; source.md includes it.

Data Selection for LLM Post-Training. The problem of data selection for LLM post-training has been extensively studied in prior work [ 48 ] , with most efforts focusing on data selection for supervised fine-tuning (instruction tuning). These approaches include LLM-based quality assessment [ 49 ] , leveraging features from model computation [ 50 ] , gradient-based selection [ 51 ] , and more. Another line of work [ 52 , 53 , 54 ] explores data selection for human preference data in Reinforcement Learning from Human Feedback (RLHF) [ 55 ] . Data selection for RLVR remains relatively unexplored. One attempt is LIMR [ 19 ] , which selects 1.4k examples from an 8.5k full set for RLVR to match performance; however, unlike our work, they do not push the limits of training set size to the extreme case of just a single example. Another closely related concurrent work [ 56 ] shows that RLVR using PPO with only 4 examples can already yield very significant improvements; however, they do not systematically explore this observation, nor do they demonstrate that such an extremely small training set can actually match the performance of using the full dataset.

## Appendix B Experiment Setup

### B.1 Details of Loss Function

As said in the main paper, we contain three components in the GRPO loss function following verl [ 22 ] pipeline: policy gradient loss, KL divergence, and entropy loss. Details are as follows. For each question q q sampled from the Question set P ⁡ ( Q ) P(Q) , GRPO samples a group of outputs { o 1 , o 2 , … , o G } \{o_{1},o_{2},\dots,o_{G}\} from the old policy model π θ old \pi_{\theta_{\text{old}}} , and then optimizes the policy model π θ \pi_{\theta} by minimizing the following loss function: ℒ GRPO ​ ( θ ) \displaystyle\mathcal{L}_{\text{GRPO}}(\theta) = 𝔼 q ∼ P ⁡ ( Q ) { o i } i = 1 G ∼ π θ old ​ ( O | q ) ​ [ ℒ PG-GRPO ′ ​ ( ⋅ , θ ) + β ​ ℒ KL ′ ​ ( ⋅ , θ , θ ref ) + α ​ ℒ Entropy ′ ​ ( ⋅ , θ ) ] , \displaystyle=\mathbb{E}_{\begin{subarray}{c}q\sim P(Q)\\ \{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(O|q)\end{subarray}}\ \Bigg[\mathcal{L}^{\prime}_{\text{PG-GRPO}}(\cdot,\theta)+\beta\mathcal{L}^{\prime}_{\text{KL}}(\cdot,\theta,\theta_{\text{ref}})+\alpha\mathcal{L}^{\prime}_{\text{Entropy}}(\cdot,\theta)\Bigg], (3) where β \beta and α \alpha are hyper-parameters (in general β > 0 \beta>0 , α < 0 \alpha<0 ), and “ ⋅ \cdot ” is the abbreviation of sampled prompt-responses: { q , { o i } i = 1 G } \{q,\{o_{i}\}_{i=1}^{G}\} . The policy gradient loss and KL divergence loss are: ℒ PG-GRPO ′ ​ ( q , { o i } i = 1 G , θ ) \displaystyle\mathcal{L}^{\prime}_{\text{PG-GRPO}}(q,\{o_{i}\}_{i=1}^{G},\theta) = − 1 G ∑ i = 1 G ( min ( π θ ​ ( o i | q ) π θ old ​ ( o i | q ) A i , clip ( π θ ​ ( o i | q ) π θ old ​ ( o i | q ) , 1 − ε , 1 + ε ) A i ) ) \displaystyle=-\frac{1}{G}\sum_{i=1}^{G}\Big(\min\Big(\frac{\pi_{\theta}(o_{i}|q)}{\pi_{\theta_{\text{old}}}(o_{i}|q)}A_{i},\,\operatorname{clip}\bigl(\tfrac{\pi_{\theta}(o_{i}|q)}{\pi_{\theta_{\text{old}}}(o_{i}|q)},1-\varepsilon,\,1+\varepsilon\bigr)A_{i}\Big)\Big) (4) ℒ KL ′ ​ ( q , { o i } i = 1 G , θ , θ ref ) \displaystyle\mathcal{L}^{\prime}_{\text{KL}}(q,\{o_{i}\}_{i=1}^{G},\theta,\theta_{\text{ref}}) = 𝔻 KL ( π θ ∥ π θ ref ) = π θ ref ​ ( o i | q ) π θ ​ ( o i | q ) − log π θ ref ​ ( o i | q ) π θ ​ ( o i | q ) − 1 , \displaystyle=\mathbb{D}_{\mathrm{KL}}\!\left(\pi_{\theta}\|\pi_{\theta_{\text{ref}}}\right)=\frac{\pi_{\theta_{\text{ref}}}(o_{i}|q)}{\pi_{\theta}(o_{i}|q)}-\log\frac{\pi_{\theta_{\text{ref}}}(o_{i}|q)}{\pi_{\theta}(o_{i}|q)}-1, (5) Here θ ref \theta_{\text{ref}} is the reference model, ε \varepsilon is a hyper-parameter of clipping threshold. Notably, we use the approximation formulation of KL divergence [ 57 ] , which is widely used in previous works [ 8 , 2 ] . Besides, A i A_{i} is the group-normalized advantage defined below. A i = r i − mean ⁡ ( { r 1 , r 2 , … , r G } ) std ⁡ ( { r 1 , r 2 , … , r G } ) . i ∈ [ G ] A_{i}=\frac{r_{i}-\operatorname{mean}\bigl(\{r_{1},r_{2},\dots,r_{G}\}\bigr)}{\operatorname{std}\bigl(\{r_{1},r_{2},\dots,r_{G}\}\bigr)}.\quad i\in[G] (6) Since we focus on math questions, we let the reward r i r_{i} be the 0-1 accuracy score, and r i r_{i} is 1 if and only if the response o i o_{i} gets the correct answer to the question q q . What’s more, the entropy loss ℒ Entropy ′ \mathcal{L}^{\prime}_{\text{Entropy}} calculates the average per-token entropy of the responses, and its coefficient α < 0 \alpha<0 implies the encouragement of more diverse responses.

The details of entropy loss are as follows. For each query q q and set of outputs { o i } i = 1 G \{o_{i}\}_{i=1}^{G} , the model produces logits X X that determine the policy distribution π θ \pi_{\theta} . These logits X X are the direct computational link between inputs q q and outputs o o - specifically, the model processes q q to generate logits X X , which after softmax normalization give the probabilities used to sample each token in the outputs o o . The entropy loss is formally defined below. ℒ Entropy ′ ​ ( q , { o i } i = 1 G , θ ) = ∑ b , s M b , s ⋅ H b , s ​ ( X ) ∑ b , s M b , s \displaystyle\mathcal{L}^{\prime}_{\text{Entropy}}(q,\{o_{i}\}_{i=1}^{G},\theta)=\frac{\sum_{b,s}M_{b,s}\cdot H_{b,s}(X)}{\sum_{b,s}M_{b,s}} (7) Here M b , s M_{b,s} represents the response mask indicating which tokens contribute to the loss calculation (excluding padding and irrelevant tokens), with b b indexing the batch dimension and s s indexing the sequence position. The entropy H b , s ​ ( X ) H_{b,s}(X) is computed from the model’s logits X X : H b , s ​ ( X ) = log ⁡ ( ∑ v e X b , s , v ) − ∑ v p b , s , v ⋅ X b , s , v H_{b,s}(X)=\log(\sum_{v}e^{X_{b,s,v}})-\sum_{v}p_{b,s,v}\cdot X_{b,s,v} (8) where v v indexes over the vocabulary tokens (i.e., the possible output tokens from the model’s vocabulary), and the probability distribution is given by p b , s , v = softmax ​ ( X b , s ) v = e X b , s , v ∑ v ′ e X b , s , v ′ p_{b,s,v}=\text{softmax}(X_{b,s})_{v}=\frac{e^{X_{b,s,v}}}{\sum_{v^{\prime}}e^{X_{b,s,v^{\prime}}}} .

### B.2 Training Dataset

##### DeepScaleR-sub.

DeepScaleR-Preview- Dataset [ 18 ] consists of approximately 40,000 unique mathematics problem-answer pairs from AIME (1984-2023), AMC (pre-2023), and other sources including Omni-MATH [ 58 ] and Still [ 59 ] . The data processing pipeline includes extracting answers using Gemini-1.5-Pro-002, removing duplicate problems through RAG with Sentence-Transformers embeddings, and filtering out questions that cannot be evaluated using SymPy to maintain a clean training set. We randomly select a subset that contains 1,209 examples referred to as "DSR-sub".

##### MATH.

Introduced in [ 27 ] , this dataset contains 12,500 challenging competition mathematics problems designed to measure advanced problem-solving capabilities in machine learning models. Unlike standard mathematical collections, MATH features complex problems from high school mathematics competitions spanning subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus, with each problem assigned a difficulty level from 1 to 5 and accompanied by detailed step-by-step solutions. It’s partitioned into a training subset comprising 7,500 problems (60%) and a test subset containing 5,000 problems (40%).

### B.3 Evaulation Dataset

All evaluation sets are drawn from the Qwen2.5-Math evaluation repository 5 5 5 https://github.com/QwenLM/Qwen2.5-Math , with the exception of AIME2025 6 6 6 https://huggingface.co/datasets/opencompass/AIME2025 . We summarize their details as follows:

##### MATH500.

MATH500, developed by OpenAI [ 29 ] , comprises a carefully curated selection of 500 problems extracted exclusively from the test partition (n=5,000) of the MATH benchmark [ 27 ] . It is smaller, more focused, and designed for efficient evaluation.

##### AIME 2024/2025.

The AIME 2024 and 2025 datasets are specialized benchmark collections, each consisting of 30 problems from the 2024 and 2025 American Invitational Mathematics Examination (AIME) I and II, respectively [ 30 ] .

##### AMC 2023.

AMC 2023 dataset consists of 40 problems, selected from two challenging mathematics competitions (AMC 12A and 12B) for students grades 12 and under across the United States [ 31 ] . These AMC 12 evaluates problem-solving abilities in secondary school mathematics, covering topics such as arithmetic, algebra, combinatorics, geometry, number theory, and probability, with all problems solvable without calculus.

##### Minerva Math.

Implicitly introduced in the paper "Solving Quantitative Reasoning Problems with Language Models" [ 32 ] as OCWCourses, Minerva Math consists of 272 undergraduate-level STEM problems harvested from MIT’s OpenCourseWare, specifically designed to evaluate multi-step scientific reasoning capabilities in language models. Problems were carefully curated from courses including solid-state chemistry, information and entropy, differential equations, and special relativity, with each problem modified to be self-contained with clearly-delineated answers that are automatically verifiable through either numeric (191 problems) or symbolic solutions (81 problems).

##### OlympiadBench.

OlympiadBench [ 33 ] is a large-scale, bilingual, and multimodal benchmark designed to evaluate advanced mathematical and physical reasoning in AI systems. It contains 8,476 Olympiad-level problems, sourced from competitions and national exams, with expert-annotated step-by-step solutions. The subset we use for evaluation consists of 675 open-ended text-only math competition problems in English.

We also consider other non-mathematical reasoning tasks: ARC-Challenge and ARC-Easy [ 34 ] .

##### ARC-Challenge/Easy.

The ARC-Challenge benchmark represents a subset of 2,590 demanding science examination questions drawn from the broader ARC (AI2 Reasoning Challenge) [ 34 ] collection, specifically selected because traditional information retrieval and word co-occurrence methods fail to solve them correctly. This challenging evaluation benchmark features exclusively text-based, English-language multiple-choice questions (typically with four possible answers) spanning diverse grade levels, designed to assess science reasoning capabilities rather than simple pattern matching or information retrieval. The complementary ARC-Easy [ 34 ] subset contains 5197 questions solvable through simpler approaches. We use 1.17k test split for ARC-Challenge evaluation and 2.38k test split for ARC-Easy evaluation, respectively.

### B.4 More Training Details

For DeepSeek-R1-Distill-Qwen-1.5B, we let the maximum response length be 8192, following the setup of stage 1 in DeepScaleR [ 18 ] . The learning rate is set to 1e-6. The coefficient of weight decay is set to 0.01 by default. We store the model checkpoint every 20 steps for evaluation, and use 8 A100 GPUs for each experiment. For Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Llama-3.2-3B-Instruct, and DeepSeek-R1-Distill-Qwen-1.5B, we train for 2000, 1000, 1000, and 1200 steps, respectively, unless the model has already shown a significant drop in performance. We use the same approach as DeepScaleR [ 18 ] (whose repository is also derived from the verl) to save the model in safetensor format to facilitate evaluation.

### B.5 More Evaluation Details

In evaluation, the maximum number of generated tokens is set to be 3072 by default. For Qwen-based models, we use the “ qwen25-math-cot ” prompt template in evaluation. For Llama and distilled models, we use their original chat templates. We set the evaluation seed to 0 and top_p to 1 by default. For evaluation on DeepSeek-R1-Distill-Qwen-1.5B, following DeepSeek-R1 [ 2 ] and DeepScaleR [ 18 ] , we set the temperature to 0.6 and top_p to 0.95, and use avg@16 for MATH500, Minerva Math, and OlympiadBench, and avg@64 for AIME24, AIME25, and AMC23. Since our training length is 8192, we provide results for both 8192 (8k) and 32768 (32k) evaluation lengths (Appendix C.1.6 ). By default, we report the performance of the checkpoint that obtains the best average performance on 6 benchmarks. But in Sec. 3.2.3 and Sec. 4.1 , since we only evaluate MATH500 and AIME2024, we report the best model performance on each benchmark separately, i.e., the best MATH500 checkpoint and best AIME2024 checkpoint can be different (This will not influence our results, as in Tab. 9 and Tab. 11 , we still obtain similar conclusions as in main paper.) We use 4 GPUs for the evaluation. Finally we mention that there are slightly performance difference on initial model caused by numerical precision, but it does not influence our conclusions (Appendix B.6 ).

### B.6 Performance Difference on Initial Model

We mention that there is a precision inconsistency between models downloaded from Hugging Face repositories and initial checkpoints saved by the verl/deepscaler reinforcement learning pipeline in Tab. 7 . This discrepancy arises from the verl/DeepScaleR pipeline saving checkpoints with float32 precision, whereas the original base models from Hugging Face utilize bfloat16 precision.

The root cause appears to be in the model initialization process within the verl framework. The fsdp_workers.py 7 7 7 https://github.com/volcengine/verl/blob/main/verl/workers/fsdp_workers.py file in the verl codebase reveals that models are deliberately created in float32 precision during initialization, as noted in the code comment: "note that we have to create model in fp32. Otherwise, the optimizer is in bf16, which is incorrect". This design choice was likely made to ensure optimizer stability during training. When examining the checkpoint saving process, the precision setting from initialization appears to be preserved, resulting in saved checkpoints retaining float32 precision rather than the original bfloat16 precision of the base model.

Our empirical investigation demonstrates that modifying the torch_dtype parameter in the saved config.json file to match the base model’s precision (specifically, changing from float32 to bfloat16 ) successfully resolves the observed numerical inconsistency. Related issues are documented in the community 8 8 8 https://github.com/volcengine/verl/issues/296 , and we adopt the default settings of the verl pipeline in our experiments.

## Appendix C Evaluation Result

### C.1 Main Experiments

#### C.1.1 Detailed performance on Qwen2.5-Math-1.5B.

In Tab. 8 , we show the detailed performance that shown in Fig. 1 . Results are reported for the checkpoint achieving the best average performance.

#### C.1.2 Detailed Performance on More Models and Training Examples.

In Tab. 10 , we also show the 1(few)-shot RLVR results on the base model (Qwen2.5-1.5B [ 24 ] ) and instruction model (Qwen2.5-Math-1.5B-Instruct [ 25 ] ). More detailed test curves are shown in Fig. 10 and Fig. 11 . We can see that (1) for Qwen2.5-1.5B, the gap between 1-shot RLVR with π 1 \pi_{1} and full-set RLVR is larger, but the former still improves model performance significantly (e.g., MATH500: 3.2% to 43.6%), and 16-shot RLVR works very closely to full-set RLVR. (2) for Qwen2.5-Math-1.5B-Instruct, both full-set RLVR and 1-shot RLVR have limited improvement as the initial model already has good performance. Interestingly, as shown in Fig. 11 , we observe that 1-shot RLVR is more stable than full-set RLVR.

Besides, we also consider other single training examples like π 605 \pi_{605} and π 1209 \pi_{1209} on Qwen2.5-Math-7B. We can see that they behave relatively worse than π 1 \pi_{1} , and 16-shot RLVR provides a more consistent approach to closing the performance gap relative to full-set RLVR.

#### C.1.3 Detailed performance with best per-benchmark results