Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning — part 6/6

Cleaned-text code points [78622, 90518). Bibliography omitted here; source.md includes it.

Blocksworld ( Valmeekam et al., 2023 ) is a dataset used to evaluate the planning capabilities of LLMs. Each task involves transitioning from an initial block configuration to a target configuration, which requires LLMs to generate a sequence of actions, or plan to achieve the goal. Tasks become more difficult as the required number of steps increases, since the model must reason over longer sequences and maintain correct intermediate states. To study this, we group tasks into four in-distribution difficulty levels: Trivial with 1 step, Easy with 2 steps, Medium with 4 steps, and Hard with 6 steps. Additionally, we include an out-of-distribution (OOD) split with 8-step plans to test generalization beyond the training distribution. Trivial tasks are especially simple because the model only needs to predict one correct action out of four possible choices to complete the plan. This setup allows the LLM to first grasp fundamental planning mechanics, which can then be leveraged to learn more complex multi-step tasks.

Countdown ( Gandhi et al., 2024 ) is a task where the model must reach a target value by combining given numbers using basic arithmetic operations. While the original dataset uses four numbers per instance, we extend it to create a range of difficulty levels based on the number of input numbers, mainly, Trivial (2), Easy (3), Medium (4), Hard (5), and OOD (6). As the number of inputs increases, the space of possible operation sequences grows rapidly, making it harder for the model to identify the correct combination and order of operations. In contrast, the trivial setting is extremely simple, requiring just one operation between two numbers to reach the target, allowing the model to first learn basic arithmetic before scaling to more complex multi-step problems.

MATH ( Hendrycks et al., 2021 ) is a benchmark of 7,500 training and 5,000 test problems covering high-school level mathematics, with each problem labeled from Level 1 (easiest) to Level 5 (hardest). The dataset covers topics such as algebra, geometry, number theory, and probability, including step-by-step solutions. We create a difficulty-based setup using the existing labels, specifically, Trivial (Level 1), Easy (Level 2), Medium (Level 3), Hard (Level 4), and OOD (Level 5). As difficulty increases, problems require more complex reasoning, multi-step solutions, and deeper mathematical understanding. Trivial problems are typically short and rely on basic techniques, making them ideal for teaching foundational reasoning. We use Levels 1 through 4 for training and treat Level 5 as out-of-distribution to assess generalization to the most difficult problems.

We provide the summery of the difficulty splits for each dataset in Table 9 .

### C.2 Datasets without Human Annotated Difficulties

GSM8K ( Cobbe et al., 2021 ) is a dataset of high-quality, linguistically diverse grade school math word problems designed to evaluate multi-step arithmetic reasoning. Each problem typically requires between two and eight steps involving basic arithmetic operations such as addition, subtraction, multiplication, and division. To assess performance across varying difficulty levels, we create a 4-way split into Trivial, Easy, Medium, and Hard, based on model error rates, as described in Section. 4 and illustrated in Figure. 4.2

AQuA ( Ling et al., 2017b ) is a dataset of algebraic word problems with multiple-choice answers and detailed rationales, designed to test arithmetic reasoning and symbolic manipulation. For our experiments, we randomly sample 5,000 problems for training. Similar to GSM8K, we use model error rates to define four difficulty levels, Trivial, Easy, Medium, and Hard (see Figure. 6 ).

Since both GSM8K and AQuA lack explicit difficulty annotations, we construct difficulty splits using model error rates and do not include an out-of-distribution (OOD) category for these datasets.

## Appendix D Model License Information

For our experiments, we use LLaMA 3.2 3B Instruct, Qwen 2.5 1.5B Instruct, and Qwen 2.5 3B Instruct. Qwen 2.5 1.5B Instruct is released under the Apache 2.0 License, Qwen 2.5 3B Instruct under the Qwen Research License, and LLaMA 3.2 3B Instruct under the Meta Community License.

## Appendix E Implementation Details

In this section, we provide implementation details for E2H Reasoner . We begin by describing the hardware setup, followed by training parameters, and finally the inference settings used to report the final results.

### E.1 Hardware Details

For our experiments, we use up to three 80GB NVIDIA A100 GPUs, particularly for the 3B parameter models. One GPU is dedicated to vLLM for fast inference, while the remaining two are used for model training. On average our experiments require anywhere between 16-18 hours for training.

### E.2 Training Details

We use GRPO as the policy optimization algorithm, combined with parameter-efficient fine-tuning using LoRA. All models are trained for up to 1600 GRPO steps, with hyperparameters listed in Table 10 . We adopt the reward design from Guo et al. (2025) , where outputs with the correct format receive partial rewards, and full rewards are given only when both format and answer are correct.

For fair comparison, the same configuration is used in our SFT experiments. All training is conducted using the TRL library from Hugging Face, averaged across 3 seeds. For the Self-Evolve baseline ( Chen et al., 2025 ) , we use the same hyper-parameters as reported by authors. For DAPO ( Yu et al., 2025 ) , we set n = 4 n=4 on default

### E.3 Difficulty-Based Training Split Creation for GSM8K and AQuA

Since both GSM8K and AQuA lack explicit difficulty annotations, we construct difficulty splits based on model error rates. We use CoT prompting with 1-shot in-context learning (ICL) to generate answers for each question. The prompts and ICL examples have been directly adopted from prior works ( Hao et al., 2024 ; Parashar et al., 2025 ) . The model is queried 20 times per question, and the error is computed as the fraction of incorrect responses relative to the ground truth. Based on this, samples with 0–25% error rate are labeled as trivial , 25–50% as easy , 50–75% as medium , and 75–100% as hard .

### E.4 Inference Details

For inference, we use a temperature of 0.0 in all experiments to ensure deterministic outputs and reproducibility. For pass@ k k evaluations, we enable sampling with a temperature of 0.7, top_p of 0.9, and top_k of 50, where top_k controls the number of candidate tokens considered at each decoding step.

## Appendix F Comparisons with SFT

We compare vanilla supervised fine-tuning (SFT) with RL-based post-training methods (see Table 11 ). The results show that SFT performance varies significantly across tasks. For instance, in Blocksworld ( Valmeekam et al., 2023 ) , where most problems involve fewer than four blocks, SFT performs well—the LLM learns effectively due to the small gap between the training and test distributions. This aligns with prior work suggesting that SFT is most effective when the training data is high quality and closely aligned with the downstream task ( Muennighoff et al., 2025 ; Chu et al., 2025 ) . In contrast, on tasks like Countdown, where the problem distribution is much broader, SFT performs notably worse than RL-based post-training. This gap is especially clear in the out-of-distribution (OOD) setting, where SFT fails to solve any Countdown OOD examples, unlike in Blocksworld, where its performance generalizes more successfully. We highlight these two tasks to illustrate how SFT performance can vary across distributions, in contrast to the more consistent behavior of RL-based post-training methods.

## Appendix G Additional Analysis and Ablations

### G.1 Results on More LLMs

Additional results on more LLMs are reported in this subsection.

### G.2 Results Across Gaussian Parameters

In this section, we expand on the results presented in the main paper by reporting all three parameter settings of Gaussian scheduling in Table 13 . Similarly, for Qwen 2.5 1.5B Instruct we include the results for all Gaussian scheduling variants on GSM8K and AQuA in Table 14 .

### G.3 Choice of Number of Difficulty Splits

The number of difficulty splits can be treated as a hyperparameter in E2H Reasoner , particularly for datasets without human-annotated difficulty labels such as GSM8K ( Cobbe et al., 2021 ) and AqUA ( Ling et al., 2017a ) . We set this value to 4, consistent with the range of 3 to 5 used in the curriculum learning literature ( Bengio et al., 2009 ) . As shown in Table 15 , our method remains robust across different choices of this parameter.

### G.4 DAPO + E2H

Combining E2H with DAPO consistently lowers the fraction of advantage-zero batches (see Fig. 7 ), indicating that the curriculum helps select more informative batches with difficulty better aligned to the model’s competence.

### G.5 Qualitative Analysis of Scheduling Techniques

We provide a qualitative analysis of scheduling techniques in Table 16 , highlighting their respective strengths and weaknesses. The results show that different schedulers are suited to different tasks and model scales. Inspired by the effectiveness of E2H Reasoner , we hope this motivates the community to explore more sophisticated scheduling strategies for LLM post-training.

## Appendix H LLM Usage

LLMs were used for text-refining purposes only.

## Appendix I Limitations

Despite the strengths of E2H Reasoner , it has certain limitations. Our approach uses simple and intuitive probabilistic schedulers, specifically based on Gaussian and cosine functions, which do not adapt during training. While these choices are effective, our results suggest that incorporating adaptive strategies, such as advantage-based scheduling, could offer further improvements. Comparisons with adaptive curriculum methods that aim to maximize learnability reveal an important insight: maximizing learnability does not always lead to stronger reasoning. This outcome depends on the structure and difficulty of the problems within the dataset. Combining our method with adaptive approaches presents a promising direction for future work.

## Appendix J Societal Impacts

Our work introduced E2H Reasoner , a curriculum-based reinforcement learning (RL) approach designed to enhance the reasoning capabilities of small-scale large language models (LLMs). This advancement holds significant societal implications. Enhanced reasoning in LLMs can improve decision-making processes in critical domains such as healthcare, education, and legal systems, where nuanced understanding is paramount. Moreover, by empowering smaller models, E2H Reasoner promotes broader accessibility to advanced AI capabilities. Although our tasks focus on reasoning, future extensions involving real-world interaction could pose risks, including potential misuse of language models.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .