Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning — part 3/6

Cleaned-text code points [31507, 46679). Bibliography omitted here; source.md includes it.

The geometric error and L k L_{k} allocation reflect the curriculum gradually increases in difficulty while allowing larger errors in earlier stages. Since the function f ⁡ ( x ) = x 1 − K − 1 1 − x 2 f(x)=\frac{x^{1-K}-1}{1-x^{2}} monotonically decrease for x > 1 x>1 when K K is a integer larger than 1, the final condition can be reasonably satisfied, e.g. , with K = 3 K=3 , e ∗ l = 1.4 e*l=1.4 , and m = 1.8 m=1.8 in practice.

The first half of Theorem 3.2 analysis highlights the dual benefit of curriculum design in CRL. First, by constructing intermediate distributions d 1 , … , d K − 1 d_{1},\dots,d_{K-1} close to d K d_{K} , the curriculum error term ‖ Q K ∗ − Q k ∗ ‖ \|Q^{*}_{K}-Q^{*}_{k}\| can be made small. Second, easier curriculum tasks improve estimation accuracy and yield more stable approximate greedy updates, enhancing sample efficiency and policy improvement. For the second half of Theorem 3.2 , the final condition in Eq. 2 is satisfied when curriculums are appropriately learned, the allocation of accuracy targets ϵ k \epsilon_{k} is gradually optimized across curriculum steps, and the designed curriculums with increasing difficulties effectively bridge the gap between source and target distributions. This mathematical derivation shows CRL requires fewer total samples than direct learning on the final task, aligning with experimental observations (see Appendix B ).

## 4 Experiments

We conduct experiments investigating the following research questions. RQ1: What role does task decomposition play in RL-based post-training? RQ2: How does task scheduling impact the learning process? RQ3: Can small-scale LLMs learn to reason on hard tasks?

### 4.1 Experimental Setup

We evaluate our method on a diverse set of reasoning and planning tasks, covering both datasets with and without human-annotated difficulty levels. For datasets with difficulty labels, such as Blocksworld (using plan length) ( Valmeekam et al., 2023 ) , Countdown (using number of operations) ( Gandhi et al., 2025 ) , and MATH (using problem levels) ( Hendrycks et al., 2021 ) , we use the provided annotations and define an out-of-distribution (OOD) split to assess generalization. For datasets without explicit difficulty, namely GSM8K ( Cobbe et al., 2021 ) and AQuA ( Ling et al., 2017a ) , we estimate difficulty using the zero shot error rate of the base model. Specifically, for each question in the training set, we generate 20 responses and compute the error rate as 1 − Number of Correct Responses 20 1-\frac{\text{Number of Correct Responses}}{20} and bucket the samples into trivial, easy, medium, and hard, based on quartiles Fig. 4.2 . More dataset details are in Appendix C . We conduct experiments on Qwen 2.5/1.5B Instruct ( Yang et al., 2024 ) , LLaMa 3.2 3B Instruct ( Grattafiori et al., 2024 ) . More models are included in Appendix G.1 . We use Qwen 1.5B for research questions RQ1 and RQ2.

### 4.2 Baselines

First, we report the Chain of Thought ( Wei et al., 2022 ) ( CoT ) performance for all models. All post-training experiments use GRPO ( Shao et al., 2024 ) as the reinforcement learning algorithm (see Appendix E for implementation). GRPO, by default, employs balanced scheduling over all tasks, which we use as our baseline and refer to as GRPO in Table 3 . We also assess whether models can learn directly from the most challenging examples by training only on the Hard and OOD subsets. In addition, we evaluate traditional curriculum learning ( CL ) as used by Team et al. (2025) ; Bercovich et al. (2025) . We compare against Self-Evolve ( Chen et al., 2025 ) , an adaptive curriculum baseline that samples problems with a 50 % 50\% solve rate to maximize learnability. Finally, we provide a comparison against SFT in Appendix F .

### 4.3 Experimental Results

We conduct experiments addressing the research questions listed above. We examine how task decomposition impacts LLM post-training using Qwen-1.5B-Instruct (RQ1) with a balanced scheduler (Table 1 ). We find that including trivial and easy examples helps the model build core skills. This enables effective transfer from simpler to harder tasks and better OOD performance, consistent with our view of reasoning as learning core principles and applying them to harder tasks.

Next, we examine how scheduling impacts post-training (RQ2) in Table. 2 . Balanced scheduling serves as a strong baseline but lacks structure, leading to suboptimal learning. CL schedules tasks in a fixed difficulty order, which can cause forgetting of earlier tasks or overfitting to easier ones. In our experiments, easier tasks help initiate learning due to their dense rewards, but overexposure hinders generalization to harder, sparse-reward tasks. We address this through our two schedulers, i.e., E2H-C and E2H-G (see Section 3.2 ). For tasks like MATH, where models perform reasonably well across all difficulty levels (as seen with CoT in Table 2 ), the cosine scheduling in E2H-C is effective, beginning with a focus on trivial and easy tasks and gradually shifting toward harder ones. However, on tasks like Blocksworld, where rewards are sparse since harder examples are more challenging, this leads to overfitting and degraded performance. E2H-G addresses this issue by using a Gaussian schedule that quickly decays the sampling probability of trivial and easy tasks. As shown in Fig. 4 , it provides enough exposure early on to support initial learning while rapidly shifting focus to harder examples. This prevents overfitting and improves generalization in sparse-reward settings where E2H-C struggles.

Next, we investigate whether models can learn to reason directly from difficult examples (RQ3). As shown in Table 3 , the answer is largely no. For instance, Qwen-2.5 1.5B when trained on Level 5 MATH directly, underperforms the CoT baseline on the model! This failure highlights the need for CRL methods for LLM reasoning. To this end, we compare with a CRL method, Self-Evolve (Table 3 ). Empirically, we find that its reasoning performance varies significantly across models and tasks. Self-Evolve during training, samples problems at a 50% success rate to maximize learnability and gradually reduces easier ones once this threshold is surpassed. However, being only halfway proficient at solving a problem is often not enough. For example, learning calculus requires strong command of school mathematics. In addition, implementing Self-Evolve also requires careful tuning of hyper-parameters for different models and datasets.

In contrast, E2H Reasoner guides learning scheduling tasks from easy to hard, improving generalization, as reflected in stronger OOD performance (RQ3). This is further supported by the results, where E2H Reasoner shows better performance over baselines as task difficulty rises . Note that for E2H-G we report the best numbers out of our 3 parameter settings with extensive results in Table 13 and does not need extensive hyper-parameter tuning . Finally, E2H Reasoner remains effective even without human difficulty labels by using error rates as a proxy for difficulty (Table 4 ).

In Table 5 , we compare against DAPO, which builds on GRPO by filtering out problems that are either too easy (all rewards = 0) or too hard (all rewards = 1). In principle, if DAPO were allowed to resample up to N N times (where N N is the dataset size) before forming each batch, it would always find the perfect batch, but this is computationally infeasible for training. Our E2H Reasoner mitigates this issue by gradually focusing on harder task groups through probabilistic difficulty-based scheduling. As shown in Figure 7 , combining E2H with DAPO reduces the the fraction of batches with zero advantage during training, indicating that it selects prompts whose overall difficulty is better aligned with the model’s current competence. Taken together, these results show that E2H and DAPO are complementary; E2H shapes the difficulty conditioned sampling distribution that DAPO resamples from, and their combination yields the strongest performance across benchmarks.

Finally, we conduct an extended evaluation of our models trained on the MATH dataset using E2H and vanilla GRPO on two challenging evaluation benchmarks, AIME24 ( Veeraboina, 2023 ) and OlympiadBench ( He et al., 2024 ) . E2H demonstrates stronger generalization to both tasks in Table 7

## 5 Conclusion

We introduce the E2H Reasoner (E2H), a CRL-based method for LLM post-training. E2H enables models to learn tasks they initially failed at by scheduling tasks from easy to hard. E2H challenges the assumption that small LLMs cannot reason and demonstrates strong empirical performance supported by theoretical analysis, offering convergence guarantees and improved sample efficiency over direct RL. In summary, E2H provides a scalable, theoretically grounded, and practical method for LLM reasoning.

## Appendix

## Appendix A Proofs

We now present the proofs for the main theoretical results stated above. Our analysis closely follows the derivation style of Chen and Maguluri (2025) ; Scherrer (2014) , extending it to the curriculum setting.

### A.1 Proof of Theorem 3.1

###### Proof.

We aim to derive a tight bound for the final performance error in task 𝒯 K \mathcal{T}_{K} : ℰ K := ‖ Q K ∗ − Q K π K ‖ d K , \mathcal{E}_{K}:=\|Q^{*}_{K}-Q^{\pi_{K}}_{K}\|_{d_{K}}, where Q K ∗ Q^{*}_{K} is the optimal Q-function for the final task 𝒯 K \mathcal{T}_{K} , and Q K π K Q^{\pi_{K}}_{K} is the Q-function under the learned policy π K \pi_{K} .

We begin by analyzing the policy performance error under API. For each task k k , we define:

η k := ‖ Q k ∗ − Q k π k ‖ ∞ . \eta_{k}:=\|Q^{*}_{k}-Q^{\pi_{k}}_{k}\|_{\infty}.

By norm monotonicity, we have:

‖ Q K ∗ − Q K π K ‖ d K ≤ ‖ Q K ∗ − Q K π K ‖ ∞ . \|Q^{*}_{K}-Q^{\pi_{K}}_{K}\|_{d_{K}}\leq\|Q^{*}_{K}-Q^{\pi_{K}}_{K}\|_{\infty}.

We proceed by recalling the result from Theorem 2.1 of Chen and Maguluri (2025) for a fixed task k k , applied with a step size β \beta : ‖ Q k ∗ − Q k π k ‖ ∞ ≤ γ T ​ ‖ Q k ∗ − Q k π k 0 ‖ ∞ + 2 ​ γ 1 − γ ​ ∑ t = 0 T − 1 γ T − 1 − t ​ δ k + 2 ​ γ β ​ ( 1 − γ ) 2 , \|Q^{*}_{k}-Q^{\pi_{k}}_{k}\|_{\infty}\leq\gamma^{T}\|Q^{*}_{k}-Q^{\pi_{k}^{0}}_{k}\|_{\infty}+\frac{2\gamma}{1-\gamma}\sum_{t=0}^{T-1}\gamma^{T-1-t}\delta_{k}+\frac{2\gamma}{\beta(1-\gamma)^{2}}, where δ k \delta_{k} is the approximate value evaluation error at curriculum step k k ; γ T ​ ‖ Q k ∗ − Q k π k 0 ‖ ∞ \gamma^{T}\|Q^{*}_{k}-Q^{\pi_{k}^{0}}_{k}\|_{\infty} reflects initialization at π k 0 \pi_{k}^{0} ; and β > 0 \beta>0 is the API soft update step size parameter. The finite geometric series ∑ t = 0 T − 1 γ T − 1 − t \sum_{t=0}^{T-1}\gamma^{T-1-t} has closed-form expression ∑ i = 0 T − 1 γ i = 1 − γ T 1 − γ ​ for ​ γ ≠ 1 and ∑ i = 0 T − 1 γ i = 1 , for ​ γ = 1 . \sum_{i=0}^{T-1}\gamma^{i}=\frac{1-\gamma^{T}}{1-\gamma}\text{ for }\gamma\neq 1\quad\text{and}\quad\sum_{i=0}^{T-1}\gamma^{i}=1,\text{ for }\gamma=1. Thus we simplify with ∑ i = 0 T − 1 γ i = 1 − γ T 1 − γ . \sum_{i=0}^{T-1}\gamma^{i}=\frac{1-\gamma^{T}}{1-\gamma}.

Since we denote: η k := ‖ Q k ∗ − Q k π k ‖ ∞ ; \eta_{k}:=\|Q^{*}_{k}-Q^{\pi_{k}}_{k}\|_{\infty};

then, the bound becomes: η k ≤ γ T ​ η k + 2 ​ γ ​ ( 1 − γ T ) ( 1 − γ ) 2 ​ δ k + 2 ​ γ β ​ ( 1 − γ ) 2 . \eta_{k}\leq\gamma^{T}\eta_{k}+\frac{2\gamma(1-\gamma^{T})}{(1-\gamma)^{2}}\delta_{k}+\frac{2\gamma}{\beta(1-\gamma)^{2}}.

We isolate η k \eta_{k} : η k ​ ( 1 − γ T ) ≤ 2 ​ γ ​ ( 1 − γ T ) ( 1 − γ ) 2 ​ δ k + 2 ​ γ β ​ ( 1 − γ ) 2 ⇒ η k ≤ 1 1 − γ T ​ ( 2 ​ γ ​ ( 1 − γ T ) ( 1 − γ ) 2 ​ δ k + 2 ​ γ β ​ ( 1 − γ ) 2 ) . \eta_{k}(1-\gamma^{T})\leq\frac{2\gamma(1-\gamma^{T})}{(1-\gamma)^{2}}\delta_{k}+\frac{2\gamma}{\beta(1-\gamma)^{2}}\Rightarrow\eta_{k}\leq\frac{1}{1-\gamma^{T}}\left(\frac{2\gamma(1-\gamma^{T})}{(1-\gamma)^{2}}\delta_{k}+\frac{2\gamma}{\beta(1-\gamma)^{2}}\right).

We want to bound: ‖ Q K ∗ − Q K π K ‖ d K ; \|Q^{*}_{K}-Q^{\pi_{K}}_{K}\|_{d_{K}}; using triangle inequality: ‖ Q K ∗ − Q K π K ‖ d K ≤ ‖ Q K ∗ − Q K π K ‖ ∞ ≤ η K + ∑ k = 1 K − 1 ‖ Q K ∗ − Q k ∗ ‖ d K . \|Q^{*}_{K}-Q^{\pi_{K}}_{K}\|_{d_{K}}\leq\|Q^{*}_{K}-Q^{\pi_{K}}_{K}\|_{\infty}\leq\eta_{K}+\sum_{k=1}^{K-1}\|Q^{*}_{K}-Q^{*}_{k}\|_{d_{K}}.

Let us now plug in the bound for η K \eta_{K} : ‖ Q K ∗ − Q K π K ‖ d K ≤ 1 1 − γ T ​ ( 2 ​ γ ​ ( 1 − γ T ) ( 1 − γ ) 2 ​ δ k + 2 ​ γ β ​ ( 1 − γ ) 2 ) + ∑ k = 1 K − 1 ‖ Q K ∗ − Q k ∗ ‖ d K . \|Q^{*}_{K}-Q^{\pi_{K}}_{K}\|_{d_{K}}\leq\frac{1}{1-\gamma^{T}}\left(\frac{2\gamma(1-\gamma^{T})}{(1-\gamma)^{2}}\delta_{k}+\frac{2\gamma}{\beta(1-\gamma)^{2}}\right)+\sum_{k=1}^{K-1}\|Q^{*}_{K}-Q^{*}_{k}\|_{d_{K}}.

The distribution mismatch is defined by C k := ‖ d K d k ‖ ∞ C_{k}:=\left\|\frac{d_{K}}{d_{k}}\right\|_{\infty} . If d k d_{k} is constructed by curriculum to smoothly interpolate toward d K d_{K} , and both d K d_{K} and d k d_{k} are supported on the same or growing state space. In our analysis, we use: ‖ Q k ∗ − Q k π k ‖ d K ≤ C k ⋅ ‖ Q k ∗ − Q k π k ‖ ∞ , \|Q^{*}_{k}-Q^{\pi_{k}}_{k}\|_{d_{K}}\leq C_{k}\cdot\|Q^{*}_{k}-Q^{\pi_{k}}_{k}\|_{\infty}, where ∥ ⋅ ∥ d K \|\cdot\|_{d_{K}} is the L 2 L_{2} norm under the distribution d K d_{K} and ∥ ⋅ ∥ ∞ \|\cdot\|_{\infty} is the sup norm (worst-case).

In general, for any probability distribution μ \mu , and any function f f , ‖ f ‖ L 2 ​ ( μ ) = ( ∫ f ​ ( x ) 2 ​ 𝑑 μ ​ ( x ) ) 1 / 2 ≤ ‖ f ‖ ∞ . \|f\|_{L_{2}(\mu)}=\left(\int f(x)^{2}d\mu(x)\right)^{1/2}\leq\|f\|_{\infty}. So, in fact, we have the reverse: ‖ f ‖ d K ≤ ‖ f ‖ ∞ ⇒ C k ≤ 1 . \|f\|_{d_{K}}\leq\|f\|_{\infty}\quad\Rightarrow\quad C_{k}\leq 1. If we do want to write ‖ f ‖ d K ≤ C k ⋅ ‖ f ‖ ∞ , \|f\|_{d_{K}}\leq C_{k}\cdot\|f\|_{\infty}, then the tightest possible value for C k C_{k} is exactly 1, and any value C k < 1 C_{k}<1 may be possible depending on the support of f f under d K d_{K} . This holds when later curriculum stages subsume earlier ones, a design principle in CRL. Hence, using C k ≤ 1 C_{k}\leq 1 simplifies bounds without loosening them unnecessarily.

Because curriculum proceeds sequentially, and assuming that earlier tasks are easier and learned more accurately, we can sum the per-step bounds for curriculum steps: ℰ K = ‖ Q K ∗ − Q K π K ‖ d K ≤ ∑ k = 1 K [ 1 1 − γ T ​ ( 2 ​ γ ​ ( 1 − γ T ) ( 1 − γ ) 2 ​ δ k + 2 ​ γ β ​ ( 1 − γ ) 2 ) ] + ∑ k = 1 K − 1 ‖ Q K ∗ − Q k ∗ ‖ d K . \mathcal{E}_{K}=\|Q^{*}_{K}-Q^{\pi_{K}}_{K}\|_{d_{K}}\leq\sum_{k=1}^{K}\left[\frac{1}{1-\gamma^{T}}\left(\frac{2\gamma(1-\gamma^{T})}{(1-\gamma)^{2}}\delta_{k}+\frac{2\gamma}{\beta(1-\gamma)^{2}}\right)\right]+\sum_{k=1}^{K-1}\|Q^{*}_{K}-Q^{*}_{k}\|_{d_{K}}.

This completes the proof.

Optionally, assuming γ T \gamma^{T} is small for large T T , 1 / ( 1 − γ T ) ≤ 2 1/(1-\gamma^{T})\leq 2 leads to: ℰ K ≲ ∑ k = 1 K ( 4 ​ γ ​ ( 1 − γ T ) ( 1 − γ ) 2 ​ δ k + 4 ​ γ β ​ ( 1 − γ ) 2 ) + ∑ k = 1 K − 1 ‖ Q K ∗ − Q k ∗ ‖ d K . \mathcal{E}_{K}\lesssim\sum_{k=1}^{K}\left(\frac{4\gamma(1-\gamma^{T})}{(1-\gamma)^{2}}\delta_{k}+\frac{4\gamma}{\beta(1-\gamma)^{2}}\right)+\sum_{k=1}^{K-1}\|Q^{*}_{K}-Q^{*}_{k}\|_{d_{K}}.

∎

#### A.1.1 Theorem A.1 and Proof

###### Theorem A.1 (Finite-Sample Guarantee) .