Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning โ€” part 1/6

Cleaned-text code points [0, 15914). Bibliography omitted here; source.md includes it.

# Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning

###### Abstract

We aim to improve the reasoning capabilities of language models via reinforcement learning (RL). Recent RL post-trained models like DeepSeek-R1 have demonstrated reasoning abilities on mathematical and coding tasks. However, prior studies suggest that using RL alone to improve reasoning on inherently difficult tasks is less effective. Here, we draw inspiration from curriculum learning and propose to schedule tasks from easy to hard (E2H), allowing LLMs to build reasoning skills gradually. Our method is termed E2H Reasoner. Empirically, we observe that, although easy tasks are important initially, fading them out through appropriate scheduling is essential in preventing overfitting. Theoretically, we establish convergence guarantees for E2H Reasoner within an approximate policy iteration framework. We derive finite-sample complexity bounds and show that, when tasks are appropriately decomposed and conditioned, learning through curriculum stages requires fewer total samples than direct learning. Our code is publicly available at https://github.com/divelab/E2H-Reasoning .

## 1 Introduction

Large Language Models (LLMs) have demonstrated reasoning capabilities in tasks such as multi-step arithmetic, planning, and code generation. However, the notion of reasoning in LLMs remains vague, with some works equating it to generating intermediate steps during problem solving ( Wei et al., 2022 ; Cobbe et al., 2021 ) . This view, although intuitive, blurs the line between genuine reasoning and surface-level pattern recognition ( Stechly et al., 2024 ; Valmeekam et al., 2023 ) . Therefore, we adopt a view focusing on generalization, defining reasoning as the ability to extract principles from simpler tasks and apply them to harder ones . Supporting this capability requires training methods that go beyond imitation and help models learn underlying problem-solving strategies.

In this direction, the success of DeepSeek R1 ( Guo et al., 2025 ) and OpenAI o1 ( Jaech et al., 2024 ) shows that reinforcement learning (RL) based post-training enhances reasoning. RL uses task-specific rewards based on output correctness, unlike supervised fine-tuning (SFT), which trains models to imitate fixed input-output examples ( Zelikman et al., 2022 ) . However, RL struggles on harder tasks on which pre-trained models have low zero shot performance ( Shao et al., 2024 ; Zeng et al., 2025 ) and since rewards are granted only for correct answers, resulting in sparse learning signals.

To address the sparse reward problem, curriculum learning has been applied to reinforcement learning (CRL) by structuring training from easier to harder tasks ( Bengio et al., 2009 ) . This idea has recently been extended to LLM post-training ( Team et al., 2025 ; Bercovich et al., 2025 ) . However, these initial efforts primarily rely on simplified strategies, such as training on easy tasks before switching to hard ones after a fixed number of iterations. In contrast, we introduce E2H Reasoner (E2H), a CRL approach with a probabilistic scheduler that gradually shifts focus from easy to hard tasks, enabling even LLMs to develop core reasoning abilities and generalize to more complex problems. We show that LLMs can learn tasks they initially failed in the zero-shot setting (Fig. 1 ). Empirically, E2H achieves state-of-the-art performance across five reasoning tasks, i.e. Blocksworld ( Valmeekam et al., 2024 ) and Countdown ( Gandhi et al., 2024 ) , as well as three arithmetic reasoning benchmarks ( Hendrycks et al., 2021 ; Agarwal et al., 2021 ; Cobbe et al., 2021 ) . On the theoretical front, we provide a comprehensive analysis of CRL through the lens of Approximate Policy Iteration, establishing convergence guarantees for the final performance gap and finite-sample complexity bounds. Importantly, we prove that with a well-designed curriculum, CRL requires fewer total samples than direct learning, aligning with and supporting our empirical observations.

## 2 Background and Related Work

Reasoning in Large Language Models remains loosely defined, with interpretations varying by tasks and contexts Yu et al. (2024) . Prior work describes it as generating logical chains of thought Wei et al. (2022) ; Wang et al. (2023) , performing multi-step deductions ( Saparov and He, 2023 ; Yao et al., 2023 ; Ling et al., 2025 ) , or simulating human-like problem solving ( Parashar et al., 2025 ) . These views, lack clear boundaries between reasoning and pattern recognition. To address this gap, we are inspired by prior work that frames reasoning as generalization or abstraction ( Valmeekam et al., 2024 ; Valmeekam et al., 2023 ) , building on the idea that reasoning involves learning core principles and applying them broadly ( Stechly et al., 2024 ; Huang et al., 2024 ) .

Post-training of Large Language Models has emerged as a popular approach to improve reasoning ( Snell et al., 2025 ) . These methods are grouped into Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) based techniques. In SFT, the model is trained to imitate outputs from carefully curated human-like reasoning examples ( Zelikman et al., 2022 ; Muennighoff et al., 2025 ) . However, studies have shown that SFT can lead models to overfit to surface-level patterns ( Chu et al., 2025 ) , limiting generalization. In contrast, RL-based post-training uses task-specific rewards and updates the model through policy optimization algorithms ( Schulman et al., 2017 ; Rafailov et al., 2023 ; Shao et al., 2024 ) , instead of imitation. This approach has shown greater potential in improving reasoning performance, as demonstrated by the success of models fine-tuned with RL ( Guo et al., 2025 ; Jaech et al., 2024 ) . Still, for inherently difficult tasks that LLMs struggle to solve in zero-shot settings, post-training with RL alone has been insufficient Shao et al. (2024) ; Zeng et al. (2025) .

Curriculum Learning organizes tasks by increasing difficulty to promote smoother and more effective training ( Bengio et al., 2009 ; Graves et al., 2017 ) . In the context of RL, it has been applied to help agents acquire complex behaviors by first mastering simpler tasks ( Narvekar et al., 2020 ; Tajwar et al., 2025 ) . Recently efforts have been made to investigate how curriculum-based RL can enhance reasoning and generalization in LLMs ( Qiu et al., 2025 ; Bae et al., 2025 ; Zeng et al., 2025 ) . Others improve learning by removing examples that are too easy or too hard Bae et al. (2025) ; Yu et al. (2025) , or by maintaining a balanced mix of task difficulties Zeng et al. (2025) . Similarly, Chen et al. (2025) , and Foster et al. (2025) adaptively sample problems with a 50 % 50\% solve rate to maximize the GRPO advantage during training. Other recent efforts implement manual curricula switching from easy to hard tasks after a fixed number of training iterations ( Xie et al., 2025 ; Team et al., 2025 ; Bercovich et al., 2025 ) . In contrast, our work schedules tasks probabilistically from easy to hard, improving reasoning and generalization to out-of-distribution tasks.

## 3 Method

RL for LLM reasoning. We formulate the reasoning process of LLMs as a RL problem defined over a discounted Markov Decision Process (MDP) โ„ณ = ( ๐’ฎ , ๐’œ , P , r , ฮณ ) \mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma) , where ๐’ฎ \mathcal{S} is the state space, ๐’œ \mathcal{A} is the finite action space, P : ๐’ฎ ร— ๐’œ โ†’ ฮ” โก ( ๐’ฎ ) P:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) is the transition kernel, r : ๐’ฎ ร— ๐’œ โ†’ [ 0 , R max ] r:\mathcal{S}\times\mathcal{A}\to[0,R_{\max}] is the reward function, and ฮณ โˆˆ ( 0 , 1 ) \gamma\in(0,1) is the discount factor. The state space ๐’ฎ \mathcal{S} consists of all valid token prefixes, where each state s t = ( x 0 , x 1 , โ€ฆ , x t ) s_{t}=(x_{0},x_{1},\ldots,x_{t}) is a sequence of tokens from the vocabulary ฮฃ \Sigma . The action space is the vocabulary itself, ๐’œ = ฮฃ \mathcal{A}=\Sigma . A policy ฯ€ ฮธ \pi_{\theta} corresponds to the pre-trained LLM, which defines a distribution over the next token conditioned on the current prefix: ฯ€ ฮธ โ€‹ ( x t + 1 | s t ) = p ฮธ โ€‹ ( x t + 1 | x 0 , โ€ฆ , x t ) \pi_{\theta}(x_{t+1}|s_{t})=p_{\theta}(x_{t+1}|x_{0},\ldots,x_{t}) . We adopt the tags used in ( Guo et al., 2025 ) , i.e . , <think> </think> and <answer> </answer>, to distinguish intermediate reasoning from final answers. The reward function is sparse: r โก ( s , a , s โ€‹ โ€™ ) = 0 r(s,a,s\textquoteright)=0 for all intermediate states, and r โก ( s , a , s โ€‹ โ€™ ) = r โก ( y ) r(s,a,s\textquoteright)=r(y) only when the final predicted answer y y wrapped by <answer> </answer> is completed. The goal is to optimize the policy to maximize the expected cumulative reward, encouraging the model to generate correct and well-formatted answers through effective reasoning.

### 3.1 Task decomposition for RL post-training

While SFT provides strong supervision signals, Chu et al. (2025) suggest that reasoning and generalization ability are more effectively enhanced through RL-based post-training. However, applying RL techniques similar to DeepSeek-R1-Zero ( Guo et al., 2025 ) to learn complicated reasoning tasks remains challenging. In this work, we analyze these challenges in two key aspects, the distribution gap and reward design .

Challenge 1: Distribution gap. Learning tasks that exceed the base LLMโ€™s reasoning capabilities introduce significant learning challenges. These challenges are often caused by a non-trivial distribution gap between the model pre-training source data distribution d 0 d_{0} and the target data distribution d K d_{K} . As shown in Fig. 2 , because rewards are only given for correct outputs, large distribution shifts can lead to low accuracy and sparse reward signals ( Shao et al., 2024 ; Zeng et al., 2025 ) . Moreover, fitting the model to a single target distribution can lead to overfitting and memorization, undermining the modelโ€™s generalization and reasoning ability.

Challenge 2: Reward design. Challenging reasoning tasks often require LLMs to combine multiple skills to arrive at a solution. While designing a fine-grained, step-by-step reward function could potentially guide the model effectively, such design is generally task-specific and labor-intensive. For example, a computer science student is supposed to learn basic mathematics and linear algebra before learning machine learning. Similarly, a typical Countdown ( Gandhi et al., 2024 ) task involves skills like basic arithmetic, estimating the distance to the goal, and backtracking. While it is possible to include a supervision signal for each intermediate step, doing so is not sustainable and generalizable across diverse tasks.

To overcome these RL learning challenges, we propose task decomposition by splitting training data into subsets of increasing difficulty, based on either human annotations or model performance. This aligns with curriculum learning, where we interpolate between the pre-training distribution d 0 d_{0} and target task distribution d K d_{K} via intermediate stages d k k = 1 K {d_{k}}_{k=1}^{K} , reducing distribution shift and improving training stability (see Fig. 2 ). From the reward design perspective, decomposing tasks by difficulty breaks complex skill acquisition into simpler steps. For instance, the 6-number Countdown task involves using six integers and four operators ( + , โˆ’ , ร— , รท +,-,\times,\div ) to reach a target number, requiring the model to perform arithmetic, estimate distances to the goal, and backtrack effectively. In contrast, a 2-number problem focuses primarily on arithmetic, allowing the model to build foundational competence before scaling to harder variants. This avoids handcrafted reward shaping and improves transferability.

For tasks like Blocksworld, MATH, and Countdown, we use human-aligned difficulty signals such as plan length, labeled difficulty, or operand count (see Table 9 ). For others like AQuA and GSM8K, where such annotations are unavailable, we automatically estimate difficulty based on model error rates under CoT prompting and group examples accordingly (see Figs. 4.2 and 6 ). We generate 20 responses per question using 1-shot CoT prompting, with prompts and in-context examples from ( Hao et al., 2024 ; Parashar et al., 2025 ) , and assign difficulty by grouping questions based on quartiles of the error rate distribution.

In this work, we tackle the challenge of learning a hard reasoning task by partitioning the training distribution into four difficulty levels , namely, trivial , easy , medium , and hard , then we adapt our pre-trained LLM to these tasks in a sequential curriculum. In Section 3.3 , we provide a theoretical justification that, under fixed sampling and training resources, learning step by step leads to better performance than training directly on the hard task.

### 3.2 Training schedulers for LLM reasoning ability

While task decomposition simplifies RL post-training, traditional curriculum learning poses two main challenges, mainly task forgetting and overfitting, due to the rigid progression through tasks in a fixed task order.

Challenges: Task forgetting and overfitting. The first challenge, task forgetting, refers to the degradation in performance on earlier (easier) tasks as the model adapts to later (harder) tasks. According to the traditional model generalization literature ( Arjovsky et al., 2019 ; Gulrajani and Lopez-Paz, 2021 ) , the task distribution shifts are considered as explicit signals for the model generalization direction; thus, retaining strong performance across all task distributions is essential for generalization. Therefore, the task forgetting will undermine the modelโ€™s generalization capability, i.e . , the reasoning ability. Task overfitting, the second challenge, arises when the model overfits to trivial tasks and prefers simplistic patterns or short answers that bypass meaningful reasoning. This phenomenon is called reward hacking ( Laidlaw et al., 2025 ) , where the model exploits shortcuts on easy tasks and fails to learn harder ones, resulting in poor reasoning performance. To address both challenges, we explore different training sampling techniques, forming four different scheduling strategies as follows.

Traditional scheduling. We first formulate the traditional sequential curriculum learning sampler with T T training steps as ๐’ trad โ€‹ ( t , k ) = 1 \mathbf{S}_{\text{trad}}(t,k)=1 when ฯ„ k โ‰ค t โ‰ค ฯ„ k + 1 \tau_{k}\leq t\leq\tau_{k+1} , otherwise, 0, where t t denotes the current training step; for K K tasks k = 1 , โ€ฆ , K k=1,\ldots,K , ฯ„ k \tau_{k} denotes the threshold when the curriculum learning proceeds to the k k -th stage, while ฯ„ 1 = 0 \tau_{1}=0 and ฯ„ K + 1 = T \tau_{K+1}=T . The output of the sampler denotes the probability of sampling data from the k k -th task; therefore, at the t t step, the sampling distribution will be [ ๐’ trad โ€‹ ( t , 1 ) \mathbf{S}_{\text{trad}}(t,1) , ๐’ trad โ€‹ ( t , 2 ) \mathbf{S}_{\text{trad}}(t,2) , โ€ฆ, ๐’ trad โ€‹ ( t , K ) \mathbf{S}_{\text{trad}}(t,K) ].

Balanced scheduling. To avoid forgetting, the simplest way is to mix all data with different difficulties together and sample randomly, which can be considered as a trivial case of curriculum learning. Alternatively, this can be interpreted as the default behavior of any policy optimization algorithm ( Shao et al., 2024 ; Schulman et al., 2017 ) , where training occurs without considering task difficulty. This balanced sampler can be written as: ๐’ balanced โ€‹ ( t , k ) = 1 K , \mathbf{S}_{\text{balanced}}(t,k)=\frac{1}{K}, where each task difficulties have the same probability to be selected at each training step. Although this is an efficient way to avoid forgetting, this sampling introduces harder tasks too early, leading to sparser rewards and suboptimal CRL.