From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning — part 1/3

Cleaned-text code points [0, 15039). Bibliography omitted here; source.md includes it.

marginparsep has been altered. topmargin has been altered. marginparpush has been altered.

The page layout violates the ICML style.

Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you.

We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again.

From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning

Xitai Jiang 1 , 2 ∗ † {}^{\,1,2\,*\,\dagger} , Zihan Tang 2 ∗ {}^{\,2\,*} , Wenze Lin 1 , 2 ∗ {}^{\,1,2\,*} , Yang Yue 1 , Shenzhi Wang 1 , and Gao Huang 1 ​ 🖂 {}^{\,1\,\textrm{\Letter}}

1 {}^{1\,} LeapLab, Tsinghua University 2 {}^{2\,} Qiuzhen College, Tsinghua University

∗ Equal Contribution † Project Lead 🖂 {}^{\textrm{\Letter}} Corresponding Author

## 1 Introduction

Reinforcement learning from verifiable rewards (RLVR) has emerged as a dominant paradigm for training large language models on mathematical reasoning, delivering strong empirical gains across benchmarks spanning grade-school arithmetic to olympiad-level competition ( Guo et al., 2025 ; Yu et al., 2025 ; Shao et al., 2024 ; Jaech et al., 2024 ; Wen et al., 2025 ) . The key to its success is that a correct final answer provides an unambiguous and automatically checkable reward signal. This removes the need for costly human annotation and avoids the reward hacking risks of learned reward models ( Skalse et al., 2022 ) .

A central goal of RLVR is to help models solve previously unsolved problems and improve their reasoning ability. However, prior work suggests that direct RLVR often improves sampling efficiency more than it substantially expands the model’s capability boundary ( Yue et al., 2025 ; Shojaee et al., 2025 ; Alam & Rastogi, 2025 ) . Further studies indicate that training at the edge of the model’s current capability with challenging problems is key for better reasoning ability ( Pikus et al., 2025 ; Li et al., 2026a ; Dai et al., 2026 ; Ma et al., 2025 ) . This makes hard problems particularly valuable for RL training. Yet typical RLVR methods like GRPO ( Guo et al., 2025 ) struggle precisely on these problems. First, rewards are normalized within a group of rollouts sampled from the same prompt, so a group in which all rollouts fail provides no learning signal. Second, outcome-based RLVR assigns one sample-level advantage to the entire rollout. Thus, a near-miss attempt receives the same credit as an immediate failure. It is therefore crucial to extract learning signals from such hard-but-informative problems.

A natural way to learn from hard problems is to make better use of expert trajectories. Existing methods mainly follow two routes. One route is compensating for sparse rewards by training the model to imitate expert-generated trajectories, such as supervised fine-tuning and some off-policy RL methods ( Li et al., 2025a ; Yan et al., 2025 ; Fu et al., 2025 ; Zhang et al., 2025a ; Lv et al., 2025 ) . However, they replace the model’s own on-policy exploration with supervised imitation, and the resulting distribution shift between the expert and student policies can hurt training stability and out-of-distribution generalization ( Shenfeld et al., 2025 ; Chu et al., 2025 ) . The other route uses on-policy curriculum RL. These methods provide an expert reasoning prefix or other hints and train the model to complete the remaining solution. ( Amani et al., 2025 ; Zhang et al., 2025b ; Wu et al., 2025a ; Qiyuan et al., 2026 ; Qu et al., 2026 ; Yan et al., 2025 ; Shi et al., 2026 ) . However, these hints are treated as fixed conclusions rather than targets the model must derive, so the model does not need to discover the critical reasoning steps on its own, and the supplied context still shifts the model away from its own generation distribution. In fact, solving hard problems requires the model to explore and master the intermediate conclusions behind these hints by itself. This raises a central question: how can we build a curriculum for hard problems that keeps the model exploring on its own, while also properly giving credit to the intermediate progress it solves along the way?

We propose SCRL ( Subproblem Curriculum Reinforcement Learning ), drawing inspiration from a familiar structure in mathematical competitions: the multi-part problem. In a competition exam, a hard problem is broken into a sequence of subproblems of increasing difficulty, all visible at once; solving an earlier part yields a result that serves as a natural basis for the next. Given the expert solution to a hard problem, we offline construct a sequence of K K verifiable subproblems using an external LLM. The subproblems are ordered from easier to harder, with each later subproblem building on the previous ones, and each subproblem has a verifiable answer. We fix the final subproblem as the original problem itself and ask the model to answer all K K subproblems in a single on-policy rollout . This organically realizes a curriculum learning structure: when the model correctly solves an earlier subproblem, its answer becomes a natural basis for the next, guiding the model toward increasingly difficult reasoning. Critically, the reasoning steps that bridge consecutive subproblems are self-produced, earned through the model’s own on-policy rollout. These intermediate results provide verifiable process-level supervision, naturally enabling finer-grained credit assignment within the rollout. We realize this through subproblem-level normalization , a novel RLVR training technique that normalizes rewards independently at each subproblem position and assigns the resulting advantages to the corresponding answer spans. In particular, to prevent the model from rewarding later subproblems without solving earlier ones, we align credit with curriculum progress by counting only the longest consecutively solved subproblem sequence. For example, the subproblem reward [ 1 , 1 , 0 , 1 ] [1,1,0,1] is treated as [ 1 , 1 , 0 , 0 ] [1,1,0,0] , because progress after the first failed subproblem is not credited.

We validate SCRL with both theory and experiments. Theoretically, we show that subproblem decomposition lifts hard problems out of gradient dead zones by recovering non-degenerate learning signals from earlier subproblems. We formalize this as a metric recovery result, where optimization is lifted from the original policy manifold to a subproblem product manifold and the recovery ratio grows with problem difficulty. The empirical results are consistent with this prediction: SCRL improves over strong curriculum-learning baselines across mathematical reasoning benchmarks. Ablations further confirm the effectiveness of subproblem-level credit assignment and show that SCRL does not rely on highly curated subproblems or strong subproblem generators.

Our main contributions are: • SCRL framework for curriculum learning. We propose a curriculum RL framework that turns each hard problem into a sequence of verifiable subproblems, enabling process-level supervision within a single on-policy rollout. This keeps the model exploring near the boundary of its current capability, making hard problems more effective for training.

• Subproblem-level normalization for fine-grained credit assignment. We introduce subproblem-level normalization , which normalizes rewards independently at each subproblem position and assigns the resulting advantages to the corresponding answer spans, enabling fine-grained credit assignment without external rubrics or additional reward models.

• Theoretical and empirical validation. We provide a metric recovery analysis showing that subproblem decomposition lifts hard problems out of gradient dead zones, with larger relative gains as the original problem becomes harder. Experiments across seven mathematical reasoning benchmarks verify these predictions and show consistent gains over strong baselines (+4.1/+1.9 average-point gains on Qwen3-4B/14B; +3.7 pass@ 1 1 and +4.6 pass@ 64 64 points on three hard benchmarks).

## 2 Related Work

##### Reinforcement Learning with Verifiable Rewards (RLVR)

Recent advances in Large Language Models (LLMs) have highlighted the effectiveness of Reinforcement Learning (RL) in domains with deterministic verifiers such as mathematics and programming Shao et al. (2024) ; Jaech et al. (2024) ; Trinh et al. (2024) ; Yang et al. (2024) ; Qu et al. (2025) ; Wang et al. (2025) . Unlike open-ended generation, these tasks provide unambiguous feedback, allowing for the optimization of policy models through algorithms like Proximal Policy Optimization (PPO) Schulman et al. (2017) or the more memory-efficient Group Relative Policy Optimization (GRPO) Guo et al. (2025) . However, RLVR faces a significant challenge: for difficult problems, the reward signal becomes extremely sparse, leading to a failure in obtaining meaningful policy gradients Uesato et al. (2022) . This challenge is often framed as a credit assignment problem: outcome-based rewards provide a global signal but fail to pinpoint which specific reasoning steps contributed to the final success or failure Lightman et al. (2023) . While iterative self-improvement methods like STaR Zelikman et al. (2022) and ReST Gulcehre et al. (2023) ; Zhang et al. (2024) attempt to bridge this gap through rejection sampling on easier instances, they still struggle when the task’s difficulty exceeds the model’s current exploration horizon. Consequently, curriculum learning Bengio et al. (2009) ; Yang et al. (2025) ; Li et al. (2025b) ; Parashar et al. (2025) ; Wu et al. (2025b) has become a common way to densify learning signals for hard problems by breaking hard tasks into manageable stages.

##### Curriculum Learning for Reasoning

Existing curriculum learning methods for mathematical reasoning can be broadly categorized into two paradigms. The first category focuses on providing external hints or guidance when the model fails to solve a challenging problem. Notable works such as StepHint Zhang et al. (2025a) , Scaf-GRPO Zhang et al. (2025b) and other hint-driven RL frameworks Wu et al. (2025a) ; Qiyuan et al. (2026) ; Qu et al. (2026) ; Yan et al. (2025) ; Shi et al. (2026) , utilize teacher model or self-generated rationales as auxiliary prefixes to lower the exploration threshold. The second category involves rewriting the original problem into simpler versions or augmenting the prompt with supplementary information to facilitate reasoning Chen et al. (2026) ; Wu et al. (2025b) ; Li et al. (2026b) ; Dai et al. (2026) ; Li et al. (2026a) ; Liang et al. (2025) . As seen in MQR Dai et al. (2026) and QuestA Li et al. (2026a) , these methods effectively create a difficulty gradient by manipulating the problem context. However, a fundamental limitation shared by these methods is their reliance on additional context. By providing the hint or reformulated problem as a static prefix, these approaches primarily optimize the model’s continuation capability. As a result, the model fails to internalize the underlying scaffolding logic, as it is never required to generate the hints or auxiliary structures itself. In contrast, SCRL requires the model to generate the entire scaffolded multi-part sequence within a structured response, ensuring that the policy learns to both construct the intermediate reasoning steps and solve the final target problem.

## 3 Method

We propose SCRL ( Subproblem Curriculum Reinforcement Learning ), a curriculum RL framework that turns hard problems into verifiable subproblem curricula for finer-grained credit assignment. SCRL has three steps. First, given a reference solution, an external LLM derives K K verifiable subproblems from the reasoning chain and constructs the subproblem curriculum. Second, the policy answers all K K subproblems in one on-policy rollout. We then verify each subproblem answer and apply progress-aware correction to obtain progress-aware subproblem rewards. Subproblem-level normalization computes an advantage for each subproblem position, which is then used for token-level credit assignment. Finally, to reduce prompt mismatch, SCRL uses mixed-group training, jointly optimizing curriculum rollouts and original-problem rollouts in the same update.

### 3.1 Preliminaries: GRPO

Given a prompt q q , GRPO samples G G rollouts { o i } i = 1 G ∼ π θ ( ⋅ ∣ q ) \{o_{i}\}_{i=1}^{G}\sim\pi_{\theta}(\cdot\mid q) and assigns each rollout a scalar verifiable reward r i r_{i} . It then optimizes the clipped objective ℒ GRPO ( θ ) = − 1 ∑ i L i ∑ i = 1 G ∑ t = 1 L i min ( ρ i , t A i , clip ( ρ i , t , 1 − ε , 1 + ε ) A i ) − β D KL ( π θ ∥ π ref ) . \mathcal{L}_{\mathrm{GRPO}}(\theta)=-\frac{1}{\sum_{i}L_{i}}\sum_{i=1}^{G}\sum_{t=1}^{L_{i}}\min\!\Bigl(\rho_{i,t}A_{i},\;\mathrm{clip}(\rho_{i,t},1-\varepsilon,1+\varepsilon)A_{i}\Bigr)-\beta D_{\mathrm{KL}}(\pi_{\theta}\|\pi_{\mathrm{ref}}). (1)

Here A i = r i − mean ⁡ ( { r i } i = 1 G ) std ⁡ ( { r i } i = 1 G ) A_{i}=\frac{r_{i}-\mathrm{mean}(\{r_{i}\}_{i=1}^{G})}{\mathrm{std}(\{r_{i}\}_{i=1}^{G})} is the group-normalized advantage, and ρ i , t = π θ ​ ( o i , t ∣ q , o i , < t ) π θ old ​ ( o i , t ∣ q , o i , < t ) \rho_{i,t}=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})} is the importance sampling ratio at token t t . Since the same A i A_{i} is assigned to every token in o i o_{i} , GRPO performs sample-level credit assignment.

### 3.2 SCRL Framework

##### Build subproblems.

For each hard problem x x , we start from an existing chain-of-thought reference solution. An external LLM rewrites its intermediate progress nodes into K K verifiable subproblems, rather than solving the problem from scratch. The exact generation prompt is provided in Appendix I , and the main guidelines are summarized below.

##### Curriculum prompt.

Let x x denote the original problem. We define the curriculum prompt t K ​ ( x ) t_{K}(x) as the prompt that presents all K K subproblems s ( 1 ) , … , s ( K ) s^{(1)},\ldots,s^{(K)} simultaneously and asks the model to solve them in order. Thus, x x corresponds to the original-problem rollout, while t K ​ ( x ) t_{K}(x) corresponds to the curriculum rollout. The detailed prompt template is provided in Appendix J.1 .

##### Response format.

During curriculum rollouts, the model is asked to answer the K K subproblems using explicit tags <pj> and </pj> : <p1> a ( 1 ) </p1> ⋯ <pK> a ( K ) </pK> , \texttt{<p1>}\;a^{(1)}\;\texttt{</p1>}\;\cdots\;\texttt{<pK>}\;a^{(K)}\;\texttt{</pK>}, where a ( j ) a^{(j)} is the response to subproblem j j . These tags not only specify the response format, but also mark the token span of each subproblem answer. This allows us to verify each answer separately and later assign the corresponding subproblem-level advantage back to the tokens inside that span.

#### 3.2.1 Progress-Aware Subproblem Rewards

##### Curriculum progress.