From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning — part 2/3

Cleaned-text code points [15039, 30748). Bibliography omitted here; source.md includes it.

For a curriculum rollout o i ∼ π θ ( ⋅ ∣ t K ( x ) ) o_{i}\sim\pi_{\theta}(\cdot\mid t_{K}(x)) , verifying the K K extracted subproblem answers gives a raw reward vector 𝐫 i = ( r i ( 1 ) , … , r i ( K ) ) ∈ { 0 , 1 } K \mathbf{r}_{i}=(r_{i}^{(1)},\ldots,r_{i}^{(K)})\in\{0,1\}^{K} . If the response does not follow the required format, we set 𝐫 i = 𝟎 \mathbf{r}_{i}=\mathbf{0} . We define the curriculum progress k i ∈ { 0 , 1 , … , K } k_{i}\in\{0,1,\ldots,K\} as the maximum number of consecutively solved subproblems from the beginning: k i := max ⁡ { j ∈ { 0 , 1 , … , K } | r i ( 1 ) = ⋯ = r i ( j ) = 1 } . k_{i}:=\max\bigl\{j\in\{0,1,\ldots,K\}\;\big|\;r_{i}^{(1)}=\cdots=r_{i}^{(j)}=1\bigr\}. (2) Thus, k i = 0 k_{i}=0 means the first subproblem is incorrect, while k i = K k_{i}=K means all subproblems are solved. The curriculum progress k i k_{i} tracks the current policy’s capability boundary on the hard problem, and also identifies the intermediate progress actually achieved by the rollout.

##### Progress-aware correction.

Directly rewarding each subproblem independently may credit later subproblems despite earlier failures, creating a potential reward-hacking shortcut. We therefore align rewards with curriculum progress by keeping only the consecutively solved prefix: r ~ i ( j ) := { r i ( j ) , j ≤ k i , 0 , j > k i , 𝐫 ~ i = ( r ~ i ( 1 ) , r ~ i ( 2 ) , … , r ~ i ( K ) ) . \tilde{r}_{i}^{(j)}:=\begin{cases}r_{i}^{(j)},&j\leq k_{i},\\ 0,&j>k_{i},\end{cases}\qquad\tilde{\mathbf{r}}_{i}=\bigl(\tilde{r}_{i}^{(1)},\tilde{r}_{i}^{(2)},\ldots,\tilde{r}_{i}^{(K)}\bigr). (3) For example, when K = 4 K=4 , ( 1 , 1 , 0 , 1 ) (1,1,0,1) is corrected to ( 1 , 1 , 0 , 0 ) (1,1,0,0) . For notational convenience, we use R i ( j ) := r ~ i ( j ) R_{i}^{(j)}:=\tilde{r}_{i}^{(j)} as the final subproblem reward for training.

#### 3.2.2 SCRL Training Algorithm

In this section, we describe the training details of SCRL, including subproblem-level normalization for advantage computation, token-level credit assignment, and mixed-group training. The full training procedure is summarized in Appendix C .

##### Subproblem-level normalization.

Given G G curriculum rollouts o i o_{i} for i = 1 , … , G i=1,\ldots,G , we normalize the final subproblem rewards at each subproblem position j j across the rollout group: A i ( j ) = R i ( j ) − mean ⁡ ( { R i ( j ) } i = 1 G ) std ⁡ ( { R i ( j ) } i = 1 G ) . A_{i}^{(j)}=\frac{R_{i}^{(j)}-\mathrm{mean}\bigl(\{R_{i}^{(j)}\}_{i=1}^{G}\bigr)}{\mathrm{std}\bigl(\{R_{i}^{(j)}\}_{i=1}^{G}\bigr)}. (4) Thus, the subproblem-level advantage A i ( j ) A_{i}^{(j)} measures the relative success of rollout i i at subproblem position j j within the rollout group, independent of rewards at other subproblem positions.

##### Token-level credit assignment.

After computing the subproblem-level advantages, we assign them back to the tokens of the corresponding subproblem answers. Using the structured response format, we define sub i ​ ( t ) = j \mathrm{sub}_{i}(t)=j if token o i , t o_{i,t} lies between <pj> and </pj> ; then A i , t = A i ( sub i ​ ( t ) ) A_{i,t}=A_{i}^{(\mathrm{sub}_{i}(t))} gives the token-level advantage. Tokens outside all answer spans receive zero advantage, and if the response does not follow the required format, all tokens in that response receive zero advantage. This converts subproblem-level progress into token-level learning signals for the corresponding answer spans.

##### Mixed-group training.

Training only on the curriculum prompt t K ​ ( x ) t_{K}(x) can cause prompt mismatch, because evaluation uses the original prompt x x . We therefore use mixed-group training: for each problem x x , G / 2 G/2 rollouts are sampled from t K ​ ( x ) t_{K}(x) and optimized with token-level advantages from subproblem-level normalization, while the other G / 2 G/2 rollouts are sampled from x x and optimized with standard outcome-based GRPO. The final SCRL objective is ℒ SCRL ​ ( θ ) \displaystyle\mathcal{L}_{\mathrm{SCRL}}(\theta) = − 1 ∑ i = 1 G L i [ ∑ i = 1 G / 2 ∑ t = 1 L i min ( ρ i , t A i , t , clip ( ρ i , t , 1 − ε , 1 + ε ) A i , t ) \displaystyle=-\frac{1}{\sum_{i=1}^{G}L_{i}}\Bigg[\sum_{i=1}^{G/2}\sum_{t=1}^{L_{i}}\min\!\Bigl(\rho_{i,t}A_{i,t},\mathrm{clip}(\rho_{i,t},1-\varepsilon,1+\varepsilon)A_{i,t}\Bigr) + ∑ i = G / 2 + 1 G ∑ t = 1 L i min ( ρ i , t A i , clip ( ρ i , t , 1 − ε , 1 + ε ) A i ) ] − β D KL ( π θ ∥ π ref ) . \displaystyle\quad+\sum_{i=G/2+1}^{G}\sum_{t=1}^{L_{i}}\min\!\Bigl(\rho_{i,t}A_{i},\mathrm{clip}(\rho_{i,t},1-\varepsilon,1+\varepsilon)A_{i}\Bigr)\Bigg]-\beta D_{\mathrm{KL}}(\pi_{\theta}\,\|\,\pi_{\mathrm{ref}}). (5) The two bracketed terms correspond to curriculum rollouts and original-problem rollouts respectively. The complete training procedure is summarized in Algorithm 1 .

## 4 Theoretical Analysis

Using the information geometry of the policy manifold ℳ = { π θ : θ ∈ Θ } \mathcal{M}=\{\pi_{\theta}:\theta\in\Theta\} equipped with the Fisher–Rao metric ( Amari, 2016 ) , we show that hard problems can place outcome-based GRPO in a gradient dead zone , while subproblem decomposition lifts optimization to a product manifold that recovers useful gradient information. Full discussions and proofs are provided in Appendix B .

###### Definition 4.1 (Effective and Lifted Gradient Information Matrices) .

Under GRPO, let o 1 , … , o G ∼ i . i . d . π θ ( ⋅ ∣ x ) o_{1},\ldots,o_{G}\overset{\mathrm{i.i.d.}}{\sim}\pi_{\theta}(\cdot\mid x) be G G sampled rollouts. The effective gradient information matrix (EGIM) of x x and the lifted EGIM of its subproblem transformation 𝒯 ​ ( x ) = t K ​ ( x ) \mathcal{T}(x)=t_{K}(x) are 𝑭 x ​ ( θ ) \displaystyle\bm{F}_{x}(\theta) = 1 G ​ ∑ i = 1 G 𝔼 ⁡ [ g i ​ ( x ) ​ g i ​ ( x ) ⊤ ] , where ​ g i ​ ( x ) = A ^ i ​ ( x ) ​ ∇ θ ​ log ⁡ π θ ​ ( o i ∣ x ) , \displaystyle=\frac{1}{G}\sum_{i=1}^{G}\mathbb{E}\!\left[g_{i}(x)g_{i}(x)^{\top}\right],\qquad\text{where }g_{i}(x)=\hat{A}_{i}(x)\nabla_{\theta}\log\pi_{\theta}(o_{i}\mid x), (6) 𝑭 𝒯 ⁡ ( x ) ​ ( θ ) \displaystyle\bm{F}_{\mathcal{T}(x)}(\theta) = 1 K ​ ∑ j = 1 K 1 G ​ ∑ i = 1 G 𝔼 ⁡ [ A ^ i ( j ) ​ 2 ​ ∇ θ ​ log ⁡ π θ ​ ( o i ( j ) ∣ t K ​ ( x ) ) ​ ∇ θ ​ log ​ π θ ​ ( o i ( j ) ∣ t K ​ ( x ) ) ⊤ ] , \displaystyle=\frac{1}{K}\sum_{j=1}^{K}\frac{1}{G}\sum_{i=1}^{G}\mathbb{E}\!\left[\hat{A}_{i}^{(j)2}\nabla_{\theta}\log\pi_{\theta}(o_{i}^{(j)}\mid t_{K}(x))\nabla_{\theta}\log\pi_{\theta}(o_{i}^{(j)}\mid t_{K}(x))^{\top}\right], (7) Here A ^ i ​ ( x ) \hat{A}_{i}(x) and A ^ i ( j ) \hat{A}_{i}^{(j)} denote the original-problem and subproblem-position advantages respectively, and the smallest eigenvalue λ min ​ ( 𝐅 𝒯 ⁡ ( x ) ​ ( θ ) ) \lambda_{\min}(\bm{F}_{\mathcal{T}(x)}(\theta)) measures the weakest useful gradient signal.

###### Theorem 4.2 (Gradient Dead Zone) .

Let p ( x ; θ ) = Pr π θ [ r ( x , o ) = 1 ] p(x;\theta)=\Pr_{\pi_{\theta}}[r(x,o)=1] be the probability that the current policy solves x x . If p ⁡ ( x , θ ) < δ p(x;\theta)<\delta , then λ min ​ ( 𝑭 x ​ ( θ ) ) ≤ G ​ δ ⋅ C A ^ 2 ⋅ B s 2 = O ⁡ ( δ ) , \lambda_{\min}\!\left(\bm{F}_{x}(\theta)\right)\leq G\delta\cdot C_{\hat{A}}^{2}\cdot B_{s}^{2}=O(\delta), (8)

where C A ^ ≤ G − 1 C_{\hat{A}}\leq\sqrt{G-1} bounds the normalized advantage magnitude and B s B_{s} bounds the score norm (both derived in Appendix B.3 ).

Theorem 4.2 shows that direct RLVR training becomes ineffective on hard problems: when correct rollouts are rare, reward groups collapse and the worst-case effective gradient signal vanishes.

###### Theorem 4.3 (Metric Recovery via Subproblem Decomposition) .

Let x x be in the gradient dead zone with p ⁡ ( x , θ ) < δ p(x;\theta)<\delta . Suppose the subproblem construction satisfies p j ​ ( x , θ ) := Pr ⁡ [ R ( j ) = 1 ∣ t K ​ ( x ) ] ∈ [ p ⋆ , 1 − p ⋆ ] for all ​ j < K , p_{j}(x;\theta):=\Pr[R^{(j)}=1\mid t_{K}(x)]\in[p^{\star},1-p^{\star}]\quad\text{for all }j<K, where p ⋆ ∈ ( δ , 1 / 2 ] p^{\star}\in(\delta,1/2] . Under the conditional identifiability assumption ( 𝔼 ⁡ [ ( v ⊤ ​ ∇ θ ​ log ⁡ π θ ) 2 ∣ r = r 0 ] ≥ σ min 2 > 0 \mathbb{E}[(v^{\top}\nabla_{\theta}\log\pi_{\theta})^{2}\mid r{=}r_{0}]\geq\sigma_{\min}^{2}>0 for all unit v v , r 0 ∈ { 0 , 1 } r_{0}\in\{0,1\} ), λ min ​ ( 𝑭 𝒯 ⁡ ( x ) ​ ( θ ) ) \displaystyle\lambda_{\min}\!\left(\bm{F}_{\mathcal{T}(x)}(\theta)\right) ≥ 1 K ​ c ​ ( p ⋆ , G , σ min ) > 0 , \displaystyle\geq\frac{1}{K}\,c(p^{\star},G,\sigma_{\min})>0, λ min ​ ( 𝑭 𝒯 ⁡ ( x ) ​ ( θ ) ) λ min ​ ( 𝑭 x ​ ( θ ) ) \displaystyle\frac{\lambda_{\min}(\bm{F}_{\mathcal{T}(x)}(\theta))}{\lambda_{\min}(\bm{F}_{x}(\theta))} = Ω ⁡ ( 1 δ ) , \displaystyle=\Omega\!\left(\frac{1}{\delta}\right), (9) where c ⁡ ( p ⋆ , G , σ min ) = ( 1 − ( p ⋆ ) G − ( 1 − p ⋆ ) G ) ​ σ min 2 c(p^{\star},G,\sigma_{\min})=\bigl(1-(p^{\star})^{G}-(1-p^{\star})^{G}\bigr)\sigma_{\min}^{2} is a positive constant independent of δ \delta .

Theorem 4.3 shows that subproblem curriculum helps hard problems by recovering a non-degenerate learning geometry, even when the original problem provides almost no useful gradient signal. Moreover, the recovery ratio grows as p ⁡ ( x , θ ) → 0 p(x;\theta)\to 0 , predicting larger relative gains on harder problems.

## 5 Experiment

### 5.1 Experimental Setup

##### Models.

To investigate the scalability and effectiveness of our proposed method across different model capacities, we conduct experiments on the Qwen and Llama series. Specifically, we utilize Qwen3-4B-Base , Qwen3-14B-Base and Llama3.2-3B-Instruct as our base policies.

##### Training Setup.

We use the training set hard_1024 , a subset of 1,024 problems randomly selected from the high-difficulty competition mathematics dataset provided by Yang et al. (2026) . For SCRL, subproblems are generated with the DeepSeek-V3.2 API with K = 4 K=4 . All models are trained using the Verl framework Sheng et al. (2025) for a total of 300 steps. Detailed hyperparameter configurations are provided in Appendix F.1 .

##### Benchmark.

We evaluate the models on seven widely used mathematical reasoning benchmarks: OlympiadBench , Minerva , MATH-500 , AIME 2024 , AIME 2025 , AMC , and IMO-Bench .

##### Baseline Settings.

We compare our method against the following competitive baselines: SFT , GRPO Guo et al. (2025) , DAPO Yu et al. (2025) , QuestA Li et al. (2026a) and NuRL Chen et al. (2025) .Implementation details are provided in Appendix F.3 .

### 5.2 Main Results and Further Analysis

The main results across seven mathematical reasoning benchmarks are summarized in Table 1 , with full experimental results provided in Appendix E .

##### Superior Performance Across All Benchmarks.

As shown in Table 1 , SCRL consistently outperforms vanilla GRPO and competitive baselines including DAPO, QuestA, and NuRL across three model scales: Llama3.2-3B, Qwen3-4B, and Qwen3-14B. In terms of average accuracy (Avg), SCRL achieves the best performance in all settings. The gain is especially clear on Qwen3-4B, where SCRL reaches an average score of 35.0% , improving over the second-best baseline QuestA ( 32.0% ) by 3.0 points and over vanilla GRPO ( 30.9% ) by 4.1 points. On challenging benchmarks such as AIME’25, SCRL also shows strong gains, achieving 15.3% compared with QuestA’s 11.7% .

##### Curriculum progress transfers to hard-problem solving.

Figure 4 shows pass@ k k curves on AIME24, AIME25, and IMO-Bench. SCRL consistently outperforms GRPO and other curriculum RL baselines across the entire evaluated range of k k , indicating stronger hard-problem solving ability.

Figure 6 further tracks the ratio of solvable problems during training, where a problem is counted as solvable once it is fully solved at least once. The full group statistic counts success in either the original-problem or curriculum format, while the half group statistic uses only half-budget original-problem rollouts, matching SCRL’s mixed-group setting. SCRL achieves a higher solvable ratio than GRPO under both protocols, showing that curriculum progress transfers back to direct hard-problem solving rather than only improving curriculum-format rollouts.

##### SCRL does not rely on highly curated subproblems.

We further examine whether SCRL depends on high-quality subproblem construction. Table 2 compares subproblems generated by DeepSeek-V3.2 and a weaker Qwen3-4B-Instruct generator, using the same generation prompt and downstream training pipeline. In both cases, the generator is given the dataset reference solution, so it only decomposes an already solved problem rather than solving it from scratch. SCRL remains effective with the weaker generator, improving over GRPO by +2.7 points on average, while DeepSeek-V3.2 further increases the gain to +3.9 points.

Figure 6 shows that even with DeepSeek-V3.2, the ratio of curriculum instances fully solved at k i = 4 k_{i}=4 remains lower than the GRPO solvable ratio (the GRPO bar at k i = 4 k_{i}=4 reports the ratio of original problems solved by GRPO under the same half-group counting protocol). Nevertheless, SCRL still achieves a substantial performance gain. This indicates that SCRL does not require subproblems to be easy or perfectly curated. At the same time, DeepSeek-V3.2 produces a larger k i = 4 k_{i}=4 ratio than Qwen3-4B-Instruct, suggesting that better subproblem quality can further increase SCRL’s gains.

##### Subproblem-level normalization enables better credit assignment.

We further ablate on Qwen3-4B-Base how credit is assigned within curriculum rollouts. Table 3 compares the full method with two alternatives: removing progress-aware correction, and Both-GRPO, which keeps mixed training but verifies only the final subproblem and applies sample-level GRPO to curriculum rollouts.

Subproblem-level normalization with progress-aware correction performs best. Without correction, dense subproblem signals may reward later steps after earlier failures, while Both-GRPO uses subproblems only as hints and cannot credit valid intermediate progress. Thus, effective curriculum training needs both subproblem-specific signals and progress-aware correction.

## 6 Conclusion

We propose SCRL, a subproblem curriculum RL framework for hard LLM reasoning with verifiable rewards. SCRL derives verifiable subproblems from reasoning chains and uses subproblem-level normalization to convert partial rollout progress into token-level learning signals, enabling fine-grained credit assignment without external reward models or process annotations. Our theory shows that subproblem decomposition can lift hard problems out of gradient dead zones, and experiments show consistent gains over strong RLVR and curriculum-learning baselines.

## Appendix A More Ablation Study

##### Number of subproblems.

We first ablate the number of subproblems K K . Our main setting uses K = 4 K=4 , with the final subproblem fixed as the original problem. For K = 3 K=3 and K = 2 K=2 , we take the last three or last two subproblems from the K = 4 K=4 sequence, so the shorter curricula preserve the nested structure and still end with the original problem.

Table 4 shows that increasing K K improves the average performance, with K = 4 K=4 performing best. This supports the role of subproblem curricula: more subproblems expose a finer progression toward the original problem, creating more verifiable intermediate signals within each rollout. Figure 7 explains why longer subproblem curricula help. As K K increases, fewer rollouts make zero progress, meaning more rollouts solve at least one subproblem and receive a non-empty learning signal. Thus, increasing K K improves both the granularity of credit assignment and the availability of reward signal on hard problems.