From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning — part 3/3

Cleaned-text code points [30748, 45874). Bibliography omitted here; source.md includes it.

However, longer curricula also increase rollout complexity. The model must answer more subproblems, and progress-aware correction requires earlier subproblems to be solved before later ones can receive credit. If an intermediate subproblem is ambiguous or poorly constructed, it can block credit for later progress. We therefore use K = 4 K=4 as a practical trade-off between denser supervision and curriculum complexity.

##### Training data construction.

Each SCRL curriculum rollout contains K = 4 K=4 subproblems, so we ask whether the gain comes from the training algorithm or simply from exposing the model to more questions. We compare the default setting, hard_1024 with SCRL, against two data-scaling controls. hard_4096 expands the original-problem set by adding 3 × 1024 3\times 1024 hard problems from the InT dataset ( Yang et al., 2026 ) . subproblem_4096 instead splits each four-subproblem curriculum instance into four standalone questions; since the last subproblem is the original problem, it also contains hard_1024 .

Table 5 shows that SCRL outperforms both data-scaling controls. This suggests that the gain does not mainly come from seeing more questions, but from using subproblems as a curriculum inside each rollout. Subproblems serve as intermediate anchors: after solving an earlier subproblem, the model can build on that result when solving later ones, while subproblem-level normalization assigns credit to the corresponding answer spans. The average response length is computed over the first 20 training epochs. Although each SCRL curriculum rollout contains four subproblems, its average response length is only about 1.5 × 1.5\times that of GRPO on hard_1024 . This indicates that SCRL does not spend response length proportional to the number of subproblems, but instead uses the subproblem curriculum structure to support more efficient exploration.

The two controls further clarify this point. hard_4096 improves over hard_1024 , but the gain is limited, showing that simply adding more hard problems is less effective under our training budget. subproblem_4096 brings a slightly larger gain, but still falls behind SCRL, suggesting that training on isolated easier subproblems does not by itself teach the model to solve harder target problems. In contrast, SCRL keeps the model exploring near its current capability boundary by preserving the dependency among subproblems, making the curriculum more useful than either data scaling strategy.

## Appendix B Proofs for Section 4

### B.1 Proof of Theorem 4.2 (Bound on λ min ​ ( 𝑭 x ​ ( θ ) ) \lambda_{\min}(\bm{F}_{x}(\theta)) )

We bound λ min ​ ( 𝑭 x ​ ( θ ) ) \lambda_{\min}(\bm{F}_{x}(\theta)) when p ⁡ ( x , θ ) < δ p(x;\theta)<\delta .

Let E 0 E_{0} denote the event that all G G rollouts share the same reward (either all fail or all succeed). On E 0 E_{0} , the group is degenerate ( σ ^ r x = 0 \hat{\sigma}_{r_{x}}=0 ), so by the GRPO convention A ^ i = 0 \hat{A}_{i}=0 for all i i , and every term in ( 6 ) is zero. By the law of total expectation: 𝑭 x ​ ( θ ) = Pr ⁡ [ E 0 c ] ⋅ 𝔼 ⁡ [ 1 G ​ ∑ i A ^ i 2 ​ s i ​ s i ⊤ | E 0 c ] , \bm{F}_{x}(\theta)\;=\;\Pr[E_{0}^{c}]\cdot\mathbb{E}\!\left[\frac{1}{G}\sum_{i}\hat{A}_{i}^{2}s_{i}s_{i}^{\top}\,\Big|\,E_{0}^{c}\right], (10) where s i = ∇ θ ​ log ​ π θ ​ ( o i ∣ x ) s_{i}=\nabla_{\theta}\log\pi_{\theta}(o_{i}\mid x) . For any unit vector v v , using A ^ i 2 ≤ C A ^ 2 \hat{A}_{i}^{2}\leq C_{\hat{A}}^{2} a.s. and ( v ⊤ ​ s i ) 2 ≤ B s 2 (v^{\top}s_{i})^{2}\leq B_{s}^{2} a.s. (by regularity): v ⊤ ​ 𝑭 x ​ ( θ ) ​ v ≤ Pr ⁡ [ E 0 c ] ⋅ C A ^ 2 ⋅ B s 2 . v^{\top}\bm{F}_{x}(\theta)\,v\;\leq\;\Pr[E_{0}^{c}]\cdot C_{\hat{A}}^{2}\cdot B_{s}^{2}. (11) Since Pr ⁡ [ E 0 c ] = 1 − p G − ( 1 − p ) G ≤ 1 − ( 1 − p ) G ≤ 1 − ( 1 − δ ) G ≤ G ​ δ \Pr[E_{0}^{c}]=1-p^{G}-(1-p)^{G}\leq 1-(1-p)^{G}\leq 1-(1-\delta)^{G}\leq G\delta (using p < δ p<\delta and Bernoulli’s inequality): λ min ​ ( 𝑭 x ​ ( θ ) ) ≤ sup ‖ v ‖ = 1 v ⊤ ​ 𝑭 x ​ v ≤ G ​ δ ⋅ C A ^ 2 ⋅ B s 2 = O ⁡ ( δ ) . \lambda_{\min}\!\left(\bm{F}_{x}(\theta)\right)\;\leq\;\sup_{\|v\|=1}v^{\top}\bm{F}_{x}v\;\leq\;G\delta\cdot C_{\hat{A}}^{2}\cdot B_{s}^{2}\;=\;O(\delta). (12) This establishes Theorem 4.2 .∎

### B.2 Proof of Theorem 4.3

##### First claim.

We bound the column- 1 1 contribution to 𝑭 𝒯 ⁡ ( x ) ​ ( θ ) \bm{F}_{\mathcal{T}(x)}(\theta) . Let E 1 ≠ E_{1}^{\neq} denote the event that column 1 is non-degenerate.

Lower bound on Pr ⁡ [ E 1 ≠ ] \Pr[E_{1}^{\neq}] . Since p 1 ∈ [ p ⋆ , 1 − p ⋆ ] p_{1}\in[p^{\star},1-p^{\star}] by assumption, and g ⁡ ( p ) = 1 − p G − ( 1 − p ) G g(p)=1-p^{G}-(1-p)^{G} is symmetric around 1 / 2 1/2 and non-decreasing on [ 0 , 1 / 2 ] [0,1/2] , its minimum on [ p ⋆ , 1 − p ⋆ ] [p^{\star},1-p^{\star}] is attained at the endpoints: Pr ⁡ [ E 1 ≠ ] = g ⁡ ( p 1 ) ≥ g ⁡ ( p ⋆ ) = 1 − ( p ⋆ ) G − ( 1 − p ⋆ ) G = : q min > 0 . \Pr[E_{1}^{\neq}]\;=\;g(p_{1})\;\geq\;g(p^{\star})\;=\;1-(p^{\star})^{G}-(1-p^{\star})^{G}\;=:\;q_{\min}\;>\;0. (13)

Lower bound on the conditional EGIM. We compute v ⊤ ​ 𝑭 𝒯 ⁡ ( x ) ( 1 ) ​ v v^{\top}\bm{F}^{(1)}_{\mathcal{T}(x)}v by conditioning on the reward vector ( R 1 ( 1 ) , … , R G ( 1 ) ) (R_{1}^{(1)},\ldots,R_{G}^{(1)}) . Given the rewards, A ^ i ( 1 ) \hat{A}_{i}^{(1)} is fully determined, and since rollouts are i.i.d., o i ( 1 ) o_{i}^{(1)} is conditionally independent of ( R j ( 1 ) ) j ≠ i (R_{j}^{(1)})_{j\neq i} given R i ( 1 ) R_{i}^{(1)} , so 𝔼 ⁡ [ ( v ⊤ ​ s i ( 1 ) ) 2 ∣ ( R 1 ( 1 ) , … , R G ( 1 ) ) ] = 𝔼 ⁡ [ ( v ⊤ ​ s i ( 1 ) ) 2 ∣ R i ( 1 ) ] \mathbb{E}[(v^{\top}s_{i}^{(1)})^{2}\mid(R_{1}^{(1)},\ldots,R_{G}^{(1)})]=\mathbb{E}[(v^{\top}s_{i}^{(1)})^{2}\mid R_{i}^{(1)}] . By the tower property: v ⊤ ​ 𝑭 𝒯 ⁡ ( x ) ( 1 ) ​ v \displaystyle v^{\top}\bm{F}^{(1)}_{\mathcal{T}(x)}v = 1 G ​ ∑ i = 1 G 𝔼 ⁡ [ A ^ i ( 1 ) ​ 2 ​ 𝔼 ​ [ ( v ⊤ ​ s i ( 1 ) ) 2 ∣ R i ( 1 ) ] ] \displaystyle\;=\;\frac{1}{G}\sum_{i=1}^{G}\mathbb{E}\!\left[\hat{A}_{i}^{(1)2}\,\mathbb{E}\!\left[(v^{\top}s_{i}^{(1)})^{2}\mid R_{i}^{(1)}\right]\right] ≥ σ min 2 ⋅ 1 G ∑ i = 1 G 𝔼 [ A ^ i ( 1 ) ​ 2 ] , \displaystyle\;\geq\;\sigma_{\min}^{2}\cdot\frac{1}{G}\sum_{i=1}^{G}\mathbb{E}\!\left[\hat{A}_{i}^{(1)2}\right], (14) where the inequality uses the conditional identifiability assumption ( 𝔼 ⁡ [ ( v ⊤ ​ s ) 2 ∣ r = r i ( 1 ) ] ≥ σ min 2 \mathbb{E}[(v^{\top}s)^{2}\mid r=r_{i}^{(1)}]\geq\sigma_{\min}^{2} for all r i ( 1 ) ∈ { 0 , 1 } r_{i}^{(1)}\in\{0,1\} ) and A ^ i ( 1 ) ​ 2 ≥ 0 \hat{A}_{i}^{(1)2}\geq 0 . By definition of σ ^ r \hat{\sigma}_{r} , the sample average of squared advantages satisfies 1 G ∑ i = 1 G A ^ i ( 1 ) ​ 2 = 1 on E 1 ≠ and = 0 otherwise, \frac{1}{G}\sum_{i=1}^{G}\hat{A}_{i}^{(1)2}\;=\;1\quad\text{on }E_{1}^{\neq}\quad\text{and}\quad=0\text{ otherwise,} (15) so 1 G ​ ∑ i 𝔼 ⁡ [ A ^ i ( 1 ) ​ 2 ] = Pr ⁡ [ E 1 ≠ ] ≥ q min \frac{1}{G}\sum_{i}\mathbb{E}[\hat{A}_{i}^{(1)2}]=\Pr[E_{1}^{\neq}]\geq q_{\min} . Hence:

Conclusion: Substituting into ( 14 ): v ⊤ ​ 𝑭 𝒯 ⁡ ( x ) ( 1 ) ​ v ≥ σ min 2 ⋅ Pr ⁡ [ E 1 ≠ ] ≥ q min ⋅ σ min 2 . v^{\top}\bm{F}^{(1)}_{\mathcal{T}(x)}\,v\;\geq\;\sigma_{\min}^{2}\cdot\Pr[E_{1}^{\neq}]\;\geq\;q_{\min}\cdot\sigma_{\min}^{2}. (16) Since all terms in ( 7 ) are PSD, 𝑭 𝒯 ⁡ ( x ) ⪰ 1 K ​ 𝑭 𝒯 ⁡ ( x ) ( 1 ) \bm{F}_{\mathcal{T}(x)}\succeq\tfrac{1}{K}\bm{F}^{(1)}_{\mathcal{T}(x)} , hence: λ min ​ ( 𝑭 𝒯 ⁡ ( x ) ​ ( θ ) ) ≥ 1 K ⋅ q min ⋅ σ min 2 = : 1 K ⋅ c ⁡ ( p ⋆ , G , σ min ) > 0 . \lambda_{\min}\!\left(\bm{F}_{\mathcal{T}(x)}(\theta)\right)\;\geq\;\tfrac{1}{K}\cdot q_{\min}\cdot\sigma_{\min}^{2}\;=:\tfrac{1}{K}\cdot c(p^{\star},G,\sigma_{\min})\;>\;0. (17)

##### Second claim.

Combining ( 12 ) and ( 17 ): λ min ​ ( 𝑭 𝒯 ⁡ ( x ) ​ ( θ ) ) λ min ​ ( 𝑭 x ​ ( θ ) ) ≥ 1 K ⋅ c ⁡ ( p ⋆ , G , σ min ) G ​ δ ⋅ C A ^ 2 ⋅ B s 2 = Ω ⁡ ( 1 δ ) , \frac{\lambda_{\min}(\bm{F}_{\mathcal{T}(x)}(\theta))}{\lambda_{\min}(\bm{F}_{x}(\theta))}\;\geq\;\frac{\tfrac{1}{K}\cdot c(p^{\star},G,\sigma_{\min})}{G\delta\cdot C_{\hat{A}}^{2}\cdot B_{s}^{2}}\;=\;\Omega\!\left(\frac{1}{\delta}\right), (18) since the numerator is independent of δ \delta .∎

### B.3 Bound on C A ^ C_{\hat{A}}

We show that for binary rewards r i ∈ { 0 , 1 } r_{i}\in\{0,1\} , the group-normalized GRPO advantage satisfies C A ^ := sup i | A ^ i | ≤ G − 1 C_{\hat{A}}:=\sup_{i}|\hat{A}_{i}|\leq\sqrt{G-1} . For a non-degenerate group with k k successes ( 1 ≤ k ≤ G − 1 1\leq k\leq G-1 ), r ¯ = k / G \bar{r}=k/G and σ ^ r = ( k / G ) ​ ( 1 − k / G ) \hat{\sigma}_{r}=\sqrt{(k/G)(1-k/G)} . The advantage of a success rollout is: | A ^ success | = 1 − k / G ( k / G ) ​ ( 1 − k / G ) = 1 − k / G k / G = G − k k , |\hat{A}_{\text{success}}|\;=\;\frac{1-k/G}{\sqrt{(k/G)(1-k/G)}}\;=\;\sqrt{\frac{1-k/G}{k/G}}\;=\;\sqrt{\frac{G-k}{k}}, (19) which is maximized at k = 1 k=1 , giving | A ^ | = G − 1 |\hat{A}|=\sqrt{G-1} . By symmetry, the advantage of a failure rollout is k / ( G − k ) \sqrt{k/(G-k)} , also maximized at k = G − 1 k=G-1 , giving G − 1 \sqrt{G-1} . Hence C A ^ = G − 1 < G C_{\hat{A}}=\sqrt{G-1}<\sqrt{G} .∎

## Appendix C SCRL Training Algorithm

## Appendix D OOD Task Performance

##### SCRL generalizes to out-of-distribution tasks.

To examine whether the gains from SCRL transfer beyond the mathematical benchmarks used for training, we evaluate the Qwen3-14B-Base model on three out-of-distribution benchmarks: GPQA, HumanEval, and LiveCodeBench v6. These benchmarks cover different reasoning domains, including scientific question answering and code generation, and are not used for constructing the subproblem curriculum.

As shown in Table 6 , SCRL achieves the best average OOD score, reaching 51.67 compared with 47.20 for the base model and 48.37 for GRPO. SCRL also improves consistently across all three OOD benchmarks, with gains on GPQA (41.41 vs. 38.89 for the base model and 36.86 for GRPO), HumanEval (89.02 vs. 82.93 and 84.15), and LiveCodeBench v6 (24.57 vs. 19.80 and 24.10). These results suggest that SCRL does not merely overfit to the generated curriculum prompts or the training benchmark distribution. Instead, the subproblem curriculum appears to improve transferable reasoning behavior, including domains where solutions require multi-step reasoning or program synthesis rather than the exact mathematical format used during training.

## Appendix E Detailed Experimental Results

Here we provide the complete Pass@ k k performance ( k ∈ { 1 , 2 , 4 , 8 , 16 , 32 , 64 } k\in\{1,2,4,8,16,32,64\} ) for Qwen3-4B-Base.

## Appendix F Implementation Details

### F.1 Hyperparameters

We provide the detailed hyperparameter configurations used in our experiments in Table 8 . All models are trained using the Verl Sheng et al. (2025) framework with the settings specified below.

### F.2 Low-Variance pass@ k k Estimation

We follow the unbiased pass@ k k estimator of Chen (2021) . For each problem x i ∈ 𝒟 x_{i}\in\mathcal{D} , we generate n n sampled rollouts and let c i c_{i} be the number of correct responses. The estimator is pass ​ @ ​ k := 𝔼 x i ∼ 𝒟 ​ [ 1 − ( n − c i k ) ( n k ) ] . \mathrm{pass}@k:=\mathbb{E}_{x_{i}\sim\mathcal{D}}\left[1-\frac{\binom{n-c_{i}}{k}}{\binom{n}{k}}\right]. (20) For evaluation, we select the checkpoint with the best average validation score for each baseline. Using the selected checkpoint, we generate n = 64 n=64 rollouts for each test problem and compute pass@ k k with the estimator above. This protocol is used consistently for all methods and all reported pass@ k k values.

### F.3 Baseline Implementation Details

##### SFT.

We perform Supervised Fine-Tuning (SFT) on the training set using reasoning trajectories synthesized via the DeepSeek V3.2 API. Specifically, we leverage the API to elicit detailed Chain-of-Thought (CoT) reasoning paths for all training samples. The models are fine-tuned on these synthesized trajectories to establish a strong supervised baseline.

##### GRPO.

We utilize the standard implementation of Group Relative Policy Optimization Guo et al. (2025) without any additional reward shaping or gradient modification terms.

##### DAPO.

An RL algorithm featuring decoupled clipping and dynamic sampling mechanisms. We set the clip_ratio_high=0.28 and max_num_gen_batches=10 for filter groups.

##### QuestA.

A curriculum-based reinforcement learning baseline using question augmentation Li et al. (2026a) . We divide the training process into two 150-step phases: (1) an initial phase where the model is provided with a "partial-50" hint (50% of the solution), followed by (2) a second phase where the hint is reduced to "partial-25" (25% of the solution).

##### NuRL.

NuRL ( Chen et al., 2025 ) uses self-generated hints as abstract cues to reduce problem difficulty during RL. Following its offline hint collection setting, we first run 150 steps of GRPO in Stage 1, then use the DeepSeek-V3.2 API to construct a filtered dataset with abstract cues and train NuRL for another 150 steps in Stage 2.

## Appendix G Limitations and Future Work

SCRL has two main limitations. First, subproblem construction relies on an external LLM, which introduces additional preprocessing cost and makes the quality of the curriculum partly dependent on the generator. Second, SCRL is still based on RLVR and therefore requires verifiable answers for subproblems, making it less directly applicable to open-ended tasks without reliable automatic verifiers.

Future work may proceed in two directions. One direction is to extend SCRL’s credit-assignment mechanism to broader multi-turn agent settings, where tasks often naturally contain subgoal-like intermediate progress. Another direction is to design better subproblems, including more fine-grained, robust, and automatically validated curriculum construction methods.

## Appendix H Hardware Setup

All experiments in this work are conducted on three types of NVIDIA GPUs: NVIDIA GeForce RTX 5090, NVIDIA A100-PCIE-40GB, and NVIDIA H20-PCIE-96GB.

## Appendix I Prompt for Subproblem Generation

## Appendix J Chat Template

### J.1 Chat Template of Curriculum Learning

### J.2 Chat Template of Original Problem

## Appendix K Case Study

We present detailed comparisons between the baseline GRPO and our method.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .