LADDER: Self-Improving LLMs Through Recursive Problem Decomposition — part 2/3
Cleaned-text code points [15681, 31670). Bibliography omitted here; source.md includes it.
Integral variant generation occurs at the beginning of LADDER. Throughout the process of performing RL in LADDER we must also perform solution verification. In order to do so, we developed a robust numerical verification framework that balanced accuracy with computational efficiency. The key challenge was ensuring reliable verification across different types of integrals while handling edge cases and potential numerical instabilities.
For each candidate solution to an integral, we employed a multi-point numerical comparison approach. We randomly sampled five points from the domain [-10, 10] and evaluated both the candidate solution and the original integral over small intervals of length 0.1 centered at these points. This interval length was chosen empirically to minimize numerical integration errors while maintaining sensitivity to local solution behavior. The use of multiple small intervals, rather than a single large one, helped detect both local and global errors in the solutions.
Our verification protocol included several key components: • Singularity handling : When a sampled point was near a singularity (detected through rapid value changes or numerical overflow), we automatically resampled to a new point. This adaptive sampling ensured robust verification even for functions with challenging behavior.
• Numerical precision : Solutions were deemed correct if their values exhibited a relative difference of 10 − 2 10^{-2} or less compared to numerical integration results across all test intervals. We found empirically that this tolerance level effectively balanced false positives and negatives.
• Timeout management : To handle computationally intensive integrals, we implemented a 2-second timeout for each verification attempt. If timeout occurred, new evaluation points were sampled, with up to three retries before marking the solution as unverifiable.
• Edge case detection : Early iterations revealed that models could exploit verification weaknesses by outputting trivial solutions (like the integration symbol itself) or degenerate forms. We added specific filters to detect and reject such cases, ensuring solutions demonstrated genuine understanding.
All numerical integrations were performed using adaptive quadrature methods, with automatic precision adjustment based on the integrand’s complexity. This approach provided reliable results even for oscillatory and highly nonlinear functions, while maintaining reasonable computational efficiency.
We note, however, that the numerical verifier may not be 100% accurate. We opted for a numerical approach rather than a symbolic one because many problems beyond integration can be numerically verified despite lacking a symbolic solver.
We apply the above solution verification in our RL Protocol in section 3.1.4 and experiments in sections 3.2.1 , 3.2.2 and 3.2.3 . In our MIT experiments, final benchmark results are verified directly against the official solutions.
#### 3.1.4 Reinforcement Learning Protocol
We decided to employ Group Relative Policy Optimization (GRPO) in both LADDER and TTRL [ 11 ] . GRPO does not use a separate critic model and instead estimates the baseline from group scores, improving efficiency and reducing memory. For each question q q , GRPO samples a group of outputs o 1 , o 2 , … , o G {o_{1},o_{2},...,o_{G}} from the old policy π θ o l d \pi_{\theta_{old}} and then optimizes the policy model π θ \pi_{\theta} by maximizing the following objective:
J GRPO ( θ ) = 𝔼 [ q ∼ P ( Q ) , { o i } i = 1 G ∼ π θ old ( O ∣ q ) ] \displaystyle J_{\mathrm{GRPO}}(\theta)=\mathbb{E}\Bigl[q\sim P(Q),\;\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}(O\mid q)\Bigr] (1) 1 G ∑ i = 1 G ( min ( π θ ( o i ∣ q ) π θ old ( o i ∣ q ) A i , clip ( π θ ( o i ∣ q ) π θ old ( o i ∣ q ) , 1 − ε , 1 + ε ) A i ) − β D KL ( π θ ∥ π ref ) ) \displaystyle\frac{1}{G}\sum_{i=1}^{G}\Bigl(\min\Bigl(\frac{\pi_{\theta}(o_{i}\mid q)}{\pi_{\theta_{\mathrm{old}}}(o_{i}\mid q)}\,A_{i},\;\mathrm{clip}\Bigl(\frac{\pi_{\theta}(o_{i}\mid q)}{\pi_{\theta_{\mathrm{old}}}(o_{i}\mid q)},\,1-\varepsilon,\,\,1+\varepsilon\Bigr)\,A_{i}\Bigr)\;-\;\beta\,D_{\mathrm{KL}}\bigl(\pi_{\theta}\,\|\,\pi_{\mathrm{ref}}\bigr)\Bigr)
D KL ( π θ ∥ π ref ) = π ref ( o i ∣ q ) π θ ( o i ∣ q ) − log ( π ref ( o i ∣ q ) π θ ( o i ∣ q ) ) − 1 D_{\mathrm{KL}}(\pi_{\theta}\,\|\,\pi_{\mathrm{ref}})=\frac{\pi_{\mathrm{ref}}(o_{i}\mid q)}{\pi_{\theta}(o_{i}\mid q)}\;-\;\log\Bigl(\frac{\pi_{\mathrm{ref}}(o_{i}\mid q)}{\pi_{\theta}(o_{i}\mid q)}\Bigr)\;-\;1 (2)
where ϵ \epsilon and β \beta are hyperparameters, and A i A_{i} is the advantage, computed using a group of rewards r 1 , r 2 , … , r G {r_{1},r_{2},...,r_{G}} corresponding to the outputs within each group:
A i = r i − mean ( { r 1 , r 2 , … , r G } ) std ( { r 1 , r 2 , … , r G } ) A_{i}=\frac{r_{i}-\mathrm{mean}\bigl(\{r_{1},r_{2},\dots,r_{G}\}\bigr)}{\mathrm{std}\bigl(\{r_{1},r_{2},\dots,r_{G}\}\bigr)} (3)
We adopted a simple, rule-based reward model. The reward model is kept simple in order to be straightforward to apply to other verifiable domains in the future. The reward model consists of two rewards:
• Accuracy reward: Using the solution verification method described in 3.1.3 , we evaluate whether the response is correct or not. The model is required to provide the final answer in a specified format (i.e. in <ANSWER> </ANSWER> tags), enabling reliable rule-based verification of correctness.
• Format reward: In addition to the accuracy reward model, we employ a format reward that encourages the model to put its answer in <ANSWER> </ANSWER> tags.
We perform the RL runs with a KL divergence coefficient of 0.001. Training batch size and number of epochs differ among our experiments.
#### 3.1.5 Test-Time Reinforcement Learning
Test-Time Reinforcement Learning (TTRL) extends our LADDER framework to inference time, enabling dynamic adaptation to challenging problems directly during testing. Upon encountering a difficult integration problem, TTRL generates a focused set of related variants and conducts targeted reinforcement learning specifically for that problem. By recursively decomposing the challenging problem into simpler variants and learning from this focused curriculum, TTRL allows the model to rapidly develop problem-specific expertise without hand-crafting variants or architectural modifications. This provides a novel approach to scaling compute at test-time - rather than simply increasing output sampling or model size, TTRL leverages compute to dynamically improve the model’s problem-solving capabilities through focused practice on variants of the specific challenge at hand.
The same three components of LADDER are also used in Test-Time Reinforcement Learning (TTRL). For each question at test-time, there are two steps. First, we generate a tree of variants for the test question at hand. Second, we perform the reinforcement learning protocol described in section 3.1.4 on a base model using the variant tree as the training set. The resulting model should then have significantly improved mathematical integration capabilities tuned to the test question at hand. The test question is then answered using the tuned model, and finally the model is rolled back to its original parameters for the next test question. See Algorithm 2 for TTRL pseudocode, where V TTRL , i V_{\text{TTRL},i} is the set of variants generated for the i i th test question and a i a_{i} is the answer to the i i th test question.
As in LADDER, the number of variants generated per question is a hyperparameter, which we vary for different experiments. We experiment with TTRL in section 3.2.3 .
### 3.2 Experiments
#### 3.2.1 Llama 3B Experiments
We first experiment with LADDER using a Llama 3.2 3B parameter model as our base architecture to allow us to more quickly run ablation experiments. To establish our evaluation dataset, we developed a comprehensive collection of 110 indefinite integration problems, combining questions sourced from university-level mathematics curricula with synthetically generated problems using GPT-4o. To ensure appropriate difficulty calibration, we benchmarked each problem to verify it was solvable by GPT-4o but beyond the capabilities of the base Llama 3.2 3B model. This was done purposefully to benchmark performance on problems that can still be solved by an LLM but are outside the scope of Llama 3.2 3B.
We randomly split this collection into a training set of 10 problems and a test set of 100 problems. we generated approximately 500 variants for each of our 10 training problems. To validate the diversity of our variant set, we performed exact matching against our test set and found minimal overlap (only 5 matches among 5,000 variants). The small training set was deliberately chosen to demonstrate our method’s effectiveness in generating useful variants from limited examples. Approximately half of the problems were sourced from curriculum textbooks, with the remainder synthetically generated by GPT-4o. Notably, the synthetic problems were indistinguishable from curriculum-sourced ones.
During training, models were prompted to express solutions in sympy algebraic notation without integration constants, ensuring consistent evaluation across different but mathematically equivalent expressions of the same solution. This standardization was crucial for reliable verification of solution correctness during the training process.
We performed two RL experiments with Llama 3.2 3B. The first experiment trained on only the 10 question training set (RL w/o variants). The second experiment trained on only variants generated from the 10 question training set (RL w/ variants, i.e. LADDER). Both runs used identical hyper parameters and were continued until performance plateaued.
#### 3.2.2 MIT Integration Bee (LADDER)
Building off the findings from our Llama 3B experiemtns, we extended our methodology to the 2025 MIT Integration Bee qualifying examination, an annual competition that attracts both undergraduate and graduate students from MIT, with participants often having competitive mathematics backgrounds. The qualifying exam, which serves to select 16 finalists, contains 20 questions and has a typical qualifying threshold of 73%, though most participants score between 15-30%, with 50% to 73% out of 20 considered strong performance. Students are given 20 minutes to attempt the entire exam.
Using the DeepSeek-R1 distilled Qwen 2.5 7B model, we applied our variant generation approach to historical qualifying exams from 2010-2024. Our variant generation followed a two-level tree structure - first generating variants from each source problem, then generating additional variants from those first-level variants. We capped recursion at depth two, as preliminary experiments showed third-level variants became trivially simple and lost mathematical relevance to the original problems. With this approach we generated 9,000 variants.
Through experimentation with different difficulty distributions, we settled on prompting the model to generate 70% easier and 30% equivalent variants than their parent problems. While we initially attempted to include more challenging variants, we found this skew toward easier problems created more effective learning trajectories. Notably, many "equivalent" variants still introduced novel mathematical patterns while maintaining similar difficulty levels, contributing to the model’s generalization capabilities.
We verified that none of the 2025 exam questions appeared in our variant set, though we note this precaution was primarily for methodological rigor rather than necessity, as our model never accessed the test set or solutions during training. We apply the same hyperparameters as in the 3B experiments, except for modifying the batch size to 128.
#### 3.2.3 MIT Integration Bee (LADDER + TTRL)
After applying LADDER, there remain certain questions which the tuned model continues to fail to answer. For each of these questions we further apply TTRL. For each unsolved problem, we generate a tree of variants following the same process illustrated in Figure 1 , but limited to two levels of depth and approximately 800 total variants. Using these problem-specific variants, we conduct 100 steps of reinforcement learning with identical RL parameters as in the MIT Integration Bee LADDER setup. The problem is considered "solved" if the model produces a correct solution at any point during this process, as verified by our numerical integration framework.
TTRL’s approach of on-the-fly data synthesis means it can handle novel integrals better by expanding its training set - focused on that problem - at test time. This is especially important for challenges such as the MIT Integration Bee, where many integrals are cleverly constructed and might be out-of-distribution relative to standard calculus problems. Further, TTRL provides a systematic approach to scaling performance through additional compute. Rather than simply increasing sampling or temperature parameters, TTRL enables active improvement of problem-solving capabilities through focused practice at test-time.
## 4 Results
### 4.1 Llama 3B Experiments
Our experiments evaluated whether generating easier variants could enable effective reinforcement learning in mathematical integration tasks on Llama 3.2 3B. We tested four conditions: base model with pass@1 samping, base model with pass@10 sampling, RL without variants (only on the training set), and LADDER (RL with variants from the training set).
In Figure 3 , we observe a consistent and rapid improvement from step 0 to 250, with the model’s performance quickly ascending before ultimately plateauing. This swift progression suggests an effective learning mechanism, characterized by substantial performance gains that stabilize around 82% accuracy as the model approaches its performance ceiling.
LADDER achieved 82% accuracy on the test set, significantly outperforming both the base model with pass@1 sampling (1%) and with pass@10 sampling (2%). RL without variants consistently failed, with performance never exceeding 3% before collapsing to 0% within 30 training steps, highlighting the necessity of a smooth difficulty gradient for successful training (see Figure 4 ).
Analysis of the remaining unsolved problems reveals nuanced challenges in the model’s mathematical integration capabilities. The model demonstrates difficulty with integrals requiring multiple technique combinations and novel substitution patterns. These complex scenarios highlight limitations in the current approach, suggesting that while the variant generation successfully teaches individual techniques, synthesizing these methods for more intricate problems remains a challenge.
We observed a clear upward trend in performance as the number of generated variants increased, with no definitive plateau in the graph (see Figure 3 ). While the rate of improvement appears to slow beyond 6,000 variants, the data suggests that continued scaling may still yield further gains. This indicates that generating more variants could enhance performance, though with diminishing returns relative to earlier stages of training. Rather than identifying a strict optimal corpus size, our findings highlight a trade-off between computational cost and incremental improvements, suggesting that additional scaling may remain beneficial depending on resource availability.
The stark improvement from base model pass@10 (2%) to LADDER (82%) demonstrates genuine acquisition of mathematical abilities rather than merely improved output sampling. The consistent failure of RL without variants under identical hyperparameters confirms that success stems from the carefully constructed difficulty gradient rather than the RL algorithm itself, suggesting potential applications to other complex reasoning tasks where direct training proves ineffective.