LADDER: Self-Improving LLMs Through Recursive Problem Decomposition — part 1/3

Cleaned-text code points [0, 15681). Bibliography omitted here; source.md includes it.

# LADDER: Self-Improving LLMs Through Recursive Problem Decomposition

###### Abstract

We introduce LADDER (Learning through Autonomous Difficulty-Driven Example Recursion), a framework which enables Large Language Models to autonomously improve their problem-solving capabilities through self-guided learning by recursively generating and solving progressively simpler variants of complex problems. Unlike prior approaches that require curated datasets or human feedback, LADDER leverages a model’s own capabilities to generate easier question variants. We demonstrate LADDER’s effectiveness on the subject of mathematical integration, improving Llama 3.2 3B’s accuracy from 1% to 82% on undergraduate-level problems and enabling Qwen2.5 7B Deepseek-R1 Distilled to achieve 73% on the MIT Integration Bee qualifying examination. We also introduce TTRL (Test-Time Reinforcement Learning), where we perform reinforcement learning on variants of test problems at inference time. TTRL enables Qwen2.5 7B Deepseek-R1 Distilled to achieve a state-of-the-art score of 90% on the MIT Integration Bee qualifying examination, surpassing OpenAI o1’s performance. These results show how self-directed strategic learning can achieve significant capability improvements without relying on architectural scaling or human supervision.

August 24, 2026

## 1 Introduction

Reinforcement Learning (RL) has emerged as a highly effective approach for training Large Language Models (LLMs), yet its success hinges critically on the availability of appropriate training tasks [ 5 , 9 , 10 , 12 ] . A fundamental challenge lies in obtaining verifiable tasks that match the model’s current capabilities. For RL to be effective, tasks must form a gradient of difficulties that allows for incremental learning progress [ 8 ] . When tasks exceed the model’s current abilities, the training process not only stalls but can lead to catastrophic collapse, resulting in degraded performance. This challenge is particularly acute in domains requiring complex reasoning, where the gap between simple and advanced tasks can be substantial [ 4 , 12 ] .

We propose Learning through Autonomous Difficulty-Driven Example Recursion (LADDER), a framework that enables LLMs to autonomously improve their problem-solving capabilities through strategic self-guided learning. The key insight is that models can bootstrap their own learning by recursively generating and solving progressively simpler variants of complex problems. For each challenging problem, LADDER prompts the model to create multiple easier variants, forming a natural difficulty gradient. This process continues recursively, with each variant spawning simpler sub-variants, until reaching problems the model can reliably solve. The solutions to these simpler problems then provide stepping stones for tackling progressively harder variants. We demonstrate that this self-bootstrapping approach achieves dramatic improvements beyond what’s possible through standard techniques like pass@k sampling - enabling models to reliably solve problems that were previously far beyond their capabilities.

Unlike previous approaches requiring carefully curated datasets or human feedback, LADDER leverages the model’s existing capabilities to create a natural difficulty gradient, allowing for systematic improvement through reinforcement learning with verifiable rewards. The framework requires only a reliable verification mechanism - in our case, numerical integration for checking solutions. This enables the model to assess its own progress and guide its learning trajectory without human intervention.

We demonstrate LADDER’s effectiveness on mathematical integration tasks, achieving remarkable improvements across multiple benchmarks. Using this approach, we improve a Llama 3B model’s accuracy from 1% to 82% on undergraduate-level integration problems. When applied to the challenging 2025 MIT Integration Bee examination, LADDER enables a 7B parameter model to achieve 73% accuracy, significantly outperforming much larger models, such as GPT-4o (42%), and typical human performance (15-30%). These results showcase how strategic problem decomposition and verified self-learning can achieve substantial capability improvements without relying on architectural scaling or human supervision.

Building on LADDER’s self-improvement framework, we propose Test-Time Reinforcement Learning (TTRL), a novel approach that extends these principles to inference time. TTRL dynamically generates problem variants during test-time and applies reinforcement learning to refine the model’s solutions, effectively creating a micro-learning process for each test instance. By leveraging the same verification mechanisms used in training, TTRL enables the model to further improve its performance. When applied to the 2025 MIT Integration Bee, TTRL boosts accuracy from 73% - with just LADDER - to 90%, demonstrating how scaling test-time compute through strategic problem decomposition can yield substantial performance improvements. We achieve state of the art accuracy, outperforming significantly larger models, such as OpenAI’s o1.

Thus, we make the following contributions: • We propose a novel framework for autonomous model improvement through recursive problem decomposition and self-guided learning via reinforcement learning with GRPO.

• We develop a systematic method for generating and verifying problem variants that create natural difficulty gradients, requiring only numerical verification.

• We demonstrate significant empirical improvements on mathematical reasoning tasks, improving a Llama 3B model from 2% to 82% on undergraduate integration problems and achieving 73% accuracy on the MIT Integration Bee with a 7B model, matching SoTA performance

• We introduce Test-Time Reinforcement Learning (TTRL), a method for scaling compute at inference time through variant generation and reinforcement learning, boosting performance on the MIT Integration Bee from 73% to 90%.

## 2 Related Work

##### Self-Improvement and Automated Curriculum Generation in LLMs.

Haluptzok et al. (2023) present a self-play setup where a code-focused LLM continually invents and solves new programming puzzles, checks solutions with a Python interpreter, and fine-tunes on correct results [ 3 ] . We build upon the generate → solve → verify → learn cycle, adding a more explicit curricular element to guide the model’s step-by-step improvement.

Recent advances such as STaR (Self-Taught Reasoner) have illustrated that LLMs can improve their reasoning skills by learning from their own generated chain-of-thoughts—effectively acting as both student and teacher [ 15 ] . Unlike traditional self-play techniques that require competitive dynamics, these methods harness the model’s intrinsic capacity to generate and evaluate intermediate reasoning steps, automatically curating training examples that progressively reduce task complexity. Our work extends this line of research by systematically transforming unsolvable problems into tractable sub-problems, thereby constructing an end-to-end self-improving framework that requires no human annotation or intervention.

##### Test-Time Compute Scaling and Reasoning Improvements.

Recent work has explored various approaches to improving model performance through increased compute at inference time. "Let’s Verify Step by Step" [ 6 ] demonstrated that breaking down verification into explicit steps and allocating additional computation to each verification component can significantly improve performance. Similarly, Tree of Thoughts [ 14 ] showed how systematically exploring multiple reasoning paths during inference can lead to better problem-solving capabilities. These approaches typically focus on increasing the length and deliberation of model outputs, either through structured prompting, such as s1, or systematic exploration of solution spaces, such as self-consistency [ 7 , 12 ] . Our work differs fundamentally from these approaches by introducing dynamic learning at test-time - rather than just increasing reasoning length or exploring multiple paths, TTRL enables the model to actually improve its capabilities through practice on related problems. This represents a shift from static inference-time techniques to dynamic adaptation through targeted learning.

##### Reinforcement Learning Approaches for LLM Self-Improvement.

In recent years, reinforcement learning has emerged as a pivotal strategy for enabling language models to self-improve through iterative self-correction and feedback. Notably, OpenAI’s o1 model uses reinforcement learning to refine its chain-of-thought reasoning—learning to “think” step by step [ 9 ] . Similarly, DeepSeek’s R1 model leverages a reinforcement learning framework that minimizes human supervision by generating and self-evaluating its own training data [ 2 ] .

Importantly, recent work has shown that RL approaches may generalize much better than Supervised Fine-Tuning (SFT), which appears to memorize instead [ 1 , 13 ] . RL approaches have also shown to be highly effective in generalizing to out-of-distribution tasks, particularly in competitive mathematics [ 13 ] . These results strongly suggest that reinforcement learning offers a more promising path to self-improvement than supervised fine-tuning.

These innovations underscore a paradigm shift—from externally curated curricula and human-labeled data toward autonomous, feedback-driven self-improvement in language models.

## 3 Methodology

### 3.1 Algorithm Design (LADDER & TTRL)

#### 3.1.1 LADDER

LADDER is a structured framework designed to enhance LLM problem-solving capabilities through recursive problem decomposition and reinforcement learning. We demonstrate the effectiveness of LADDER in the domain of mathematical integration. LADDER consists of the following components:

• Variant Generation : A structured method for generating trees of progressively simpler variants of complex problems, establishing a natural difficulty gradient.

• Solution Verification : A numerical integration method to verify the solution of an integral.

• Reinforcement Learning : The protocol used to train the base model on the variant trees.

LADDER is a two-step algorithm. First, the training set of integrals is collected and a tree of variants is generated for each integral as described in section 3.1.2 . Second, we perform the reinforcement learning protocol described in section 3.1.4 on a base model using the set of variant trees as the training set. The resulting model should then have significantly improved and generalized mathematical integration capabilities, and can then be prompted to solve new integrals. See Algorithm 1 for LADDER pseudocode. V LADDER V_{\text{LADDER}} is the set of variants generated for all questions in the train set of integrals and a i a_{i} is the answer to the i i th test question.

The number of variants N N generated per question is a hyperparameter, which we vary for different experiments. We experiment with LADDER in sections 3.2.1 and 3.2.2 .

#### 3.1.2 Variant Generation

The variant generation process required careful design to ensure sufficient diversity and appropriate difficulty scaling. We found that naive approaches to variant generation - simply asking the model to generate easier versions - led to repetitive patterns and limited diversity. Through experimentation, we developed a multi-stage process that dramatically improved the quality and variety of generated variants.

V LADDER ← { v i , j ∈ G ( Q train [ i ] , j ) | 1 ≤ i ≤ | Q train | , 1 ≤ j ≤ N } V_{\text{LADDER}}\leftarrow\{v_{i,j}\in\text{G}(Q_{\text{train}}[i],j)\;|\;1\leq i\leq|Q_{\text{train}}|,1\leq j\leq N\}

First, we used the base model to generate an extensive set of mathematical transformations that could be applied to integrals, categorized by their impact on problem difficulty. These included operations ranging from simple (e.g., reducing exponent values, simplifying denominators) to more complex (e.g., introducing nested functions, combining multiple function types). This transformation library served as a foundation for guiding variant generation.

Second, for each integral, we randomly sampled 3-5 transformations and provided them as explicit suggestions to the variant generation model. Critically, we found that generating variants in batches of 10 per prompt significantly improved diversity - smaller batch sizes led to more repetitive outputs, while larger batches reduced quality. To further enhance variation, we employed two key techniques:

1. Temperature cycling: We dynamically varied the sampling temperature between 0.8 and 1.4 across prompts. This helped balance between creativity and mathematical validity.

2. Persona-based prompting: We prompted the model to adopt different mathematical perspectives (e.g., "think like Euler focusing on series", "approach like Gauss looking for patterns").

The combination of batch generation, varied temperatures, and rotating personas proved crucial led to significantly more diverse variants. Without them the model would often converge to generating very similar and often repeating variants.

Third, to ensure sufficient coverage of the difficulty space, we applied this generation process recursively, generating a tree of variants, each a simpler integral than its parent. We found that this also helped improve variant diversity. This recursive approach helped build natural difficulty gradients - each problem could spawn multiple easier variants, which in turn could generate even simpler problems. We capped tree depth at three to maintain problem relevance. The following is an example of one root-to-leaf path in a variant tree:

Original problem: ∫ x 2 + 1 x 4 + 2 ​ x 2 + 1 ​ 𝑑 x \displaystyle\int\frac{x^{2}+1}{x^{4}+2x^{2}+1}dx Level 1 variant: ∫ x 2 x 4 + 1 ​ 𝑑 x \displaystyle\int\frac{x^{2}}{x^{4}+1}dx Level 2 variant: ∫ 1 x 2 + 1 ​ 𝑑 x \displaystyle\int\frac{1}{x^{2}+1}dx Level 3 variant: ∫ 1 x 2 ​ 𝑑 x \displaystyle\int\frac{1}{x^{2}}dx

Quality control presented a significant challenge in our variant generation process. While we prompted the model to generate "easier" or "equivalent" variants, the actual difficulty of the resulting integrals often deviated substantially from the intended level. Small perturbations in coefficients or function composition could transform seemingly simple integrals into much harder ones - for instance, changing a coefficient from 1 to 2 in a rational function could introduce complex roots that make the integral significantly more challenging.

For our Llama experiments, approximately 8% of generated variants were effectively unsolvable (i.e. it took a symbolic system more than 5 seconds to solve), and many more were substantially harder than their parent problems despite being intended as simpler variants. While these harder variants were naturally ignored during training through zero reward signals, they consumed significant computational resources in verification attempts and represented wasted generation capacity. Future work could explore methods to better constrain variant generation to maintain intended difficulty levels while preserving mathematical diversity.

We apply the above three-step variant generation method to generate the sets of variant integrals in both LADDER and TTRL. V LADDER V_{\text{LADDER}} is the set of variants for all questions in the train set Q train Q_{\text{train}} and V TTRL , i V_{\text{TTRL},i} is the set of variants for the i i th question in the test set, Q test ​ [ i ] Q_{\text{test}}[i] . See Algorithms 1 and 2 .

#### 3.1.3 Solution Verification