RLTL;DR: Self-improvement by Internalizing Self-generated Feedback — part 2/6

Cleaned-text code points [15920, 31330). Bibliography omitted here; source.md includes it.
RLTL;DR overcomes this issue by introducing a new self-distillation objective that trains the model to connect insights directly to the task. We train the LLM to predict the insights { f i } i = 1 I \{f_{i}\}_{i=1}^{I} it generated in previous rollouts (and has in context later attempts) using the task description g g alone as input π θ  ( f 1 , … , f I | g ) \pi_{\theta}(f_{1},\dotsc,f_{I}|g) . This internalizes the knowledge "on this sort of task, keep these sort of things in mind", with generalization to similar tasks happening thanks to semantic smoothness ( Nakkiran et al., 2026 ) . We implement this objective as a standard supervised fine-tuning (SFT) loss for next-token prediction. We denote this SFT loss by ℒ SFT  ( θ ) \mathcal{L}_{\text{SFT}}(\theta) and add it to the GRPO loss to obtain the final RLTL;DR training objective ℒ = ℒ GRPO + λ  ℒ SFT \mathcal{L}=\mathcal{L}_{\text{GRPO}}+\lambda\mathcal{L}_{\text{SFT}} , where λ \lambda is the insight internalization strength. We show in Section E.3 that RLTL;DR is robust to the choice of λ \lambda and that λ = 0.5 \lambda=0.5 is a good default.

For example, in Figure 1 the agent has gathered two insights from previous failed attempts. In the third attempt, it has them in context as two additional user chat messages. ℒ SFT \mathcal{L}_{\text{SFT}} backpropagates to increase the log likelihoods of the tokens "Remember to paginate search results." given the context "<|im_start|> user\nYou are an assistant that... Task: Start a playlist that’s long enough for my workout. My workout plan is in my notes.\n<|im_end|>\n<|im_start|>user\n==> A previous Attempt 1 on this same task FAILED the verifier. <==\nHint on what went wrong: " . It backpropagates "Sort the notes to find the most recent one." the same way, conditional on the task, first insight, and start of the second insight message. Note that π θ  ( f 1 , f 2 | g ) \pi_{\theta}(f_{1},f_{2}|g) is part of the chat history h ~ 3 \tilde{h}_{3} anyways, hence the log likelihoods are already computed in the RL update phase. So, practically, ℒ SFT \mathcal{L}_{\text{SFT}} is simply implemented by editing the backpropagation mask of the context of the third attempt, without increasing runtime.

Some important distinctions between RLTL;DR and prior work are that ℒ SFT \mathcal{L}_{\text{SFT}} in RLTL;DR is used solely to internalize the insights about this (and similar) tasks. Other work ( Song et al., 2026 ) trains on π θ  ( f | τ ) \pi_{\theta}(f|\tau) , i.e., improving the policy’s capability to self-critique given an attempt. We find this to underperform compared to RLTL;DR direct prediction of insights from the instruction alone ( Section F.1 ). Further, in RLTL;DR, ℒ SFT \mathcal{L}_{\text{SFT}} is used solely as a means to internalize the insights and change behavior on the task. RLTL;DR never generates insight via the π θ  ( f | g ) \pi_{\theta}(f|g) we backpropagate on, neither during training, where the inserted insight comes from analyzing previous failed attempts, nor at test time, where the agents needs to one-shot solutions without any insight or sequential attempts. Last, RLTL;DR does not run out of context budget because each insight inserted into the context averages 17 17 tokens. We provide full implementation details in Appendix B .

## 4 RLTL;DR breaks through the learning barrier

### 4.1 Experiment setup

Datasets. In Appworld ( Trivedi et al., 2024 ) , the policy has to retrieve information and conduct state-changing actions on a simulated device via multi-step tool-calling. It has 90 train, 57 dev, 168 test-normal, and 417 test-challenge tasks. Since this dataset is relatively small (especially after filtering them to very hard splits below), we also use a proprietary dataset similar to Appworld, but with 16k train tasks, that we call Synthetic-API (SAPI). For more general coding, we use 2641 Leetcode problems ( Xia et al., 2025 ) .

The self-improvement scenarios that we aim to study are characterized by tasks that are so hard that the models are struggling to find solutions even with high budgets. To emulate this difficulty, we subsample the above datasets: We use tasks where our policy, Qwen 3.5 9B (with thinking), has no successes in 128 attempts, i.e., Pass@128=0. This filters down SAPI to 458 tasks. For the smaller Appworld dataset, we combine the train, dev, and test-normal splits (and keep test-challenge unseen), leaving 34 tasks after filtering to the Pass@128=0 set. Leetcode has 123 remaining tasks.

Baselines. We compare RLTL;DR to three baselines. RLTF-SD ( Song et al., 2026 ) is a recent method, similar in kind. It uses a Self Distillation loss to train a rollout generated with insight into the policy without insight. For fairness, we provide it with the same sequential rollouts and insight generation strategies as for our method. Second, we compare against Strategy-guided Exploration (SGE, Szot et al. , 2026 ). This aims to explore more solutions by prepending summaries of previous failed or successful attempts on a task (though without insights on what went wrong), and prompting the model to try something else. Finally, we compare against a standard GRPO baseline. We tune our GRPO baseline extensively. Our Qwen 3.5 9B GRPO baseline trained only on Appworld-train achieves 72.2% Pass@1 on test-challenge. As of the release of this paper, the best agent on the official Appworld leaderboard is a frontier model in a custom harness, at 73.4 Pass@1.

Deconfounded evaluation. Since some of the rollouts are conditioned on insights in their context during training, we deconfound our metrics. Our curves and metrics, both during train and eval, always show the performance on rollouts without insight in context (and are macro-averaged across all tasks, see Appendix D ). This allows to compare fairly, treating insight only as a train aid.

We use the official heldout sets for evaluation, without filtering for hard tasks, to show ensure the policies do not degradate outside hard tasks. Appworld test-challenge has 417 new tasks on both the seven seen and two new apps. SAPI has 1624 tasks on 4 unseen apps. Leetcode has 228 unseen problems. We take multiple attempts to achieve 2k rollouts for each dataset.

### 4.2 Results

Figure 2 shows the policy’s Pass@1 throughout training, on rollouts without insights in context. Insights thus only acted indirectly to improve learning signal in previous batches, and performance can be directly compared. On SAPI, every task has been seen once after ≈ \approx 170k environment interactions, and reported performances are before backpropagating on any given task, so that on SAPI, the train curve until ≈ \approx 170k can be seen as eval curves on a rolling basis. Appworld and Leetcode loop every 13k and 9k steps, so we defer to the heldout splits below for judging generalization.

GRPO fails to learn on these very hard tasks, staying flat at 0% to 1%. This is because GRPO is starved of successful rollouts and thus learning signal. SGE behaves similarly. Although it conditions on previous attempts and it is highlighted that they failed, it does not reflect on failed unit tests. We observe that this misleads next rollouts (reproducing Cheng et al. (2026) , see also Section 6 ).

RLTL;DR, on the other hand, breaks through the learning barrier and reaches a Pass@1 of 12–13%. As can be seen from the SAPI curve before 170k steps, and the heldout splits in Table 1 , the internalization generalizes insights to new tasks. We do not see this on Leetcode. Both RLTL;DR and RLTF-SD find solutions during training (GRPO does not) but seem to overfit in the process. We discuss this in Section 7 . When including rollouts with insights in context, train-time Pass@1 is 14–31%, and train-time Pass@k is 14–59%, demonstrating how RLTL;DR finds learning signals on previously impossible tasks.

RLTF-SD also learns. We refrain from claims on either approach outperforming. Instead, we see RLTL;DR and RLTF-SD as two promising approaches of acquiring off-policy knowledge (the same knowledge, since in our experiments they both use our sequential sampling and insight generation pipeline). But while RLTF-SD learns by backpropagating examples, RLTL;DR learns from the high-level insight. These two complementary backpropagation signals can be combined by simply changing the gradient mask in RLTF-SD. We observe performance gains with this in Section E.9 .

We also train on a slightly easier split of Synthetic API with a baseline Pass@1 = = 4% in Section E.6 , with equivalent observations. On the unfiltered datasets in Section E.7 , where only ≤ \leq 3–5% of tasks are frontier-difficult, GRPO is able to learn, and RLTL;DR neither helps nor hurts performance (except Leetcode). We thus see RLTL;DR as a method for training on challenging tasks.

## 5 Reducing to the secret sauce: From RLTL;DR to SFTL;DR

In the development of RLTL;DR, internalizing the insight via ℒ SFT \mathcal{L}_{\text{SFT}} was the switch that enabled self-improvement on very hard tasks. In this section, we reduce to only ℒ SFT \mathcal{L}_{\text{SFT}} , and make the perhaps surprising finding that we can learn only from insights, without full rollouts.

### 5.1 Experiment setup

##### Dataset.

We dedicate the remainder of this paper to SAPI, due to its sheer size. We use a split containing the 458 frontier-difficult tasks, plus 184 very difficult tasks as explained in Section E.6 . Qwen 3.5 9B achieves 4% Pass@1 on this split. The 642 tasks are split into 256 eval and 386 train tasks. GRPO training still fails, while RLTL;DR achieves 21.5% Pass@1.

##### RLTL;DR ablations.

Starting with the standard RLTL;DR run (GRPO loss and SFT loss on insight with strength λ = 0.5 \lambda=0.5 ), we first reduce and then fully deactivate λ \lambda . Then, on the contrary, we deactivate the GRPO loss and train only with the SFT loss. These trainings are all online, so rollouts are collected as the policy improves.

##### Standard SFT.

We also train on all successful rollouts collected throughout the standard RLTL;DR run via offline SFT. These SFT runs use a log likelihood loss on the agent actions in the rollouts, with insights in the contexts but without SFT loss on the insights, over 10 epochs. Besides Pass@1 on train and heldout tasks, we track the number of tokens we backpropagate on, as well as the number of tokens we need to forward calculate to generate the context KV caches. We train on different amounts of SFT data to be able to compute-match the SFTL;DR results.

##### SFTL;DR.

In the runs named SFTL;DR, we use the same rollouts but only apply the SFT loss on the insight tokens π θ  ( f 1 , … , f I | g ) \pi_{\theta}(f_{1},\dots,f_{I}|g) , without backpropagating (or even forward calculating) the actual rollouts, which would come autoregressively after the insight. The standard SFTL;DR setup backpropagates on all insights in each rollout, possibly multiple times (insights appear in multiple rollouts per task, and in multiple combinations). In SFTL;DR-deduplicated, we further simplify this training. We create unique tuples ( g , f ) (g,f) of the insights per task, 4k in total, and then train π θ  ( f | g ) \pi_{\theta}(f|g) one-by-one. This resembles a training where we only train "on this task, remember this insight".

### 5.2 Reducing RLTL;DR to just SFTL;DR

Table 2 shows the Pass@1 on the train and unseen eval tasks, both evaluated without insight in context or sequential sampling. The first four runs show that ℒ SFT \mathcal{L}_{\text{SFT}} is the driving factor in RLTL;DR. Reducing its mixture weight from λ = 0.5 \lambda=0.5 to λ = 0.01 \lambda=0.01 or 0 0 severly impacts both train and eval performance (increasing it beyond λ > 0.5 \lambda>0.5 did not further improve it). While the model might be learning how to solve tasks given insight, it does not internalize the insight itself to one-shot solve tasks once insight is not available in context. In fact, entirely removing the GRPO loss and only utilizing the SFT loss on insight tokens (but still in the RL setup of interleaved rollout and update phases) recovers almost the full performance of RLTL;DR. Nevertheless, the additional learning signal from the full rollouts lets runs with non-zero GRPO loss converge faster ( Section E.3 ), so we recommend leaving it activated.

Classical SFT on the rollouts collected during the RLTL;DR is slightly above RLTL;DR’s performance, though still close to standard deviation. Indeed, we can reduce the number of train rollouts from 3.5k to 100 without losing much performance, as performance scales sub-linearly with the amount of input. This gives a compute-matched baseline to compare to SFTL;DR.

Interestingly, simply training on (task, insight) tuples in SFTL;DR, without ever seeing the insight "in action" in a rollout, like above almost reaches SFT and RLTL;DR performance on full rollouts. This might be the most striking result: The model is able to internalize and generalize the knowledge given in an entirely different format from how it will have to put it into code at evaluation time. We discuss this finding in the light of recent findings on the surprising smoothness of training of large language models in Section 7.1 .

A final remark is that training only on (task, one-sentence insight) tuples also reduces the train compute. Its 4.6M forward (context) and 68k backwards (insight) tokens approach the performance of SFT on 100 rollouts with 9.5M forward and 217k backward tokens, and SFT on 3.5k rollouts with 400M forward and 6.6M backward tokens. We underline that this is purely a reduction in policy update compute. Full rollouts still need to be collected in order to generate the insights, which is the largest block of about ≈ \approx 1B tokens in all approaches.

## 6 What makes for good insights?

The insight we provide the model is short and procedural: typically, a single sentence (17 words on average), like “You need to mark the article as read instead of just viewing it” , see Appendix C . Crucially, the insight need not be too specific: as we show below, this level of abstraction is key for reaching good learning signal. We ablate multiple insight design choices on the SAPI-frontier split: how detailed it is, how it is generated, and how many iterations of insight are sequentially attached.

Insight content. We vary how detailed the insights are that are placed into the agent’s context. Instead of the single-sentence TL;DR, we provide a full diagnostic paragraph on why the attempt failed, optionally a summary of the attempt preceding the diagnostic paragraph, or both plus a proposed code correction. These artifacts are already generated in the main method as a byproduct when giving a TL;DR insight (we just see them as autoregressive generation aids and drop everything except the TL;DR insight), so they give the same hint, just in different level of detail.

Insight generator. First, we degrade the insights by turning off thinking during insight generation, or by not showing failed unit tests. Next, we improve insights by using a separate, more capable teacher model, GLM 5.2 ( GLM-5-Team, 2026 ) , in both thinking and non-thinking modes.

Amount of insight. We set the maximum number of insights in the context to 1, 2, 4, and 8, instead of the default 16. Note that we always use (and backpropagate) the most recent insight.