RLTL;DR: Self-improvement by Internalizing Self-generated Feedback — part 1/6

Cleaned-text code points [0, 15920). Bibliography omitted here; source.md includes it.
# RLTL;DR: Self-improvement by Internalizing Self-generated Feedback

###### Abstract

The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards the successful ones. This becomes problematic in the realms of self-improvement, where tasks are so difficult that the agent has a low or even no chance of success, and where there are no teacher models or example solutions to distill from. In this paper, we introduce RLTL;DR. After each failed attempt, we show the policy the verifier outputs and let it write its own feedback, in the form of a single TL;DR insight. The next rollout is conditioned on all previous insights, and we sequentially sample rollouts until a solution is found. Moreover, we enable backpropagation on the in-context insights to internalize a direct task → \rightarrow insight mapping. On challenging tool-calling and coding datasets (filtered to Pass@128=0), standard GRPO training of a Qwen 3.5 9B Thinking policy stays flat at a Pass@1 of 0% to 1%. RLTL;DR breaks through this learning barrier, achieving a Pass@1 of 14–31% with insights in context during training and, crucially, 12–13% when no insight is in context at eval time. We identify that the key is the task → \rightarrow insight internalization. To study this further, we reduce our approach to SFTL;DR, training only on (task, insight) tuples, without showing or backpropagating on any rollouts. Training on only 4k of these tuples recovers almost the full performance of RLTL;DR and classical SFT on full rollouts. This demonstrates a promising compacted training paradigm of the form "on this sort of task, keep this sort of thing in mind", which we hope to inspire future research on.

## 1 Introduction

Reinforcement learning with verifiable rewards (RLVR, Lambert et al. , 2024 ) presents an LLM agent policy with a task and lets the policy attempt to solve it in multiple parallel rollouts. The rewards are then checked for correctness via a verifier (unit tests and final state checks). Successful rollouts receive a positive reward and unsuccessful ones a negative one, for example via GRPO ( Shao et al., 2024 ; Liu et al., 2025b ; Yu et al., 2026 ) , improving the agent’s performance over time.

This paradigm fails when tasks are so hard that the policy does not produce any successful rollouts in 128 attempts. Starved of learning signal, the training loop does not take off. Further, in self-improvement scenarios such as agentic coding, we assume that the agent is at the frontier; there is no stronger teacher model or golden example solution to learn from. The policy has to explore the problem step by step and internalize findings from its own attempts.

We propose two techniques that, in combination, allow breaking through the learning barrier: First, we improve the exploration by moving from parallel i.i.d. rollouts to sequential rollouts. After each attempt, the verifier code is run to check for success, and if the attempt failed, the policy itself is handed the previous attempt and the error message to produce feedback, in the form of a higher-level TL;DR insight like "Remember to paginate search results." . In the next rollout, the policy is conditioned on the task and the previous TL;DR learnings in context. We find that this sequential sampling lifts the exploration phase of RLVR out of the zone of no successful signals, finding at least one solution for Pass@k = = 14–59% of the tasks. However, at test time, when the policy has to provide a solution at the first attempt, without any insights in context, it stays close to its original performance.

We thus make a minimal second change to the update phase of the model: We activate the backpropagation mask on the self-generated insight tokens that are in the context ( ≈ \approx 17 tokens per insight). This trains a task → \rightarrow something to keep in mind mapping into the policy that, although never generated at test time since insights are inserted in the form of user messages, internalizes the findings and generalizes them to similar problems through sheer smoothness of backpropagation.

We find that this breaks through the learning barrier in agentic and coding benchmarks on tasks where the Qwen 3.5 9B Thinking ( Qwen Team, 2026 ) fails for 128 attempts. While GRPO stays flat at 0% to 1%, our RLTL;DR achieves 11–14% Pass@1 (at eval time; without any insights or sequential sampling). We ablate the training and find that the key ingredient is indeed the backpropagation on the task → \rightarrow insight mapping. Even if we turn off the GRPO loss (and thus all backpropagation on tokens from the rollout), the SFT loss on the insight tokens alone achieves almost full performance.

## 2 Related Works

##### The learning barrier of RLVR.

The standard RLVR recipe samples a group of i.i.d. rollouts per task, scores them with a verifier, and updates with a group-relative advantage ( Lambert et al., 2024 ; Shao et al., 2024 ; Yu et al., 2026 ; Liu et al., 2025b ) . This has a structural failure mode: when every rollout in a group receives the same reward, either all are correct or all are incorrect, the advantage vanishes and the gradient is zero (a phenomenon variously named advantage collapse, the learning cliff, or exploration inefficiency ( Xia et al., 2026 ; Agrawal et al., 2026 ; Agashe et al., 2026 ) . When all rollouts are correct, one can apply different loss functions or mitigations such as entropy control, pass@ k k objectives, or difficulty-matched curricula ( Mahrooghi et al., 2026 ; Chen et al., 2025 ) . But on the frontier splits we target, success rates are close to zero, deprived of any learnable signal.

##### Guiding exploration with privileged information.

A growing body of work manufactures at least one success by injecting information the policy will not have at test time. These approaches differ along three axes: the source of the guidance, its explicitness , and the transfer mechanism to ensure the policy still performs when no guidance is available at evaluation time. Sources range from gold solutions and stronger teacher models ( Zhang et al., 2025 ; Zhang et al., 2026b ; Agrawal et al., 2026 ) to the policy’s own failed attempts ( Hatamizadeh et al., 2026 ; Song et al., 2026 ; Szot et al., 2026 ) . Explicitness ranges from a verbatim prefix of the reference solution ( Agrawal et al., 2026 ; Zhang et al., 2025 ) to a single conceptual pointer ( Chen et al., 2026b ) . Transfer mechanisms range from none at all ( Hatamizadeh et al., 2026 ; Zhang et al., 2025 ) , i.e., trusting that improvements under the guided prompt carry over to the bare one, through mixing guided and unguided rollouts within the same group ( Chen et al., 2026b ; Agashe et al., 2026 ) , to importance-sampling corrections that make the gradient unbiased for the unguided objective ( Agrawal et al., 2026 ) . In this paper, we assume there is no external source of guidance except failed unit tests, i.e., the policy needs to find improvements itself. We further aim to simplify both the insight itself and the transfer mechanism as much as possible. RLTL;DR notes down short self-generated insights during training, and simply backpropagates them to internalize the task → \rightarrow insight mapping and generalize it to similar tasks.

##### Internalizing context into weights.

Training a model to behave as if a context was present when it is not is called context distillation ( Askell et al., 2021 ; Snell et al., 2022 ) , recently revisited on-policy ( Wang et al., 2026 ) . Closest to us are SDPO ( Hübotter et al., 2026 ) and RLTF ( Song et al., 2026 ) , which turn feedback into a self-distillation signal over the rollout. We do not distill the full rollout: we predict the ≈ \approx 17 insight tokens. This relies on the recent finding that models can apply knowledge in a different format than trained on ( Shrivastava et al., 2026 ; Cook et al., 2026 ; Lu et al., 2026a ; Nakkiran et al., 2026 ) , going from a short insight to generating a full rollout, enabling to internalize skills and experience ( Lu et al., 2026b ; Wu et al., 2026 ; Chen et al., 2026a ) .

## 3 Methods

### 3.1 RL and GRPO preliminaries

We focus on multi-step agentic tasks where a large language model (LLM) agent interacts with an environment to achieve a goal. We formalize this as a partially observable Markov decision process (POMDP) with a goal space 𝒢 \mathcal{G} , observation space 𝒪 \mathcal{O} , and an action space 𝒜 \mathcal{A} , all of which are in natural language, and a binary reward function R R . The LLM policy π \pi generates an action a t ∼ π ( ⋅ | h t ) a_{t}\sim\pi(\cdot|h_{t}) including a think trace and a code block, given the chat history h t = ( g , a 1 , o 1 , … , o t − 1 ) h_{t}=(g,a_{1},o_{1},\dots,o_{t-1}) that starts with the task goal g ∈ 𝒢 g\in\mathcal{G} , followed by agent messages a ∈ 𝒜 a\in\mathcal{A} , and the output that their code produces in o ∈ 𝒪 o\in\mathcal{O} . The episode ends at step T T when the agent emits an end-of-task action or reaches the maximum episode horizon of 50 actions. The code verifier adds a final observation o T o_{T} and a binary reward r r . We denote this full trajectory τ = ( h T , a T , o T , r ) \tau=(h_{T},a_{T},o_{T},r) . Our objective is to train the agent π θ \pi_{\theta} , parameterized by θ \theta , to optimize the expected outcome reward r r . In the environments we consider, the per-step observation o t o_{t} offers rich information about the agent’s interactions with the environment. For example, o t o_{t} shows search results that the agent printed in its code block a t a_{t} , or tracebacks if the code execution failed. The final observation o T o_{T} includes failed asserts and is only visible to the feedback generator below.

We start from a common reinforcement learning (RL) setup for LLM agents using Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) . For a task g ∈ 𝒢 g\in\mathcal{G} , GRPO samples K K trajectories { τ k } k = 1 K \{\tau_{k}\}_{k=1}^{K} in parallel and normalizes the rewards into advantages A ^ k = r k − ∑ i = 1 K r i \hat{A}_{k}=r_{k}-\sum_{i=1}^{K}r_{i} . The action likelihoods are off-policy corrected, since θ \theta has already evolved from its version θ old \theta_{\text{old}} that collected rollouts, multiplied with the advantages, and clipped with ϵ = 0.2 \epsilon=0.2 to give the GRPO loss: ℒ GRPO ( θ ) = 𝔼 { τ k ∼ π old } k = 1 K , t = 1 , … , T ( k ) [ min ( π θ  ( a t | h t ) π θ old  ( a t | h t ) A ^ k , clip ϵ ( π θ  ( a t | h t ) π θ old  ( a t | h t ) ) A ^ k ) ] \mathcal{L}_{\text{GRPO}}(\theta)=\mathbb{E}_{\{\tau_{k}\sim\pi_{\text{old}}\}_{k=1}^{K},t=1,\dotsc,T(k)}\left[\min\left(\frac{\pi_{\theta}(a_{t}|h_{t})}{\pi_{\theta_{\text{old}}}(a_{t}|h_{t})}\hat{A}_{k},\,\operatorname{clip}_{\epsilon}\left(\frac{\pi_{\theta}(a_{t}|h_{t})}{\pi_{\theta_{\text{old}}}(a_{t}|h_{t})}\right)\hat{A}_{k}\right)\right] (3.1) RL then iterates phases of sampling rollouts given the current policy on several tasks, and updating the policy with the collected rollouts and ℒ GRPO  ( θ ) \mathcal{L}_{\text{GRPO}}(\theta) . Since we work mostly on very hard tasks, we stabilize the training with some enhancements from literature, such as dropping division by standard deviation in the above advantages, and our own, which we detail in Section A.3 .

### 3.2 Sequential self-generated insights

We introduce RLTL;DR as a new RL training method for solving training tasks that are extremely challenging for the LLM agent, and where neither a stronger teacher agent nor example solutions are available. On such challenging problems, the LLM agent is unlikely to succeed through random sampling, causing repeated attempts to all receive zero outcome rewards and thus provide no learning signal for the RL training objective in Equation 3.1 ( Yue et al., 2025 ; Wu et al., 2025 ) .

The first component of RLTL;DR addresses this problem by modifying the GRPO sampling phase so that the agent attempts the same task multiple times in a row, conditioned on self-generated insights from previous attempts. Specifically, after attempting the problem, the agent can reflect on its attempt and use information from the environment observations and failed unit tests (in o T o_{T} ) to determine how to improve the subsequent attempt. We call this natural language assessment of what should be improved in the next attempt “insight”.

For a given task, the policy generates the first trajectory as usual, with a t ∼ π θ ( ⋅ | h t ) a_{t}\sim\pi_{\theta}(\cdot|h_{t}) . After it finishes the rollout τ 1 \tau_{1} , if it failed, it self-generates a short insight text f 1 ∼ π θ ( ⋅ | τ 1 ) f_{1}\sim\pi_{\theta}(\cdot|\tau_{1}) . The insight generation prompt asks the agent to think, summarize, analyze errors, and finally output the insight f i f_{i} as a single-sentence summary of what to improve on the next attempt ( Section A.1 ). We proceed with generating the next attempt on the task. Whenever at the k k -th attempt ≤ 50 % \leq 50\% of the attempts 1 , … , k − 1 1,\dotsc,k-1 are successful, we insert all insights collected so far. We add the insights { f i } i = 1 I \{f_{i}\}_{i=1}^{I} from the previous I ≤ k − 1 I\leq k-1 failed attempts as additional chat messages h ~ t = ( g , f 1 , … , f I , … ) \tilde{h}_{t}=(g,f_{1},\dotsc,f_{I},\dotsc) after the goal, generating the next attempt with them in context via a t ∼ π θ ( ⋅ | h ~ t ) a_{t}\sim\pi_{\theta}(\cdot|\tilde{h}_{t}) . This sequential generation continues for K K attempts. We use the 50 % 50\% boundary to insert insights only on tasks where the agent is struggling, to maintain a goldilocks zone of success rates ( Mahrooghi et al., 2026 ) . Section F.2 shows that RLTL;DR is robust to the choice of the heuristic.

RLTL;DR generates insight using the policy itself, so with the same (evolving) weights θ \theta . As we later demonstrate, this successive insight and retry mechanism enables the model to solve more challenging tasks than the base model alone, other prompting approaches, or exploration approaches. To compare fairly, we match GRPO’s and our number of attempts per task. We treat the K K successive attempts as a single GRPO group. While the rollouts are conditioned on different (or no) insights, we find no performance differences when splitting advantage groups ( Section F.2 ) and prefer simplicity.

In synchronous rollout collection, one could expect sequential sampling to take K × K\times longer than parallel GRPO sampling, plus the cost of generating the insight. But with asynchronous rollout collection with continuous batching and caching, at K = 8 K=8 we observe the sampling phase to be 4.5 × 4.5\times slower. Since update phases and other fixed costs stay equal, the overall walltime increases by 1.5 × 1.5\times . This is of course not important to begin with in very difficult settings where GRPO simply fails to learn. One could also increase the number of tasks that are rolled out in parallel during rollout collection by K × K\times to alleviate any throughput differences and ensure maximum GPU utilization. We do not do this in this paper in order to give GRPO and RLTL;DR the same amount of data per update phase, for benchmarking fairness.

### 3.3 Insight internalization

While training with sequential insights improves the policy’s ability to explore solutions during training, we find that learning to solve tasks with insights in the context does not directly transfer to solving tasks without insights in the context. This is significant because, at test time, the agent must succeed in a single attempt. The second component of RLTL;DR internalizes the self-generated insights so that the performance gains from sequential sampling with insights (during training) transfer to operation without insight (during evaluation).