RLTL;DR: Self-improvement by Internalizing Self-generated Feedback — part 5/6

Cleaned-text code points [63150, 79118). Bibliography omitted here; source.md includes it.

Within the RL runs, the checkpoints obtained from the "RLTL;DR, λ = 0.01 \lambda=0.01 " and "RLTL;DR, λ = 0 \lambda=0 " runs should be viewed with caution, as the policy did not learn much and thus had less successful rollouts to learn from. However, "RLTL;DR" and "RLTL;DR, no GRPO loss, only ℒ SFT \mathcal{L}_{\text{SFT}} " are comparable since they reach a similar performance in Table 2 . Interestingly, just like ℒ SFT \mathcal{L}_{\text{SFT}} alone explained most of the performance of RLTL;DR, it also gives most of the update magnitude. In fact, the magnitude is slightly higher than RLTL;DR, since both are two independent RL runs and ℒ SFT \mathcal{L}_{\text{SFT}} -only performed slightly higher, collecting more successful rollouts. Still, it is interesting that ℒ SFT \mathcal{L}_{\text{SFT}} -only creates such big updates, although it backpropagates only on 839k tokens compared to the full RLTL;DR’s 12M. This shows that the insight internalization indeed is not just a naive next-token prediction but likely activates and updates larger semantically related parts of the network. Note that this cannot be explained by differences between bf16 and fp32 – inside the RL runs, ℒ SFT \mathcal{L}_{\text{SFT}} is applied by just changing the backprop masks, but the tensors and their precisions follow the mostly bf16 setup of the RL loop.

Inside the SFT runs, the image is straightforward: SFTL;DR produces a smaller and more concentrated update than SFT on full rollouts, but from Table 2 we also know that it trains on fewer tokens and improves performance less. This again points to the fact that, in general, SFT on only the insight tokens is structurally not too dissimilar from SFT on full rollouts.

### E.2 Performance on revisited tasks

Some tasks are sampled more than once during training, allowing us to compare a task’s first visit to its last one ( Figure 3 ). Without any insight in context , the optimized policy solves these tasks nearly twice as often as at the beginning of training ( 0.138 → 0.253 0.138\to 0.253 , n = 85 n=85 , p = 0.001 p=0.001 ). The improvement is about as large as the one the policy makes on tasks it never revisits ( + 12.4 +12.4 pp). The gain thus does not appear to be confined to rollouts with insight in context: what training yields is a general improvement in capability.

### E.3 GRPO loss is not necessary but speeds up training

The fact that RLTL;DR is almost matched by no-GRPO in Section 5 is conditioned on training both methods until convergence. The no-GRPO entry in Table 2 required approximately 25% more environment interactions than the standard 170k-step budget. Note that this is without additional data, just by increasing the number of epochs from 1 to 1.25.

To examine whether the additional signal that GRPO gives speeds up convergence, we sweep the GRPO-to-SFT loss weight ratio ( 1 / λ 1/\lambda ) across independent runs, using 3 random seeds per ratio value, and report Pass@1 (on rollouts without hints) throughout the mid-early updates 40 − 80 40-80 in Figure 4 .

Because gradient clipping fires on nearly every update, the total gradient norm remains approximately constant across all ratio values. The ratio 1 / λ 1/\lambda thus controls the direction of the update, not its magnitude, specifying what fraction steers the policy via task-success signal (GRPO) versus insight internalization (hint-SFT), and we find RLTL;DR to be relatively robust to its choice. The no-GRPO baseline (ratio = 0 =0 ) is consistently outperformed by any run that includes a non-zero GRPO component. This suggests that allocating even a modest fraction of the gradient to the task-success signal accelerates training, though hint-SFT alone accumulates sufficient signal to match full RLTL;DR performance if given more time.

### E.4 Insight evolution over time

##### Insight evolves with the policy.

A task can be occasionally sampled a few times during training, which lets us ask whether the insight it receives evolved with the policy. We compare the insight written at a task’s first visit with the insight written at its last. Similarity is the Jaccard overlap of content words, averaged over sampled pairs of insight strings. Dissimilarity is 1 − 1- similarity. Results are shown in Figure 5 . The x-axis is the mean dissimilarity between two pieces of insight drawn from the same visit, which measures how much the verifier varies its wording about a task at one moment; the y-axis is the mean dissimilarity between insight drawn from the two different visits. Every task falls above the diagonal, in the top-left region of the plot, with within-visit dissimilarity averaging 0.19 0.19 against 0.82 0.82 across visits ( n = 92 n=92 revisited tasks). The insight a task receives thus varies substantially between visits, consistent with the policy adopting a different strategy as training progresses and the verifier consequently identifying a different type of failure.

### E.5 Insight reliance

Xia et al. (2026) introduced the notion of insight reliance , i.e., ρ ⁡ ( τ , q , h ) = log ⁡ π θ ( τ ∣ q + h ) − log ⁡ π θ ( τ ∣ q ) \rho(\tau;q,h)=\log\pi_{\theta}(\tau\mid q{+}h)-\log\pi_{\theta}(\tau\mid q) , averaged over correct trajectories with insights in context and normalized by trajectory length. This measure should reflect how much a successful trajectory depends on the insight still being present. In their paper, Xia et al. (2026) show that low reliance implies successes with insights are more likely to transfer once the insight is removed, and train the insight generator explicitly to keep it low. We measure the same quantity on our run to see where our insight falls on that scale.

##### Insight reliance remains mild.

Figure 6 shows reliance rising, but only mildly: from + 0.009 +0.009 averaged over the first half of training to + 0.034 +0.034 over the second. Some increase in reliance is expected: if the insight raises the success rate at all, reliance cannot be zero. However, reliance remains low in our setup (also considering the numbers reported by Xia et al. (2026) without their transfer-weighted reward). The insight is therefore used without being leaned on: it is present in the trajectories the policy learns from, but it is not so load-bearing that such trajectories become implausible once it is removed, consistent with Section E.2 .

### E.6 Less difficult Synthetic API split

Besides the Pass@128=0 split of SAPI, which contains 458 tasks, we also train on a split which contains 642 tasks. This split came from an earlier Pass@128=0 filtering, where we used not yet optimized sampling hyperparameters. It contains the 458 tasks of the final Pass@128=0 split, plus 184 additional tasks (that with the later improved sampling hyperparameters became solvable in at least 1 of 128 attempts). None of these tasks is trivial, the Pass@1 of Qwen 3.5 9B (with optimized hyperparameters) on the 642 task split is 4%. It thus gives a good testbed where learning signal is available, if sparse. We present results in this section and note that also the ablations in Sections 5 and 6 are based on this split.

Figure 7 shows that, despite learning signal being present, GRPO and SGE still cannot learn and stay at the original Pass@1 of 4% of the Qwen 3.5 9B model. RLTL;DR and RLTF-SD both break through the learning barrier.

### E.7 Normal-difficulty dataset splits

On normal difficulty splits, where ≥ \geq 95-97% of tasks are solvable in less than 128 attempts, GRPO is able to achieve the same performance as RLTL;DR when sufficiently trained. We do not claim outperformance on such setups.

To better understand the learning dynamics, we plot the Avg@8 of the tasks during Synthetic API training in Figure 8 . Note that these include different amounts of insight in the different approaches, so they tell about training dynamics, not performance. As depicted in this figure, as training progresses, more tasks are pushed to higher solve rates by RLTL;DR and RLTF-SD while GRPO fails to present the same improvement. In particular, after 100k environment interactions, GRPO does not seem to push the tasks in the middle solve rates (Avg@8 ∈ ( 0 , 0.333 ) \in(0,0.333) and Avg@8 ∈ [ 0.333 , 0.666 ) \in[0.333,0.666) ) to higher levels while the number of tasks it always fails on (Avg@8=0) increases. Among the two insight-based methods, RLTL;DR presents a better dynamic as it consistently maintains a higher proportion of fully solved tasks (Avg@8=1) during the training compared to RLTF-SD.

Figure 9 shows the reward achieved by different algorithms during training on Synthetic API and Appworld datasets with normal difficulty. According to this figure, RLTF-SD and RLTL;DR perform similarly on Synthetic API dataset and both are consistently achieving higher rewards compared to GRPO during training. Specifically, after only 91K environment interactions, these two insight-based methods collect the same reward as GRPO collects in 200k interactions, yielding 54% efficiency that remains even when correcting for the ∼ 1.5 × \sim 1.5\times higher walltime.

The pure reward, however, includes rollouts with insights in the context. Figure 10 shows the Pass@1 on rollouts without insights in context. RLTL;DR and RLTF-SD perform similarly to GRPO on Synthetic API. On Appworld dataset however, RLTF-SD falls behind. On Leetcode, both approaches start like GRPO but then stagnate. We also test OOD performance by evaluating on the unseen Appworld test_challenge split. GRPO and RLTL;DR produce similarly strong models here. When trained on SAPI, RLTL;DR achieves 52% Pass@1 and GRPO 53%. When trained on Appworld-train, RLTL;DR achieves 72.1% and GRPO 72.2% Pass@1 on test-challenge.

### E.8 Exploration effectiveness of sequential insight conditioning

Figures 11 and 12 isolate the effectiveness of the sequential insights in finding solutions to very hard problems. They are measured before any training on the base Qwen 3.5 9B Thinking policy. The only thing that changes along the x-axis is how many insights populate the model’s context, while the y-axis measures Pass@k. We find that GRPO struggles to find valid solutions, while insight-conditioning elevates performance. The largest gains appear in the first 5 insights. Sequential insight rises faster and higher than just using the latest insight in every attempt. The advantages of conditioning on multiple hints diminish if the policy is trained on such hints, rather than kept frozen.

### E.9 Adding Internalization to RLTF-SD

Figure 13 shows that adding ℒ SFT \mathcal{L}_{\text{SFT}} to RLTF-SD, by simply flipping backpropagation masks, might further help performance. We see this as a promising direction for future work, especially since it is simple to implement. One has only to flip the backpropagation mask of the already existing context with insights in it, and, depending on the implementation of the Self-Distillation objective, make sure that the presign is correct so logits of these hints are maximized.

## Appendix F Understanding and simplifying insight

### F.1 Training the insight generator versus internalizing insight directly

RLTF ( Song et al., 2026 ) , already introduced as our RLTF-SD baseline ( Section 4.2 ), uses the same sequential RL framework as ours: it conditions each generation round on insight produced from the previous round’s rollout. But it also proposes a second variant, RLTF-FM, where it train the process of insight generation itself. That is, they put a loss on the insights f f in the context they were generated, i.e. π θ ( f | τ ) \pi_{\theta}(f|\tau) conditioned on the actual full rollout (and feedback-generation prompt format), instead of directly learning the mapping from task to insight via the ℒ SFT \mathcal{L}_{\text{SFT}} loss on π θ ( f | g ) \pi_{\theta}(f|g) like in this paper.

Training the insight generation capability itself is an established idea in the self-play literature, where teaching a model to produce more useful insight, or more relevant follow-up prompts to the original task, has proven effective. Dong and Ma (2025) train a theorem-proving LLM to act as both conjecturer and prover in self-play, rewarding the conjecturer for proposing problems at the edge of the prover’s current capability. Liu et al. (2025a) similarly train one model in two roles, a Challenger that mines a corpus to generate reasoning tasks and a Reasoner that solves them, with the Challenger’s curriculum adapting to the Reasoner’s current skill. We adapt two such approaches to our setting, both with the actor-side ℒ SFT \mathcal{L}_{\text{SFT}} insight loss disabled so we isolate the insight generator’s own training signal.

RLTF-FM. Inspired by the RLTF-FM method in RLTF ( Song et al., 2026 ) , we apply an SFT loss to the insight generator on every piece of insight it produces, regardless of whether the subsequent, insight-conditioned rollout succeeds or fails. This directly supervises the generator to reproduce its own past insight, independent of downstream outcome.

Self-play insight SFT. Aligned with self-play techniques in RL, we instead apply the SFT loss only to effective insight – insight that led the actor to solve the task in the subsequent round, conditioned on it. This steers the insight generator toward generating insight that is useful, rather than merely reproducible.

We find that training the insight generator is not able to substitute for the signal that ℒ SFT \mathcal{L}_{\text{SFT}} on the actor provides in correcting the actor’s representations: both variants land close to the GRPO baseline and well below RLTL;DR ( Table 8 ).

We further test whether insight-generator SFT still helps on top of RLTL;DR, i.e., adding it alongside (rather than instead of) the actor-side ℒ SFT \mathcal{L}_{\text{SFT}} . We do not see much gain from adding RLTF-FM this way, consistent with its lack of a standalone effect above. Adding Self-play insight SFT, however, gives a more noticeable gain over RLTL;DR alone ( Table 8 ).

### F.2 Loss functions and GRPO advantage groups

In order to make the approach as simple as possible, we have refrained from some technically correct changes to the loss functions in RLTL;DR. In Table 9 , we present ablations. They are trained on the 386 task subsplit of the frontier-difficult SAPI tasks, and evaluated on the remaining 256 tasks, as in Section 5 .

First, we ablate the decision rule of when to insert insights into the next attempts. By default we do this if the running average success rate in the current batch is ≤ 50 % \leq 50\% , to push the batch into a Goldilocks zone. We indeed find that on the very hard tasks, we benefit from more insights, and from backpropagating them with higher λ \lambda : a threshold of 0.5 increases performance over 0.33 and 0.1 would slightly decrease it. If we made our decision rule to let the first 10 rollouts be without insight, then decide for the upcoming (usually 10-11) rollouts to include insight if the first had ≤ 3 \leq 3 successes, it also improves over the other λ = 0.1 \lambda=0.1 runs. This is because net, this would result in more insights, which are generally helpful on very hard tasks.

Second, we test ablations on the GRPO loss, all compared to the 19.6 % 19.6\% run. While left unmodified in RLTL;DR for simplicity, one could debias it. We find that splitting the GRPO into two groups, one with the insight-conditioned rollouts and the other without, does not change performance. We leave further exploration of this to future work, since our primary goal is simplicity. Attempting to directly train on the rollouts generated with insights, removing the insights from context at update time (but correcting for this off-policy rollout collection via importance sampling) reduces performance. If this is the goal, we recommend a proper self-distillation loss. Last, adding the GRPO-style division by the std slightly reduces performance, validating our tuning.