RLTL;DR: Self-improvement by Internalizing Self-generated Feedback

arxiv:2609.37633

Original source · Paper JSON

Post-training · 2026-09-29

Cites (7) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• RLTL;DR: Self-improvement by Internalizing Self-generated Feedback
• Abstract
• 1 Introduction
• 2 Related Works
• The learning barrier of RLVR.
• Guiding exploration with privileged information.
• Internalizing context into weights.
• 3 Methods
• 3.1 RL and GRPO preliminaries
• 3.2 Sequential self-generated insights
• 3.3 Insight internalization
• 4 RLTL;DR breaks through the learning barrier
• 4.1 Experiment setup
• 4.2 Results
• 5 Reducing to the secret sauce: From RLTL;DR to SFTL;DR
• 5.1 Experiment setup
• Dataset.
• RLTL;DR ablations.
• Standard SFT.
• SFTL;DR.
• 5.2 Reducing RLTL;DR to just SFTL;DR
• 6 What makes for good insights?
• 7 Discussion
• 7.1 The surprising learning only from (task, insight) tuples
• 7.2 Limits of (task, insight) learning
• 7.3 Source and quality of insight
• 7.4 Beyond insight databases
• 8 Conclusion and outlook
• Appendix Contents
• Appendix A Prompts and details of RLTL;DR
• A.1 Insight generation
• A.2 Insight conditioning
• A.3 Loss details
• A.3.1 Changes taken over from DAPO
• A.3.2 Positive-ratio filtering
• A.4 Train hyperparameters
• A.5 Fixing Qwen’s tendency to overthink on hard Leetcode problems
• Appendix B Reproducing the RLTL;DR Implementation
• Appendix C Examples of generated insights
• C.1 Example TL;DR insights
• C.2 Examples including previous autoregressive fragments
• Appendix D How to evaluate without confounders
• D.1 Deconfounded RL train and eval metrics
• D.2 Gauging insight strength
• Appendix E Additional analyses of the main run
• E.1 Distance to original policy
• E.2 Performance on revisited tasks
• E.3 GRPO loss is not necessary but speeds up training
• E.4 Insight evolution over time
• Insight evolves with the policy.
• E.5 Insight reliance
• Insight reliance remains mild.
• E.6 Less difficult Synthetic API split
• E.7 Normal-difficulty dataset splits
• E.8 Exploration effectiveness of sequential insight conditioning
• E.9 Adding Internalization to RLTF-SD
• Appendix F Understanding and simplifying insight
• F.1 Training the insight generator versus internalizing insight directly
• F.2 Loss functions and GRPO advantage groups
• Instructions for reporting errors