{"id":"arxiv:2609.37633","title":"RLTL;DR: Self-improvement by Internalizing Self-generated Feedback","date":"2026-09-29","section":"Post-training","quality":null,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/","source_url":"https://arxiv.org/abs/2609.37633","text_available":true,"authors":["Michael Kirchhof","Eleonora Gualdoni","Andrew Szot","Khashayar Gatmiry"],"outgoing":[{"source":"arxiv:2609.37633","target":"arxiv:2402.03300","evidence":"id","id":"arxiv:2402.03300","title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/636fb871904bc06f742cfdea069796f9e9665f95bd58dc63260eaff5154ef985/"},{"source":"arxiv:2609.37633","target":"arxiv:2503.14476","evidence":"title","id":"arxiv:2503.14476","title":"DAPO: An Open-Source LLM Reinforcement Learning System at Scale","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/548e934b912ef84691ace63bf1cfd670e7b0257d9d6986dde926f569e7b2f8b7/"},{"source":"arxiv:2609.37633","target":"arxiv:2503.20783","evidence":"id","id":"arxiv:2503.20783","title":"Understanding R1-Zero-Like Training: A Critical Perspective","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b49a056f3ce8e269dd65e3b11605927f3c4d0ef074bc68e0b1d7f84e6c9c6c6a/"},{"source":"arxiv:2609.37633","target":"arxiv:2504.13837","evidence":"id","id":"arxiv:2504.13837","title":"Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b90f70269aeb02311d74ffaadf47e139683424a47118dc18619000ba72174d17/"},{"source":"arxiv:2609.37633","target":"arxiv:2601.03192","evidence":"id","id":"arxiv:2601.03192","title":"MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/4f5bbf3318b1fe409e0ef5dad02157180b844de366d194e82ce74bc997912eff/"},{"source":"arxiv:2609.37633","target":"arxiv:2601.20802","evidence":"id","id":"arxiv:2601.20802","title":"Reinforcement Learning via Self-Distillation","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/6073ec3d17b9a932a68e79ae078928f4635e40fc79cfcbc31709715d1f1e1ffa/"},{"source":"arxiv:2609.37633","target":"arxiv:2602.09000","evidence":"id","id":"arxiv:2602.09000","title":"iGRPO: Self-Feedback-Driven LLM Reasoning","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/1cd6a66823fe24e8e2d83e817177d3a1e8cda1d7d5b59a4397f8669bee6306ae/"}],"incoming":[],"text_parts":[{"start":0,"end":15920,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-1.html"},{"start":15920,"end":31330,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-2.html"},{"start":31330,"end":47314,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html"},{"start":47314,"end":63150,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-4.html"},{"start":63150,"end":79118,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html"},{"start":79118,"end":80082,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-6.html"}],"contents":[{"title":"RLTL;DR: Self-improvement by Internalizing Self-generated Feedback","offset":0,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-1.html#text"},{"title":"Abstract","offset":70,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-1.html#text"},{"title":"1 Introduction","offset":1768,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-1.html#text"},{"title":"2 Related Works","offset":4775,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-1.html#text"},{"title":"The learning barrier of RLVR.","offset":4795,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-1.html#text"},{"title":"Guiding exploration with privileged information.","offset":5730,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-1.html#text"},{"title":"Internalizing context into weights.","offset":7361,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-1.html#text"},{"title":"3 Methods","offset":8201,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-1.html#text"},{"title":"3.1 RL and GRPO preliminaries","offset":8215,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-1.html#text"},{"title":"3.2 Sequential self-generated insights","offset":11427,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-1.html#text"},{"title":"3.3 Insight internalization","offset":15329,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-1.html#text"},{"title":"4 RLTL;DR breaks through the learning barrier","offset":19235,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-2.html#text"},{"title":"4.1 Experiment setup","offset":19285,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-2.html#text"},{"title":"4.2 Results","offset":22167,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-2.html#text"},{"title":"5 Reducing to the secret sauce: From RLTL;DR to SFTL;DR","offset":24751,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-2.html#text"},{"title":"5.1 Experiment setup","offset":25138,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-2.html#text"},{"title":"Dataset.","offset":25164,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-2.html#text"},{"title":"RLTL;DR ablations.","offset":25542,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-2.html#text"},{"title":"Standard SFT.","offset":25901,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-2.html#text"},{"title":"SFTL;DR.","offset":26471,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-2.html#text"},{"title":"5.2 Reducing RLTL;DR to just SFTL;DR","offset":27245,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-2.html#text"},{"title":"6 What makes for good insights?","offset":29734,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-2.html#text"},{"title":"7 Discussion","offset":33967,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"7.1 The surprising learning only from (task, insight) tuples","offset":33984,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"7.2 Limits of (task, insight) learning","offset":35938,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"7.3 Source and quality of insight","offset":37691,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"7.4 Beyond insight databases","offset":38782,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"8 Conclusion and outlook","offset":39664,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"Appendix Contents","offset":41795,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"Appendix A Prompts and details of RLTL;DR","offset":41817,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"A.1 Insight generation","offset":41863,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"A.2 Insight conditioning","offset":43057,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"A.3 Loss details","offset":43837,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"A.3.1 Changes taken over from DAPO","offset":44237,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"A.3.2 Positive-ratio filtering","offset":45193,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"A.4 Train hyperparameters","offset":46291,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"A.5 Fixing Qwen’s tendency to overthink on hard Leetcode problems","offset":47243,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-3.html#text"},{"title":"Appendix B Reproducing the RLTL;DR Implementation","offset":48548,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-4.html#text"},{"title":"Appendix C Examples of generated insights","offset":50498,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-4.html#text"},{"title":"C.1 Example TL;DR insights","offset":50544,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-4.html#text"},{"title":"C.2 Examples including previous autoregressive fragments","offset":50732,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-4.html#text"},{"title":"Appendix D How to evaluate without confounders","offset":51685,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-4.html#text"},{"title":"D.1 Deconfounded RL train and eval metrics","offset":52072,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-4.html#text"},{"title":"D.2 Gauging insight strength","offset":57381,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-4.html#text"},{"title":"Appendix E Additional analyses of the main run","offset":61609,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-4.html#text"},{"title":"E.1 Distance to original policy","offset":61660,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-4.html#text"},{"title":"E.2 Performance on revisited tasks","offset":64872,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html#text"},{"title":"E.3 GRPO loss is not necessary but speeds up training","offset":65489,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html#text"},{"title":"E.4 Insight evolution over time","offset":66950,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html#text"},{"title":"Insight evolves with the policy.","offset":66987,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html#text"},{"title":"E.5 Insight reliance","offset":68106,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html#text"},{"title":"Insight reliance remains mild.","offset":68843,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html#text"},{"title":"E.6 Less difficult Synthetic API split","offset":69517,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html#text"},{"title":"E.7 Normal-difficulty dataset splits","offset":70502,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html#text"},{"title":"E.8 Exploration effectiveness of sequential insight conditioning","offset":72869,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html#text"},{"title":"E.9 Adding Internalization to RLTF-SD","offset":73635,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html#text"},{"title":"Appendix F Understanding and simplifying insight","offset":74172,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html#text"},{"title":"F.1 Training the insight generator versus internalizing insight directly","offset":74225,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html#text"},{"title":"F.2 Loss functions and GRPO advantage groups","offset":77173,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-5.html#text"},{"title":"Instructions for reporting errors","offset":79118,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/part-6.html#text"}],"text_code_points":80082,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c7a4c1dbb0493059883d9a01c1f89d4db3644209b909e1195c5afa4d3a320999/source.md"}