{"id":"arxiv:2510.11686","title":"Representation-Based Exploration for Language Models: From Test-Time to Post-Training","date":"2025-10-13","section":"Post-training","quality":null,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/","source_url":"https://arxiv.org/abs/2510.11686","text_available":true,"authors":["Jens Tuyls","Dylan J. Foster","Akshay Krishnamurthy","Jordan T. Ash"],"outgoing":[{"source":"arxiv:2510.11686","target":"arxiv:2402.03300","evidence":"id","id":"arxiv:2402.03300","title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/636fb871904bc06f742cfdea069796f9e9665f95bd58dc63260eaff5154ef985/"},{"source":"arxiv:2510.11686","target":"arxiv:2407.21783","evidence":"id","id":"arxiv:2407.21783","title":"The Llama 3 Herd of Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/de3cebb4c28bc5d41a843279f219dca14476284991ea5c7c0aeab7068a40e306/"},{"source":"arxiv:2510.11686","target":"arxiv:2503.14476","evidence":"id","id":"arxiv:2503.14476","title":"DAPO: An Open-Source LLM Reinforcement Learning System at Scale","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/548e934b912ef84691ace63bf1cfd670e7b0257d9d6986dde926f569e7b2f8b7/"},{"source":"arxiv:2510.11686","target":"arxiv:2504.13837","evidence":"id","id":"arxiv:2504.13837","title":"Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b90f70269aeb02311d74ffaadf47e139683424a47118dc18619000ba72174d17/"},{"source":"arxiv:2510.11686","target":"arxiv:2510.14901","evidence":"id","id":"arxiv:2510.14901","title":"Reasoning with Sampling: Your Base Model is Smarter Than You Think","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/46373e6f386387299bbade0a302219a611c3497c5e5b5970e3502f7a5f189687/"}],"incoming":[],"text_parts":[{"start":0,"end":15794,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-1.html"},{"start":15794,"end":31707,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html"},{"start":31707,"end":47462,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html"},{"start":47462,"end":63213,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html"},{"start":63213,"end":67093,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-5.html"}],"contents":[{"title":"Representation-Based Exploration for Language Models: From Test-Time to Post-Training","offset":644,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-1.html#text"},{"title":"Abstract","offset":733,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-1.html#text"},{"title":"1 Introduction","offset":2382,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-1.html#text"},{"title":"1.1 Contributions","offset":5086,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-1.html#text"},{"title":"Diversity-guided generation ( Section 4.2 )","offset":7889,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-1.html#text"},{"title":"2 Problem Setup: From Inference-Time to Post-Training","offset":8241,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-1.html#text"},{"title":"Methodology and motivation","offset":8824,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-1.html#text"},{"title":"2.1 Inference-time selection","offset":10496,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-1.html#text"},{"title":"2.2 Reinforcement learning post-training","offset":13498,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-1.html#text"},{"title":"3 Representation-Based Exploration: Inference-Time and RL","offset":14761,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-1.html#text"},{"title":"Representation-based exploration for inference-time selection","offset":19111,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"Representation-based exploration for RL post-training","offset":21110,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"Why representation-based elliptical bonuses?","offset":22464,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"4 Inference-Time Exploration: Experimental Results","offset":23260,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"Datasets","offset":23610,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"Models","offset":24120,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"Algorithms","offset":24546,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"4.1 Benefits of representation-based exploration","offset":25203,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"RF1 : RepExp improves verifier efficiency across models and tasks","offset":25358,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"RF2 : The benefits of RepExp grow with model strength","offset":26113,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"RF3 : RepExp provides more improvement for harder questions","offset":26958,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"RF4 : RepExp improves verifier efficiency over standard generation modifications","offset":27818,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"4.2 Extension: Representation-based exploration at the token level","offset":28715,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"Representation-based exploration for autoregressive generation","offset":29299,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"4.3 Application: Protein sequence generation","offset":31657,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-2.html#text"},{"title":"5 Exploration for RL Post-Training","offset":33495,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"Tasks and models","offset":33678,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"Baselines","offset":34040,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"RF7 : RepExp responses look novel","offset":36185,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"6 Discussion","offset":37228,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"Related work","offset":37245,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"Final remarks","offset":38959,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"Acknowledgements","offset":40305,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"Reproducibility Statement","offset":40567,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"Appendix A Additional Related Work","offset":41024,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"Exploration at test time","offset":41063,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"Exploration in RL post-training","offset":42243,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"Representation-based diversity","offset":46784,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"Adapting exploration techniques from deep reinforcement learning","offset":47391,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-3.html#text"},{"title":"Theoretical analysis of language model exploration","offset":48253,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Appendix B Details for Inference-Time Experiments ( Section 4 )","offset":49375,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"B.1 Details from Section 4.1","offset":49443,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Hyperparameters","offset":49477,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Preprocessing","offset":49665,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Datasets","offset":49943,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Experiment resources","offset":52153,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Estimating samples-to-correct","offset":52369,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Estimating pass@k","offset":52977,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"B.2 Details from Section 4.2","offset":53500,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Hyperparameters","offset":53534,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Computational expense","offset":54096,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Dataset construction","offset":54584,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Estimating samples-to-correct","offset":55229,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Experiment resources","offset":55771,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Computing the inverse covariance","offset":56175,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Additional plots","offset":61284,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Appendix C Details for RL Post-Training Experiments ( Section 5 )","offset":61495,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Hyperparameters","offset":61565,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Algorithm details","offset":62128,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Dataset splits","offset":62570,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Modifications to unlikeliness baseline","offset":63168,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-4.html#text"},{"title":"Checkpoint picking","offset":64356,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-5.html#text"},{"title":"Evaluation","offset":64569,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-5.html#text"},{"title":"Experiment resources","offset":65047,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-5.html#text"},{"title":"Computational overhead","offset":65170,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-5.html#text"},{"title":"Appendix D Beyond Sharpening: Full Results","offset":65888,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-5.html#text"},{"title":"Instructions for reporting errors","offset":66129,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/part-5.html#text"}],"text_code_points":67093,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/89b1f9c1bcd283b28d316cb6cddafdd9361be5cd510b2be2a1e761c20a0fd62e/source.md"}