Representation-Based Exploration for Language Models: From Test-Time to Post-Training

arxiv:2510.11686

Original source · Paper JSON

Post-training · 2025-10-13

Cites (5) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• Representation-Based Exploration for Language Models: From Test-Time to Post-Training
• Abstract
• 1 Introduction
• 1.1 Contributions
• Diversity-guided generation ( Section 4.2 )
• 2 Problem Setup: From Inference-Time to Post-Training
• Methodology and motivation
• 2.1 Inference-time selection
• 2.2 Reinforcement learning post-training
• 3 Representation-Based Exploration: Inference-Time and RL
• Representation-based exploration for inference-time selection
• Representation-based exploration for RL post-training
• Why representation-based elliptical bonuses?
• 4 Inference-Time Exploration: Experimental Results
• Datasets
• Models
• Algorithms
• 4.1 Benefits of representation-based exploration
• RF1 : RepExp improves verifier efficiency across models and tasks
• RF2 : The benefits of RepExp grow with model strength
• RF3 : RepExp provides more improvement for harder questions
• RF4 : RepExp improves verifier efficiency over standard generation modifications
• 4.2 Extension: Representation-based exploration at the token level
• Representation-based exploration for autoregressive generation
• 4.3 Application: Protein sequence generation
• 5 Exploration for RL Post-Training
• Tasks and models
• Baselines
• RF7 : RepExp responses look novel
• 6 Discussion
• Related work
• Final remarks
• Acknowledgements
• Reproducibility Statement
• Appendix A Additional Related Work
• Exploration at test time
• Exploration in RL post-training
• Representation-based diversity
• Adapting exploration techniques from deep reinforcement learning
• Theoretical analysis of language model exploration
• Appendix B Details for Inference-Time Experiments ( Section 4 )
• B.1 Details from Section 4.1
• Hyperparameters
• Preprocessing
• Datasets
• Experiment resources
• Estimating samples-to-correct
• Estimating pass@k
• B.2 Details from Section 4.2
• Hyperparameters
• Computational expense
• Dataset construction
• Estimating samples-to-correct
• Experiment resources
• Computing the inverse covariance
• Additional plots
• Appendix C Details for RL Post-Training Experiments ( Section 5 )
• Hyperparameters
• Algorithm details
• Dataset splits
• Modifications to unlikeliness baseline
• Checkpoint picking
• Evaluation
• Experiment resources
• Computational overhead
• Appendix D Beyond Sharpening: Full Results
• Instructions for reporting errors