Transcendence: Generative Models Can Outperform The Experts That Train Them

arxiv:2406.11741

Original source · Paper JSON

Reasoning and the "physics" of language models · 2024-06-17

Cites (2) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Transcendence: Generative Models Can Outperform The Experts That Train Them
• Abstract
• 1 Introduction
• 2 Definition of Transcendence
• Definition 1 .
• Remark 1 .
• 3 Conditions for Transcendence
• 3.1 Low-Temperature Sampling is Necessary for Transcendence
• Proposition 1 .
• 3.2 Transcendence with Low-Temperature Sampling
• Proposition 2 .
• 3.3 Denoising a Single Expert
• Proposition 3 .
• 3.4 Transcendence from Multiple Experts
• Proposition 4 .
• 4 Experiments
• 4.1 Experimental Setup
• Training Details.
• Evaluation.
• 4.2 Experimental Results
• Main Result: Low-temperature sampling enables transcendence.
• Lowering temperature increases rewards in expectation on specific states, leading to transcendence over the full game.
• Dataset diversity is essential for transcendence.
• 4.3 Additional Settings
• 5 Related Work
• Chess and AI.
• Diversity beats Strength.
• Offline Reinforcement Learning.
• 6 Discussion and Future Work
• Limitations.
• Future Work.
• Broader Impact.
• Acknowledgements
• Appendix A Proofs
• Proof.
• Proof.
• Proof.
• Proposition 5 .
• Proof.
• Proof.
• Appendix B Additional Denoising Visualizations
• Appendix C Intuition of low temperature sampling inducing transcendence
• Appendix D Further Related Work
• D.1 Label Disagreement
• D.2 Offline Reinforcement Learning
• Appendix E Training Details
• Appendix F Compute Resources
• Appendix G Full t-SNE
• Instructions for reporting errors