The Platonic Representation Hypothesis

arxiv:2405.07987

Original source · Paper JSON

Representation alignment · 2024-05-13

Cites (5) · Cited by (7)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• The Platonic Representation Hypothesis
• Abstract
• Keywords:
• 1 Introduction
• 2 Representations are converging
• Preliminaries
• 2.1 Different models, with different architectures and objectives, can have aligned representations
• 2.2 Alignment increases with scale and performance
• 2.3 Representations are converging across modalities
• 2.4 Models are increasingly aligning to brains
• 2.5 Does alignment predict downstream performance?
• 3 Why are representations converging?
• 3.1 Convergence via
• 3.2 Convergence via
• 3.3 Convergence via
• 4 What representation are we converging to?
• 4.1 An idealized world
• 4.2 A family of contrastive learners converge to a representation of ℙ ⁡ ( 𝐙 ) \mathbb{P}(\mathbf{Z})
• A study in color
• 5 What are the implications of convergence?
• Scaling is sufficient, but not necessarily efficient
• Training data can be shared across modalities
• Ease of translation and adaptation across modalities
• Scaling may reduce hallucination and bias
• 6 Counterexamples and limitations
• Different modalities may contain different information
• Not all representations are presently converging
• Sociological bias in producing AI models
• Special-purpose intelligences might not converge
• How do we measure alignment?
• Lots left to explain
• Acknowledgements
• Appendix A Mutual k k -Nearest Neighbor Alignment Metric
• The choice to use mutual nearest-neighbors
• Relationship between CKA and Mutual Nearest-Neighbors
• Appendix B Consistency across various metrics
• Vision-vision comparison
• Cross-modal comparison
• Appendix C Experiments on Evaluating Alignment and Convergence
• C.1 Vision-Vision Alignment and Representation Quality
• C.2 Cross-Modal Alignment
• Appendix D Color Cooccurrence Experiment
• Perceptual representation from CIELAB color space
• Three representations from cooccurrence in VISION and LANGUAGE
• Appendix E Caption Density Experiments
• Appendix F Analysis of Contrastive Learners
• F.1 Contrastive objectives learn pointwise mutual information
• F.2 Contrastive learners can represent K 𝖯𝖬𝖨 K_{\mathsf{PMI}} exactly under smoothness conditions
• Proposition F.1 .
• Proof.
• Remark F.2 .
• Instructions for reporting errors