From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning

arxiv:2505.17117

Original source · Paper JSON

NeuroAI · 2025-05-21

Cites (1) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
• Abstract
• 1 The Enigma of Meaning in Large Language Models
• 2 Research Questions and Scope
• 3 Benchmarking Against Human Cognition
• 3.1 Human Baselines: Empirical Data from Seminal Cognitive Science
• 3.2 Large Language Models Under Study
• 4 A Framework for Comparing Compression and Meaning
• 4.1 Theoretical Foundations
• 4.2 The ℒ \mathcal{L} Objective: Quantifying the Trade-off
• 4.2.1 The Complexity Term: Measuring Compression
• 4.2.2 The Distortion Term: Measuring Semantic Coherence
• 4.3 Connecting Framework to Research Questions
• 5 An Empirical Investigation of Representational Strategies
• 5.1 [RQ1] The Big Picture: Alignment of Conceptual Categories
• 5.2 [RQ2] Zooming In: Fidelity to Fine-Grained Semantics
• 5.3 [RQ3] The Efficiency Angle: The Compression-Meaning Trade-off
• 5.4 Emergence During Training: How Divergent Strategies Develop
• 6 Discussion and Conclusion
• 7 Ethics statement
• 8 Reproducibility statement
• 9 Acknowledgments
• Appendix A Cognitive Intuition
• A.1 A Cognitive Intuition of the compression-meaning tradeoff
• A.2 Cognitively-Inspired Inductive Biases
• Appendix B Limitations
• Appendix C Dataset Access Details
• Appendix D LLM Details
• Appendix E Contextual Prompts and Pooling Strategies
• Pooling Strategies.
• Prompt Templates.
• Models.
• Appendix F Static vs. Contextual AMI Exploration
• Appendix G Multilingual Analysis
• Appendix H Training Dynamics
• Appendix I Datasets Polysemy
• Appendix J Tokenizer Analysis
• Model Clustering By Tokenizer Family
• Appendix K Additional Clustering Metrics
• Appendix L Mini-Controlled Experiment (Matched Training Data)
• Appendix M Detailed AMI Scores per Model and Dataset
• Appendix N Correlation between Human Typicality Judgments and LLM Internal Cluster Geometry
• Appendix O Typicality and Cosine Similarity [RQ2]
• Appendix P Theoretical Extreme Case Exploration for ℒ \mathcal{L}
• Appendix Q Compression Figures
• Appendix R Complexity-Distortion Ratio (on the importance of β \beta )
• Appendix S ℒ \mathcal{L} objective vs. downstream task performance
• Instructions for reporting errors