Are Emergent Abilities of Large Language Models a Mirage?

arxiv:2304.15004

Original source · Paper JSON

Reasoning and the "physics" of language models · 2023-04-28

Cites (2) · Cited by (6)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Are Emergent Abilities of Large Language Models a Mirage?
• Abstract
• 1 Introduction
• 2 Alternative Explanation for Emergent Abilities
• 3 Analyzing InstructGPT/GPT-3’s Emergent Arithmetic Abilities
• Prediction: Emergent Abilities Disappear With Different Metrics
• Prediction: Emergent Abilities Disappear With Better Statistics
• 4 Meta-Analysis of Claimed Emergent Abilities
• Prediction: Emergent Abilities Should Appear with Metrics, not Task-Model Families
• Prediction: Changing Metric Removes Emergent Abilities
• 5 Inducing Emergent Abilities in Networks on Vision Tasks
• Emergent Reconstruction of CIFAR100 Natural Images by Nonlinear Autoencoders
• Emergent Classification of Omniglot Characters by Autoregressive Transformers
• 6 Related Work
• 7 Discussion
• Appendix A Approximate Behavior of Metrics on Sequential Data
• A.1 Per-Token Error Probability is Resolution-Limited
• A.2 Token Edit Distance
• A.3 Accuracy
• A.4 ROUGE-L-Sum
• Appendix B Inducing Emergent Abilities in Networks on Vision Tasks
• B.1 Emergent Classification of MNIST Handwritten Digits by Convolutional Networks
• Instructions for reporting errors