Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

arxiv:2206.04615

Original source · Paper JSON

Retrieval, embeddings, benchmarks · 2022-06-09

Cites (2) · Cited by (13)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7
8. Part 8
9. Part 9

Contents

• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• Abstract
• 1 Introduction
• 1.1 Quantity has a quality all its own
• 1.2 Limitations of current benchmarks
• 1.3 Beyond the imitation game
• 2 What is in BIG-bench?
• 2.1 The BIG-bench API
• 2.2 BIG-bench Lite
• 2.3 Evaluation targets
• 2.3.1 Language models evaluated on BIG-bench
• BIG-G.
• BIG-G sparse.
• GPT.
• 2.3.2 Human rater baseline performance
• 2.4 Include a canary string in all documents about BIG-bench
• 3 Behavior of language models and human raters on BIG-bench
• 3.1 Aggregate performance improves with model size but is worse than human performance
• 3.2 Model predictions grow better calibrated with increased scale
• 3.3 Model classes behave similarly, with benefits from sparsity
• 3.4 Linearity and breakthroughness: categorizing scaling behaviors
• 3.4.1 Properties of tasks with high linearity or breakthroughness
• 3.4.2 Breakthrough behavior is sensitive to details of task specification
• Using smoother metrics.
• Manual decomposition into subtasks.
• 3.4.3 Even programmatic measures of model capability can be highly subjective
• 3.5 Even large language models are brittle
• 3.6 Social bias in large language models
• What is meant by social bias?
• Bias typically increases with scale in settings with broad or ambiguous contexts.
• Bias can decrease with scale in settings with narrow, unambiguous contexts.
• Bias can potentially be steered through appropriately chosen prompting.
• 3.7 Performance on non-English languages
• Performance on non-English tasks is worse than on English tasks.
• Low-resource language tasks are particularly challenging.
• 4 Behavior on selected tasks
• 4.1 Checkmate-in-one task
• 4.2 Periodic elements task
• Zero-shot behavior across scales:
• One-shot behavior across scales:
• 5 Additional related work
• 6 Discussion
• 6.1 Overall Findings
• 6.2 Conclusion
• Broader Impact Statement
• 7 Author contributions
• 7.1 Core contributors
• 7.2 Task authors, and task-specific citations
• Acknowledgments
• Appendix
• Appendix A Model details
• Appendix B Definitions of linearity and breakthroughness
• Appendix C Additional evaluation results
• C.1 Performance at different temperatures
• C.2 Number of tasks that pass a score threshold
• C.3 Social bias results per task
• C.4 Metric dependent performance and breakthroughness
• Appendix D Performance by keyword
• D.1 Checkmate in one successful solutions
• Appendix E Expert evaluation
• Appendix F Additional Tasks