gzip Predicts Data-dependent Scaling Laws

arxiv:2405.16684

Original source · Paper JSON

Data, training, optimization · 2024-05-26

Cites (1) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• gzip Predicts Data-dependent Scaling Laws
• Abstract
• 1 Introduction
• 2 Related Work
• 2.1 Scaling Laws
• 2.2 Syntax & Information Theory
• 3 Modulating Data Complexity via Syntactic Properties of a PCFG
• 3.1 gzip -compressibility Measures Syntactic Complexity
• 4 Are Scaling Laws Sensitive to Data Complexity?
• 4.1 Computing Data-sensitive Scaling Laws from gzip -compressibility
• 4.2 Eliminating Syntactic Parameters as a Confounder of Compressibility
• 5 Discussion
• 6 Limitations
• 7 Acknowledgements
• Appendix A Appendix
• Instructions for reporting errors