Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models

arxiv:2405.20541

Original source · Paper JSON

Data, training, optimization · 2024-05-30

Cites (2) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
• Abstract
• 1 Introduction
• Contributions
• 2 Perplexity-Based Data Pruning
• 3 Experiments
• 3.1 Setup
• Models.
• Data.
• Training and hyperparameters.
• Evaluation.
• 3.2 Perplexity-Based Data Pruning Improves Downstream Performance
• 3.3 Perplexity-Based Data Pruning Improves Training Efficiency
• 3.4 Perplexity-Based Data Pruning for Over-Trained Models
• 3.5 Perplexity-Based Data Pruning for the Data Constrained Regime
• 3.6 Upstream Perplexity is not a Reliable Evaluation Metric for Data Pruning
• 4 Understanding the Effects of Perplexity-Based Pruning
• 4.1 How Are Reference Perplexities Distributed
• 4.2 How Pruning Affects Domain Composition
• 5 Related Work
• Classical methods for pruning text data.
• Neural network based methods for pruning text data.
• Data pruning on vision tasks.
• 6 Conclusion
• Acknowledgments