Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models — part 2/2

Cleaned-text code points [15228, 26571). Bibliography omitted here; source.md includes it.

Our experiments so far were conducted in the setting where there exists a sufficient abundance of data such that even after pruning with the desired selection rate there are enough data points to fill the desired token budget without requiring any data to be repeated. However, there are many training settings that do not fall under this data-abundant regime. Consequently, we evaluate how perplexity-based data pruning performs when the number of tokens is constrained, and pruning induces a greater number of repetitions of the data. For each dataset, we vary the available data such that training for a Chinchilla optimal number of tokens requires a different number of repetitions. Specifically, we investigate data budgets that require { 0.5,1,2,4,8 } repetitions to reach the Chinchilla optimal 3 3 3 Repeat=0.5 means that the available number of tokens is twice the training budget, i.e. the data-abundant setting . As each number of repeats refers to the total number of tokens available, for all pruning experiments the number of repetitions after pruning is actually greater by a factor of 1 r s \frac{1}{r_{s}} since we prune the available tokens according to r s r_{s} , the selection rate. Since all models use a selection rate of 0.5 0.5 , the models trained on the pruned data see the data for 2 × 2\times more repetitions.

We plot the average downstream performance as a function of the number of repetitions in Figure 2 . On both the Pile and Dolma, we find that training on perplexity pruned data yields an improvement for up to two repetitions. These results suggest that perplexity-based data pruning can still provide performance gains for some degree of data constraint. Furthermore, our results replicate the findings of Muennighoff et al. (2023) that more than four repetitions yields negligible gains. Specifically, the baseline model without pruning maintains commensurate performance for up to four repetitions. Similarly, models trained on perplexity-pruned data maintain commensurate performance for up to two repetitions through the base data, which corresponds to four repetitions after pruning. That training on repeated perplexity-pruned data leads to diminishing gains after four repetitions post-pruning suggests that the higher quality data resulting from pruning does not change the point for which repeating data yields diminishing improvements in performance.

### 3.6 Upstream Perplexity is not a Reliable Evaluation Metric for Data Pruning

As previous works have used the perplexity of the model on a test split of the pretraining dataset as an approximation to downstream performance, we wanted to explore how well such perplexity-based evaluations agree with downstream performance for data intervention techniques. Pruning performs an intervention on the dataset, making models trained on the pruned dataset biased estimators of the original data distribution. Therefore, it is unlikely that the performance on the original data distribution is a fair evaluation of model quality. We compare the test set perplexity and average downstream performance for 1 billion parameter models trained on the original and pruned version of the Pile and Dolma in Table 3 . For both the Pile and Dolma, training on perplexity pruned data significantly worsens perplexity on a test split of the pretraining data, while the average downstream performance is significantly improved. This result suggests that test set perplexity may not always be a sound metric for data pruning work and that researchers should instead directly evaluate on downstream benchmarks.

## 4 Understanding the Effects of Perplexity-Based Pruning

In this section, we investigate how data pruning works by exploring some of the properties of perplexity-based pruning.

### 4.1 How Are Reference Perplexities Distributed

In order to better understand how perplexity-based data pruning works, we investigate the distribution of the computed reference model perplexities for each dataset. For each dataset, we randomly sample 10% of the calculated perplexities and perform kernel density estimation to estimate the distribution of log perplexities for a given dataset. We repeat this procedure for the optimal pruned version of the dataset. We plot the resulting estimates of the log perplexity distribution in Figure 3 . We find that the log perplexity distribution for the Pile is multimodal and asymmetric, while for Dolma and it is unimodal and symmetric.

### 4.2 How Pruning Affects Domain Composition

We can also interpret the effect that perplexity-based data pruning has on a dataset by examining how pruning affects each domain’s proportion of the total dataset. We plot the pre and post-pruning domain compositions for the Pile and Dolma in Figure 4 . Interestingly, for all datasets pruning increases the proportion of data coming from web-scraped domains while decreasing the proportion of data coming from highly specific technical domains such as code or scientific papers. This trend is more pronounced in the Pile, where the proportions of Pile-CC and OpenWebText2 nearly double, while the proportions of domains such as Pubmed Central, ArXiv, and Github are all reduced by at least a factor of three. Future work should investigate how perplexity-based pruning affects a model’s performance on downstream tasks that are in the same category as the highly pruned domains.

## 5 Related Work

##### Classical methods for pruning text data.

In order to improve the quality of raw web scrapes, which often contain very noisy samples, pruning via quality filtering has become a common practice. Simple rules-based methods have been employed to prune datasets by filtering out low-quality samples according to some hand-crafted heuristic such as whether the text contains prohibited words, is predominantly English, etc. ( Bane et al., 2022 ; Raffel et al., 2020 ; Rae et al., 2022 ; Penedo et al., 2023 ) . N-gram perplexity-based methods, in which an n-gram model is first trained on a high quality, curated corpus and then used to score another corpus, have also been applied to filter text data ( Moore & Lewis, 2010 ; Axelrod, 2017 ; Gao, 2021 ; Laurençon et al., 2022 ; Muennighoff et al., 2023 ) . Although our method also uses perplexity to prune data, it does so in a very different manner. In n-gram perplexity pruning, perplexity is used to estimate whether new text is in distribution as compared to the currated text the n-gram was trained on, while in our model-based perplexity pruning, the reference model is trained on the same distribution of text and the perplexity is more akin to an estimate of the difficulty of an example. In this work, the datasets we leverage already have some basic rules-based pruning applied, and as such, the method we investigate is largely complementary to these existing techniques.

##### Neural network based methods for pruning text data.

Recently, there has been much interest in using neural networks to compute metrics that can be used to intelligently prune datasets. A common technique in this family of methods is using a model to sample high-quality data from large datasets based on the sample’s similarity to a curated high-quality corpus that serves as a target distribution ( Feng et al., 2022 ; Xie et al., 2023b ) . Xie et al. (2023a) also consider how to use a small reference model to prune pretraining data for a much larger model, by using a small reference model to learn the optimal weighting of domain proportions to maximize the "learnability" of the resulting dataset. Pruning based on the difficulty or loss of a sample has previously been explored for text data, but the majority of such work focuses on curating data for finetuning ( Swayamdipta et al., 2020 ; Attendu & Corbeil, 2023 ; Coleman et al., 2020 ; Mindermann et al., 2022 ; Mekala et al., 2024 ) . Marion et al. (2023) , however, investigate multiple model-based sample difficulty heuristics for pruning pretraining text datasets. Although we use the same method for pruning text pretraining datasets, our analysis differs substantially as we evaluate model quality based on downstream metrics and extend our analysis to multiple different dataset compositions which enables us to conclude that the reference model can be smaller than the final model.

##### Data pruning on vision tasks.

While data pruning is becoming more and more relevant with large amounts of text data, it has also been extensively applied in the vision domain ( Paul et al., 2021 ; Toneva et al., 2018 ; Park et al., 2023 ) . These works often prune data points based on their loss or gradients during training ( Killamsetty et al., 2021 ; Mirzasoleiman et al., 2020 ) . Model-based methods have also been leveraged for image data pruning ( Fang et al., 2024 ; Schuhmann et al., 2021 ) . Note that in the literature, data pruning is also sometimes referred to as coreset selection ( Guo et al., 2022 ) . More recently, Park et al. (2022) show that, somewhat surprisingly, active learning ( Castro & Nowak, 2008 ) based algorithms tend to outperform most data subset selection algorithms. In the context of contrastive learning, hard-negative mining has been effective as a data pruning method ( Kalantidis et al., 2020 ; Robinson et al., 2020 ; Zhang & Stratos, 2021 ) . Recently, Goyal et al. (2024) investigated scaling laws for training on pruned data in the context of vision models.

## 6 Conclusion

In this work, we conduct an empirical investigation of the impact that perplexity-based data pruning has on model performance. We demonstrate that small reference models can be used to prune the data of models with up to 30 × \times more parameters, leading to both significant downstream performance improvements and increased training efficiency. We then investigate perplexity-based data pruning in two non-standard settings: the over-trained and data-constrained regimes. We find that for both settings, training on perplexity pruned data can outperform training on unpruned data, demonstrating that perplexity-based data pruning is a widely applicable and extensible technique. We also investigate upstream metrics for evaluating data pruning techniques and provide an example where evaluating models based on their perplexity on the test split of the pretraining dataset does not align with evaluating based on downstream model performance. Additionally, we demonstrate that optimal pruning techniques can vary greatly for different dataset compositions. Although we do not present a predictive theory for how pruning parameters should be selected for different datasets, we demonstrate that the optimal pruning parameters for a 1 billion parameter model can successfully transfer to 3 billion parameter models, potentially suggesting that empirically determining the optimal pruning parameters can be done cheaply. Our work takes a key step towards establishing perplexity-based data pruning as a primary technique in the modern data researcher’s toolkit.

#### Acknowledgments

There are a few people who we would like to express our deepest gratitude for the assistance they provided. Sean Owen helped us with his encyclopedic knowledge of PySpark. Sam Havens and Daniel King both helped advise the early stages of this work. Brett Larsen provided feedback on the presentation of our results.