gzip Predicts Data-dependent Scaling Laws — part 2/2
Cleaned-text code points [15913, 26027). Bibliography omitted here; source.md includes it.
L ( N , D ) = E + A N α + B D β L(N,D)=E+\dfrac{A}{N^{\alpha}}+\dfrac{B}{D^{\beta}} (1)
where N N is the model’s parameter count and D D is the training dataset’s token count. They claim that E E captures the “entropy of natural text” (their Sec. 3.3) and that scaling laws are “independent of dataset” (their App. C). However, when we fit this function on the training results for each of our PCFG datasets (leveraging the helpful open-source implementation of Muennighoff et al. [2023] ), we find considerably different laws for each dataset (Tab. 2 ).
The scaling law induces a compute-optimal frontier for parameter count derived by Kaplan et al. [2020] & Hoffmann et al. [2022] and simplifiable as:
N o p t ( C ) = ( α A β B ( C 6 ) β ) 1 α + β N_{opt}(C)=\left(\dfrac{\alpha A}{\beta B}\left(\dfrac{C}{6}\right)^{\beta}\right)^{\dfrac{1}{\alpha+\beta}} (2)
where C C is a compute budget in FLOPs. We plot this compute-optimal frontier for Chinchilla as well as our fitted law for each PCFG dataset in Fig. 3 . From a cursory glance, we can see that the frontier of our fitted law progressively becomes more data-preferent as data becomes harder to compress, crossing over Chinchilla’s 1-to-1 frontier at some point 0.23 < gzip -compressibility < 0.45 0.23<\texttt{gzip}\text{-compressibility}<0.45 .
In order to predict scaling law parameters from a dataset’s compressibility, let us fit a simple linear regression on the fitted scaling law parameters for each dataset. Recall from Sec. 3.1 that we compute the compressibility H H of a dataset 𝑫 \boldsymbol{D} by taking the average ratio of compressed bits to original bits for each element d d :
H ( 𝑫 ) = 1 ‖ 𝑫 ‖ ∑ d ∈ 𝑫 ‖ gzip ( d ) ‖ ‖ d ‖ H(\boldsymbol{D})=\dfrac{1}{||\boldsymbol{D}||}\sum_{d\in\boldsymbol{D}}\dfrac{||\texttt{gzip}(d)||}{||d||} (3)
So once we’ve fit lines to predict each parameter (E, A, B, α \alpha , β \beta ) from H H , we can re-define each parameter as a function of compressibility:
∀ x ∈ { E , A , B , α , β } : x ( H ) = m x H + n x \forall x\in\{E,A,B,\alpha,\beta\}:x(H)=m_{x}H+n_{x} (4)
where m x m_{x} and n x n_{x} are just our fitted linear regressions’ parameters. These fitted values (along with the regressions’ p p -values) are presented in Tab. 3 and the linear regressions are visualized in Fig. 4 . They’re all pretty much monotonically decreasing at different rates with an interesting α \alpha - β \beta intercept at H ≈ 0.27 H\approx 0.27 . It’s also interesting to note that E E , originally set as the constant ‘entropy of natural language’, is the only parameter that loosely increases with H H (though not significantly)—our proxy for complexity or entropy.
Now, we can reparameterize Eq. 1 as a function of compressibility H H :
L ( N , D , H ) = E ( H ) + A ( H ) N α ( H ) + B ( H ) D β ( H ) L(N,D,H)=E(H)+\dfrac{A(H)}{N^{\alpha(H)}}+\dfrac{B(H)}{D^{\beta(H)}} (5)
However, since our experiments are considerably smaller scale and primarily on PCFG data, we also present a form of our data-dependent scaling law as an adjustment of Chinchilla, where ε \varepsilon is the weightage of our adjustment for the gzip -compressibility of the training data (and prime ( ′ ) parameters are the Chinchilla constants).
L ( N , D , H ) = ( 1 − ε ) E ′ + ε E ( H ) + ( 1 − ε ) A ′ + ε A ( H ) N ( 1 − ε ) α ′ + ε α ( H ) + ( 1 − ε ) B ′ + ε B ( H ) D ( 1 − ε ) β ′ + ε β ( H ) L(N,D,H)=(1-\varepsilon)E^{\prime}+\varepsilon E(H)+\dfrac{(1-\varepsilon)A^{\prime}+\varepsilon A(H)}{N^{(1-\varepsilon)\alpha^{\prime}+\varepsilon\alpha(H)}}+\dfrac{(1-\varepsilon)B^{\prime}+\varepsilon B(H)}{D^{(1-\varepsilon)\beta^{\prime}+\varepsilon\beta(H)}} (6)
### 4.2 Eliminating Syntactic Parameters as a Confounder of Compressibility
Given just the above experiments, we have not addressed the possibility that our compressibility measure is confounded by some underlying syntactic property (e.g. vocab size). To address this concern, in Fig. 5 we provide results showing that when holding vocab size steady and changing other syntactic properties (Tab. 4 ), gzip -compressibility still predicts scaling law parameter shifts (with an even stronger correlation than in the increasing vocab size setting).
We also empirically show the contrapositive in Fig. 6 , demonstrating that when we widely vary syntactic properties (Tab. 5 ) but such that the datasets’ final gzip -compressibility are all the same, there is no significant shift in scaling law parameters.
Although the intersection behavior observed in Fig. 4 is not observed in the iso-vocab case (Fig. 5 ), the steeper negative slope of α \alpha v.s. β \beta (and A A v.s. B B ) implies the same phenomenon of increased data-preference as gzip -compressibility grows.
Thus we have shown that scaling laws are dependent on training data and gzip -compressibility is a strong predictor of how data complexity will impact scaling properties. We provide fitted coefficients (Tab. 3 ) for predicting scaling parameters as a function of H H ( gzip -compressibility), and offer an adjustment-based formulation (Eq. 6 ) of the data-dependent scaling law to make our small scale synthetic experiments more adaptable to training frontier models.
## 5 Discussion
We have shown that the Chinchilla scaling law [ Hoffmann et al., 2022 ] and similar laws that claim to be data-agnostic are in fact web-text-specific instances of a broader family of scaling laws sensitive to information-theoretic measures of training data. Not only do datasets that are difficult to compress (regardless of syntactic specifics) require more compute, they also require a different trade-off between model size & data.
Why does a dataset with a certain level of compressibility H H result in the specific scaling parameters that it does? And why does increased gzip -compressibility result in increased preference for data over parameter count? These questions warrant further exploration from a theoretical perspective—we hope to leverage connections between syntax & information theory to answer this question in future work.
On the empirical side, an especially promising avenue is in LM’s for code since code datasets have significantly lower gzip -compressibility than natural language. Therefore, we can expect the compute-optimal scaling law for code to have stronger parameter-preference than Chinchilla. We are currently running experiments at 1.92e19 FLOPs (some gzip -compressibility analysis in Fig. 8 and preliminary runs in Fig. 9 ) to compare performance based on Chinchilla’s allocation and our data-adjusted scaling law’s (Eq. 6 ) allocation. As highlighted in Sec. 1 , scaling up this finding could save $278,000 in H100 hours when training a single relatively small (6B parameter) code-generation model.
Beyond a deeper theoretical understanding of scaling laws & more optimal compute allocation for code-generation LMs, our findings also open possibilities for using compressibility as a metric in training data filtering, curriculum learning ordering, and fine-tuning data requirement estimation. Anecdotally, we have heard from researchers at frontier AI labs that compression algorithms are already used in production for filtering LLM training data.
The call-to-action here is not per se to use our data-dependent scaling law to decide how to spend your next million dollars of compute, but rather to begin investigating whether performance discrepancies between models trained on different datasets are explainable by simple information-theoretic measures (e.g. gzip -compressibility) of training data. If so (as experienced by Bi et al. [2024] ), it may be worth building systems to dynamically determine compute allocation for each training dataset, with the general rule of thumb that easy-to-compress datasets prefer parameters and hard-to-compress datasets prefer data.
## 6 Limitations
The work presented here is restricted to the synthetic PCFG setting and does not yet explicitly show generalization to predicting different scaling properties of real-world datasets. Our method of modulating training data complexity by varying syntactic properties of a PCFG, though naturalistic, is fairly specific and may still be prone to confounders. In addition, there are features of training data that affect scaling but appear to not be modeled well by gzip -compressibility. Alternative compression algorithms & information-theoretic measures of training data were not tested though they likely all broadly correlate with gzip -compressibility.
## 7 Acknowledgements
We are grateful to Alon Albalak, Rylan Schaeffer, Charlie Snell, Ethan Caballero, Martin Ziqiao Ma, Sankeerth Rao Karingula, Guillaume Lample, Vibhu Sapra, Aryaman Arora, Tristan Thrush, and several anonymous Twitter users for their thorough reviews & feedback on drafts of this paper.
This work was also sponsored by generous compute grants from Luke Piette (RunPod), Kye Gomez (Swarms), Vincent Weisser & Johannes Hagemann (Prime Intellect), and of course the rest of my team at Reworkd—Asim Shrestha & Adam Watkins.
## Appendix A Appendix
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .