How much do language models memorize? — part 3/4
Cleaned-text code points [31670, 47498). Bibliography omitted here; source.md includes it.
For each of our models trained on text, we use unused non-overlapping data from FineWeb to perform a standard loss-based membership inference ( Yeom et al., 2018 ; Sablayrolles et al., 2019 ) on each model and plot performance across dataset sizes ( 10 ). For a fixed model size, membership inference gets more difficult as the size of the data increases. When comparing membership inference to extraction (Figure 10 ), membership inference is strictly higher in every case; in some cases we can infer training dataset membership quite well (score of 0.97 0.97 ) with an extraction rate of 0 0 .
### 5.2 Scaling laws for Membership
In this section we develop a set of predictive models for memorization. Specifically, we predict the F1 score of a loss-based membership attack given token count, number of examples, and model parameter count. We then validate our predictions on models from 500 500 K to 1.5 1.5 B parameters.
#### 5.2.1 Functional forms
We observe that for a fixed model capacity, membership inference follows a roughly sigmoidal form with respect to dataset size. The intuitive explanation is that M.I. is easy for large models overfit to tiny datasets, so its score begin at 1; as dataset size increases, differentiating train from test data by loss becomes more and more difficult, eventually decaying toward 0.5.
We reuse the data collected in our text experiments (Section 4 ) to solve for constants c 1 , c 2 , c 3 c_{1},c_{2},c_{3} in the following equation: Membership F 1 ( θ , 𝒟 ) = 1 2 ( 1 + c 1 σ ( c 2 ( Capacity ( θ ) | 𝒟 | + c 3 ) ) CLOSE \text{Membership}_{F_{1}}(\theta,\mathcal{D})=\dfrac{1}{2}{(1+c_{1}\sigma(c_{2}(\dfrac{\text{Capacity}(\theta)}{|\mathcal{D}|}+c_{3}))} where σ ( x ) = 1 1 + e − x \sigma(x)=\frac{1}{1+e^{-x}} .
##### Limiting behavior.
We observe that as | 𝒟 | → ∞ |\mathcal{D}|\rightarrow\infty , performance of our membership inference attack decreases to 0.5 0.5 (essentially random performance). For a model trained on an infinite dataset, our law predicts both membership inference and extraction to be impossible.
##### Fitting.
We use a non-linear least squares solver to find optimal values for c 1 , c 2 , c 3 c_{1},c_{2},c_{3} . Solutions found are c 1 = 1.34 c_{1}=1.34 , c 2 = − 0.034 c_{2}=-0.034 , and − 33.14 -33.14 . We plot the scaling laws along with observed data in Figure 7 . Although the sigmoidal function is slightly simplistic (the points do not perfectly fit) our fit produces estimates within 1 − 2 % 1-2\% of observations.
#### 5.2.2 Validation on larger models
We note that all contemporary language models trained with a tokens-per-parameter ratio of 10 2 10^{2} or higher, which according to our laws would imply membership inference score of 0.5 0.5 – that is, within our formulation, statistically significant loss-based membership inference is not possible.
To validate our predictions, we train models with expected membership F 1 F1 scores of 0.55, 0.75, and 0.95. For model sizes we select GPT-2 small ( 125 125 M params) and GPT-2 XL ( 1.5 1.5 B params). Using our scaling law, we solve for the dataset size required to get the desired membership inference score for the given model size (see Table 2 for more information). We train models on the estimated dataset size and measure F1 score (shown as circles in Figure 7 ).
Our predictions are generally within 1.5 1.5 points of the true F1 score; the score is most inaccurate for estimated F1 of 0.75, which is the point where the sigmoid is steepest. In general, the accuracy of our results indicates that our empirical model of membership inference is relatively accurate and provides evidence for why membership inference attacks fail on models trained on extremely large datasets ( Das et al., 2024 ; Duan et al., 2024 ; Maini et al., 2024 ) .
## 6 Related Work
##### Language models and compression.
Shannon’s source coding theorem ( Shannon, 1948 ) first formalized the duality between prediction and compression. The connection between language modeling and compression was studied as far back as Shannon (1950) , which observed that more accurate models of English can compress text in fewer bits. Other works note the connection between Kolmogorov complexity ( Kolmogorov, 1965 ) and Shannon information in detail ( Grunwald & Vitanyi, 2004 ) . Delétang et al. (2024) investigate using modern transformer-based language models as compressors. We use compression as a tool to measure memorization in models.
##### Language model capacity.
Early research on single-layer perceptrons found that single-layer networks can store up to 2 bits-per-parameter ( Cover, 1965 ; Gardner, 1988 ; Baldi & Hornik, 1989 ) . ( Arpit et al., 2017 ) formalize the idea of effective capacity of a model and its training procedure; they also observe that both representation capacity and training time have a strong impact on empirical model capacity. Several other works measure language model capacity in the number of facts or random labels that can be memorized by a network such as an RNN ( Collins et al., 2017 ; Boo et al., 2019 ) or transformer ( Roberts et al., 2020 ; Heinzerling & Inui, 2021 ; Allen-Zhu & Li, 2024 ) , sometimes under quantization. A few research efforts ( Yun et al., 2019 ; Curth et al., 2023 ; Mahdavi et al., 2024 ; Kajitsuka & Sato, 2024 ) have developed theoretical estimates for the capacity of different model architectures, although none have yet scaled to multi-layer modern transformers. ( Shwartz-Ziv et al., 2024 ) also analyze the ‘capacity’ of neural networks on datasets of varying size. We are the first to measure a clear upper-bound in model capacity using per-sample entropy measurements.
##### Information regularization in learning theory.
Several works have explored the role of mutual information between the input and output of a learning algorithm ( Bassily et al., 2018 ; Haghifam et al., 2020 ; Steinke & Zakynthinou, 2020 ) . This concept closely relates to the notion of memorization based on Shannon information, discussed in Section 2.1 . Some of our findings also relate to the discovery of double descent in machine learning ( Belkin et al., 2019 ; Nakkiran et al., 2019 ) and language modeling ( Xia et al., 2023 ) , as well as general discussions of memorization and generalization in deep learning ( Zhang et al., 2017 ; Tänzer et al., 2022 ) .
##### Alternative definitions of memorization.
Unintended memorization is deeply related to the many other definitions of memorization proposed in the literature. We provide a detailed comparison in the following subsections.
##### Prior definitions of memorization.
Carlini et al. (2019) defined a string m m as memorized by a language model θ \theta if the second half of m m can be generated greedily when prompting the model with the first half. Following this, Nasr et al. (2023) introduced extractable memorization , where model θ \theta is said to memorize m m if an adversarial prompt p p can be found that generates m m . Mireshghallah et al. (2022) and Schwarzschild et al. (2024) refined this definition by restricting p p to a certain number of tokens, preventing it from containing the entire m m . However, even this definition has limitations: for example, generating the sequence “cat cat cat … cat” with the prompt ”repeat cat 1000 times” does not necessarily indicate memorization. Carlini et al. (2019) use perplexity or likelihood, one measure of the compressibility of a sequence, in an effort to distinguish highly memorized sequences from merely easy-to-compress ones. One additional definition of note is counterfactual memorization ( Zhang et al., 2023 ) , which measures the impact of a single datapoint on training; this can be seen as an instantiation of our definition where a different model of the same family is used as a reference model. Overall, all these works regarded memorization in terms that can be seen as forms of compression, although did not explicitly define it as such.
Finally, a concurrent work ( Cohen et al., 2024 ) proposes a theoretical definition for memorization also relying on Kolmogorov.
## 7 Conclusion
We propose a new definition of memorization that allows us to measure the exact number of bits a model knows about a dataset. We use our definition to measure the capacity of modern transformer language models and analyze how measurements such as extraction and F1 score scale with model and dataset size. We also propose a scaling law for membership inference and validate it on larger models. Our results help further practitioner understanding of how language models memorize and what they might (or might not) be memorizing across model and dataset scales.
## 8 Acknowledgements
Thanks to the many folks who helped us improve our paper, including Karen Ullrich, Niloofar Mireshghallah, Mark Ibrahim, Preetum Nakkiran, and Léon Bottou.
## Appendix A Appendix
### A.1 How reliable are our linear estimates of capacity?
Instead of scaling the number of examples in a dataset, we scale model sequence length to adjust the size of a dataset. We use the following measurement for expected memorization of a model:
mem ( X , L ( X ) ) ≈ min ( c a p a c i t y ( L ) , H ( X ) ) \text{mem}(X,L(X))\approx\min(capacity(L),H(X))
we substitute our previous estimate of α = 3.642 \alpha=3.642 and ensure to adjust the parameter count for increases due to resizing the model’s embedding matrices. We fix the number of training samples to 4096 4096 and train a model with 2 2 layers and a hidden size of 128 128 . Results are illustrated in Figure 12 and Table 4 . Our predictions of total memorization are accurate, with an average error rate of 1.7% while scaling S S and 1.8% when scaling V V .
### A.2 Additional memorization results
Our findings indicate that memorization of text data neatly plateaus near the model capacity just as in the synthetic data case. When the dataset size increases by a factor of N N , the model divides its memorization between datapoints by an equal amount; the sum of memorization is measured to be constant, presumably at the upper bound of the model’s capacity.
When the dataset is small enough for each model to fit – that is, below the capacity of the smallest model – we observe very similar performance between the models. For larger data sizes we notice an interesting trend: unintended memorization increases with dataset size for to a point, presumably as a model fills its capacity with the available information, and then decreases, as the model replaces sample-level information with more useful, generalizable knowledge. A given model generalizes the most (and memorizes the least information about any individual sample) when the dataset is maximally large.
### A.3 Comparison of distributions memorized
##### Distribution-level analysis.
Text sequences have very different properties than uniform synthetic bitstrings. We explore how two models of equal capacity spread their memorization across datapoints. We plot a histogram (Figure 15 ) of train and test compression rates of training data from both synthetic random bitstrings and text. Random training data follows a very normal distribution with a small amount of overlap between train and test compression rates. Text loss is lower on average but more spread out, with low loss on some training points and a long tail of higher losses. There is much more overlap between the train and test loss distributions, which explains why membership inference is more difficult for text data.
##### Which datapoints are most memorized?
Our distribution-level analysis indicates that unlike in the random-bitstring case, models trained on a large amount of text are able to memorize a small number of datapoints. Prior work has indicated that a large amount of this memorization can be due to duplicated training points ( Lee et al., 2022 ) but our dataset is fully deduplicated so this cannot be an explanation in our case.
To quantitatively evaluate the number of rare words per document, we measure the TF-IDF of each training document, plotted vs. unintended memorization in Figure 16 . We use the following equation for TF-IDF:
TF-IDF ( d , 𝒟 ) = 1 | d | ∑ w ∈ d log | D | t f ( w , 𝒟 ) \text{TF-IDF}(d;\mathcal{D})=\frac{1}{|d|}\sum_{w\in d}\log\frac{|D|}{tf(w,\mathcal{D})}
where t f ( d , 𝒟 ) tf(d,\mathcal{D}) indicates the total number of times word w w appears in dataset 𝒟 \mathcal{D} . Intuitively, a higher TF-IDF score for document d d indicates that d d contains more words that are rare in 𝒟 \mathcal{D} .
We clearly observe for samples with positive unintended memorization there is a strong correlation between trainset TF-IDF and memorization: examples with more rare words are more memorized. In particular, the sample with highest TF-IDF out of the whole training dataset (a sequence of Japanese words) has the third-highest measured memorization; even though this is just one out of 260,000 260,000 training samples, the model can regurgitate the entire sequence given just a single token ( 囚). Out of the top twenty memorized sequences, all but three contain sequences of tokens from other languages (Japanese, Chinese, and Hebrew).
Manual analysis (Table 5 ) indicates that the most memorized datapoints have extremely rare tokens, typically ones not found in English.
### A.4 Scaling law fit
Here we demonstrate the fit of our sigmoidal scaling law to experimental data. We show points in tokens-per-parameter vs. fit in Figure 17 . Although the sigmoidal function is slightly simplistic (the points do not perfectly fit the curve) our fit produces estimates within 1 − 2 % 1-2\% of observations.
### A.5 Proofs
In the section we provide the proofs missing from the main body.
### A.6 Proof of Proposition 1
Here we prove Proposition 1
###### Proof.
we have mem U ( X , Θ ^ , Θ ) \displaystyle\text{mem}_{U}(X,\hat{\Theta},\Theta) = I ( X ∣ Θ , Θ ^ ) \displaystyle=I(X\mid\Theta,\hat{\Theta}) = I ( ( X 1 ∣ Θ , … , X n ∣ Θ ) , Θ ^ ) . \displaystyle=I((X_{1}\mid\Theta,\dots,X_{n}\mid\Theta),\hat{\Theta}). And since the data is sampled i.i.d., all random variables in { R i = [ X i ∣ Θ ] } i ∈ [ n ] \{R_{i}=[X_{i}~\mid~\Theta]\}_{i\in[n]} are independent. 4 4 4 Note that X i X_{i} themselves are not independent because they are sampled by first sampling an underlying model Θ \Theta . However, they are conditionally independent once the underlying model Θ \Theta is given. So we have, I ( ( X 1 ∣ Θ , … , X n ∣ Θ ) , Θ ^ ) ≥ ∑ i ∈ [ n ] I ( X i ∣ Θ , Θ ^ ) I((X_{1}\mid\Theta,\dots,X_{n}\mid\Theta),\hat{\Theta})\geq\sum_{i\in[n]}I(X_{i}\mid\Theta,\hat{\Theta}) which implies mem U ( X , Θ ^ , Θ ) ≥ ∑ i ∈ [ n ] mem U ( X i , Θ ^ , Θ ) . \text{mem}_{U}(X,\hat{\Theta},\Theta)\geq\sum_{i\in[n]}{\text{mem}_{U}(X_{i},\hat{\Theta}},\Theta). On the other hand, we have
mem U ( X , Θ ^ , Θ ) \displaystyle\text{mem}_{U}(X,\hat{\Theta},\Theta) = I ( X ∣ Θ , Θ ^ ) \displaystyle=I(X\mid\Theta,\hat{\Theta}) = H ( Θ ^ − H ( Θ ^ ∣ ( X ∣ Θ ) ) CLOSE \displaystyle=H(\hat{\Theta}-H(\hat{\Theta}\mid(X\mid\Theta)) ≤ H ( Θ ^ ) \displaystyle\leq H(\hat{\Theta}) ∎
### A.7 Proof of Proposition 4
###### Proof.
We first state a Lemma about connection between algorithmic (kolmogorov) mutual information and mutual information.
###### Lemma 6 .
[Theorem 3.6 in Grunwald & Vitányi (2004) ] Assume ( X , Y ) (X,Y) be a pair of joint random variables. Let f f be the density function, f ( x , y ) = Pr [ ( X , Y ) = ( x , y ) ] . f(x,y)=\Pr[(X,Y)=(x,y)]. Then we have
I ( X , Y ) − H K ( f ) \displaystyle I(X,Y)-H_{K}(f) ≤ E ( x , y ) ∼ ( X , Y ) [ I K ( x , y ) ] \displaystyle\leq\E_{(x,y)\sim(X,Y)}[I_{K}(x,y)] ≤ I ( X , Y ) + 2 H K ( f ) . \displaystyle\leq I(X,Y)+2H_{K}(f).