How much do language models memorize? — part 1/4
Cleaned-text code points [0, 15924). Bibliography omitted here; source.md includes it.
# How much do language models memorize?
###### Abstract
We propose a new method for estimating how much a model “knows” about a datapoint and use it to measure the capacity of modern language models. We formally separate memorization into two components: unintended memorization , the information a model contains about a specific dataset, and generalization , the information a model contains about the true data-generation process. By eliminating generalization, we can compute the total memorization of a given model, which provides an estimate of model capacity: our measurements estimate that models in the GPT family have an approximate capacity of 3.6 bits-per-parameter . We train language models on datasets of increasing size and observe that models memorize until their capacity fills, at which point “grokking” begins, and unintended memorization decreases as models begin to generalize. We train hundreds of transformer language models ranging from 500 K 500K to 1.5 B 1.5B parameters and produce a series of scaling laws relating model capacity and data size to membership inference.
## 1 Introduction
For the past several years, modern language models have been trained on increasingly large amounts of data, while parameter counts stay stagnant in the billions. For example, one recent state-of-the-art model ( Dubey & et al, 2024 ) has 8 billion parameters (around 32 G B 32GB on disk) but is trained on 15 trillion tokens (around 7 T B 7TB on disk).
A long line of work ( Carlini et al., 2019 ; Mireshghallah et al., 2022 ; Nasr et al., 2023 ; Zhang et al., 2023 ; Carlini et al., 2023b ; Schwarzschild et al., 2024 ) questions whether such pretrained language models memorize their training data in a meaningful way. Most research approaches this problem either through the lens of extraction, aiming to recover full training datapoints from model weights, or membership inference, classifying whether a training point was present in the training data of a given model.
Studies of language model extraction argue that a datapoint is memorized if we can induce the model to generate it ( Carlini et al., 2023b ; Nasr et al., 2023 ; Schwarzschild et al., 2024 ) . We argue that such generation does not necessarily serve as a proof of memorization. Language models can be coerced to output almost any string ( Geiping et al., 2024 ) ; hence the fact that a model outputs something is not necessarily a sign of memorization. To address this issue, some researchers have suggested regularizing the input to the language model, such as by limiting its length ( Schwarzschild et al., 2024 ) or matching it to the prefix ( Carlini et al., 2023b ) preceding the memorized sentence. However, none of these constraints allow us to distinguish whether a model outputs a string due to memorization or good generalization. For example, a language model prompted to add two numbers can output the answer without having seen the equation before.
To address this issue, we propose a definition of memorization that quantifies the extent to which a model retains information about a specific datapoint. Our approach leverages the concept of compression rate in bits: a model is considered to have memorized an input if the input can be compressed into a shorter encoding when the model is available. This framework draws inspiration from Kolmogorov information theory ( Kolmogorov, 1963 ) and Shannon information theory ( Shannon, 1948 ) , but remains practical by estimating information content using model likelihoods. We address the fundamental challenge of distinguishing memorization from generalization ( Prashanth et al., 2024 ) by decomposing memorization into two distinct components: unintended memorization , which captures the information the model retains about a specific dataset, and generalization , which represents the knowledge the model acquires about the underlying data-generating process. This separation is similar to the approach in ( Brown et al., 2021 ) , which defines memorization using conditional mutual information between the dataset and the trained model, conditioned on the true concept. However, our notion differs in that it enables this separation at the instance level using algorithmic definitions of information.
To understand our new quantities, we measure unintended memorization and generalization by training language models of varying capacity on datasets of different sizes. We first eliminate the question of generalization entirely by training on a dataset of random uniformly-sampled bitstrings. In this setting, we can exactly measure the amount of information contained about the data inside the model. This gives us a principled way to measure language model capacity when trained on uniform datasets of exact known information content. We find that GPT-style transformers can store between 3.5 and 4 bits of information in each model parameter, depending on model architecture and precision.
We then repeat our experiments with real text, where generalization is possible and even beneficial for learning. On real text, language models memorize up to a certain capacity, at which point they substitute unintended memorization for generalization, and begin to learn general, reusable patterns as opposed to sample-level specifics. Our framework shows that double descent phenomenon begins to occur at this point, when the data size exceeds the model capacity in bits.
Finally, we use our results to predict a scaling law for membership inference performance based on model capacity and dataset size. We show that membership inference follows a clean relationship based on model capacity and dataset size: bigger models can memorize more samples, and making datasets bigger makes membership inference harder. Our scaling laws extrapolate to larger models, and predict most modern language models are trained on too much data to do reliable membership inference on the average datapoint.
## 2 Memorization, intended and unintended
When a model θ = L ( x ) \theta=L(x) is trained using a training algorithm L L and a dataset x ∼ X x\sim X , some information is transferred from the sample x x to the model θ \theta . A key question in the memorization literature is determining how much of this stored information is intended versus unintended. In this work, we aim to provide a rigorous definition of memorization that satisfies certain properties:
1. Separation from generalization. Our notion of unintended memorization must be distinct from intended memorization, which we refer to as generalization. For example, consider a language model trained on the sample: Q: What is 2 100 2^{100} ? A: 1267650600228229401496703205376. When assessing how much of this training sample is memorized, we must account for the fact that performing simple math operations is expected from a language model.
2. Sample-level memorization. We need to define memorization for realizations of random variables, not the random variables themselves. Specifically, we want to determine how much unintended memorization of a sample x x occurs in a model θ \theta .
3. Independence from training algorithm. Our definition should be independent of the training algorithm L L and only a function of the final model θ \theta and the sample x x . This is crucial for language models, where we often only have access to the final model and target sample.
Previous works have attempted to define memorization for machine learning models. We aim to provide precise definitions of memorization that meet our criteria, and offer ways to measure it. See Appendix B for a broader discussion on definitions of memorization.
### 2.1 Warm-up: A statistical view of memorization
Notation. In this section, we use capital letters (e.g. X X , Θ \Theta ) to refer to random variables and lowercase letters to refer to instances of a random variable (e.g. x ∼ X x\sim X and θ ∼ Θ \theta\sim\Theta ).
We rely on information theory, which has developed well-understood notions of information for random variables. For a random variable X X , we use H ( X ) H(X) , the entropy of X X , to define the amount of information present in X X . For two distinct random variables X , Y X,Y , we can define X | Y X\mid Y as the uncertainty left in X X after fixing Y Y . With these definitions, we can now measure mutual information between X X and Y Y by subtracting the leftover information from the total information: I ( X , Y ) = H ( X ) − H ( X ∣ Y ) I(X,Y)=H(X)-H(X\mid Y) .
Now suppose we have a machine learning pipeline. We have a prior Θ \Theta on the underlying model that captures our dataset distribution X X , and a learning algorithm L L that maps samples from X X to a trained model Θ ^ \hat{\Theta} . To understand how much information about X X is stored in Θ ^ \hat{\Theta} , we can use the notion of mutual information: mem ( X , Θ ^ ) \displaystyle\text{mem}(X,\hat{\Theta}) = I ( X , Θ ^ ) = H ( X ) − H ( X ∣ Θ ^ ) . \displaystyle=I(X,\hat{\Theta})=H(X)-H(X\mid\hat{\Theta}).
Note that this captures all the information about X X that is stored in Θ ^ \hat{\Theta} . As we discussed, we need our notion of memorization to account for generalization as well. So when measuring unintended memorization, we are only interested in the information that is present in X | Θ X\mid\Theta , which is the uncertainty left in X X after fixing Θ \Theta . Hence, we can define unintended memorization as mem U ( X , Θ ^ , Θ ) = I ( [ X ∣ Θ ] , Θ ^ ) = H ( X ∣ Θ ) − H ( X ∣ ( Θ , Θ ^ ) ) . \displaystyle\text{mem}_{U}(X,\hat{\Theta},\Theta)=I([X\mid\Theta],\hat{\Theta})=H(X\mid\Theta)-H(X\mid(\Theta,\hat{\Theta})).
and then the generalization (or intended memorization) must be mem I ( X , Θ ^ , Θ ) = mem ( X , Θ ^ ) − mem U ( X , Θ ^ , Θ ) = I ( X , Θ ^ ) − I ( [ X ∣ Θ ] , Θ ^ ) \displaystyle\text{mem}_{I}(X,\hat{\Theta},\Theta)=\text{mem}(X,\hat{\Theta})-\text{mem}_{U}(X,\hat{\Theta},\Theta)=I(X,\hat{\Theta})-I([X\mid\Theta],\hat{\Theta})
having defined our notions of intended and unintended memorization we turn our attention to practically measuring them. Let us first state a proposition that enables measurement of unintended memorization:
###### Proposition 1 (Super-additivity of Unintended Memorization) .
Assume X = ( X 1 , … , X n ) X=(X_{1},\dots,X_{n}) is a dataset of n n i.i.d. samples. We have ∑ i ∈ [ n ] mem U ( X i , Θ ^ , Θ ) ≤ mem U ( X , Θ ^ , Θ ) ≤ H ( Θ ^ ) . \sum_{i\in[n]}\text{mem}_{U}(X_{i},\hat{\Theta},\Theta)\leq\text{mem}_{U}(X,\hat{\Theta},\Theta)\leq H(\hat{\Theta}).
This proposition shows that to measure a lower bound on the unintended memorization on the dataset level, we can sum per-sample memorization. On the other hand, the entropy of the information content of the trained model itself servers as an upper bound on the unintended memorization. Another implication of this implies that unintended memorization should scale with the dataset size but cannot exceed the total capacity of the model.
We note that this statistical definition of memorization was first introduced by Brown et al. (2021) , where they theoretically looked at the role of unintended memorization in enabling successful learning for certain tasks. However, this preliminary definition is unsuitable for us as we aim to practically measure instance-level memorization. In particular, since we observe only a single trained model and a single input sample, we are unable to define probabilities or conditional probabilities over samples conditioned on the model.
### 2.2 Measuring unintended memorization with Kolmogorov Complexity
Our definitions of memorization and generalization so far are defined using an “entropy-based” notion of information. This means our definitions can only be used for random variables. This brings big challenges in measuring memorization. All our variables in the definition of memorization are singletons. We have a single underlying model θ \theta , we have a single dataset x = ( x 1 , … , x n ) x=(x_{1},\dots,x_{n}) and we have a single trained model θ ^ \hat{\theta} 1 1 1 Note the switch to lowercase variables because we are now working with instances, not random variables. . It is impossible to measure the entropy (let alone conditional entropy) of the underlying variables using a single sample.
To this end, we switch to another notion of information based on compression, then later we show how this notion closely approximates the notion of memorization defined above. Kolmogorov complexity defines the information content of a string x x , denoted as H K ( x ) H^{K}(x) , to be the length of shortest representation of x x in a given computational model. Similarly, we can define the leftover information x | θ x\mid\theta , to be the shortest representation of x x , when we have θ \theta available as a reference. And the information content of x | θ x\mid\theta , denoted by H K ( x ∣ θ ) H^{K}(x\mid\theta) , is the length of such description. Then, we can define mutual information in a similar fashion:
###### Definition 2 (Kolmogorov complexity) .
Let f f be an arbitrary computational model that takes a set of inputs and returns an output (e.g. universal Turing machine). The shortest description of x x with respect to computational model f f is defined as H K ( x ) = min f ( p ) = x | p | . H^{K}(x)=\min_{f(p)=x}|p|. Also, the Kolmogorov complexity of x x relative to another string θ \theta is defined as H K ( x ∣ θ ) = min f ( p , θ ) = x | p | . H^{K}(x\mid\theta)=\min_{f(p,\theta)=x}|p|. And we define the Kolmogorov mutual information between x x and θ \theta by I K ( x , θ ) = H K ( x ) − H K ( x ∣ θ ) . I^{K}(x,\theta)=H^{K}(x)-H^{K}(x\mid\theta). We assume inputs are bitstrings and | p | |p| is the bit length of the input.
###### Definition 3 (Kolmogorov memorization) .
Let θ \theta be a reference model that approximates the true distribution of data, and θ ^ \hat{\theta} be a model trained on a dataset x = ( x 1 , … , x n ) x=(x_{1},\dots,x_{n}) . For each x i x_{i} we define the memorization of x i x_{i} in θ ^ \hat{\theta} as mem K ( θ ^ , x ) = I K ( θ ^ , x ) . \text{mem}^{K}(\hat{\theta},x)=I^{K}(\hat{\theta},x). We also define intended and unintended variants of memorization: mem U K ( x , θ , θ ^ ) = H K ( x ∣ θ ) − H K ( x ∣ ( θ , θ ^ ) ) , and mem I K ( x , θ , θ ^ ) = mem K ( x , θ ^ ) − mem U K ( x , θ , θ ^ ) . \text{mem}^{K}_{U}(x,\theta,\hat{\theta})=H^{K}(x\mid\theta)-H^{K}(x\mid(\theta,\hat{\theta}))\text{,~and~mem}^{K}_{I}(x,\theta,\hat{\theta})=\text{mem}^{K}(x,\hat{\theta})-\text{mem}^{K}_{U}(x,\theta,\hat{\theta}).
There are known connections between Kolmogorov complexity and Shannon Entropy ( Grunwald & Vitanyi, 2004 ) . These results point at the conceptual connection between the two notions and imply that E x ∼ X [ H K ( x ) ] ≈ H ( X ) \E_{x\sim X}[H^{K}(x)]\approx H(X) . Interestingly, this implies that our notion of Kolmogorov memorization closely approximates Shannon memorization.
###### Proposition 4 .
Let X = ( X 1 , … , X n ) X=(X_{1},\dots,X_{n}) be an i.i.d, dataset distribution parametrized by ground-truth model θ \theta . Let L L be a training algorithm mapping X X to Θ ^ \hat{\Theta} . Assume H ( Θ ^ ) = ℓ H(\hat{\Theta})=\ell and H ( X i ) = ℓ ′ H(X_{i})=\ell^{\prime} 2 2 2 The trained model and each data sample can be presented using ℓ \ell and ℓ ′ \ell^{\prime} bits respectively. . Then we have | E x ∼ X θ ^ ∼ L ( x ) [ mem U K ( x i , θ ^ , θ ) ] ] − mem U ( X i , Θ ^ , θ ) | ≤ ϵ . \Big|\E_{\begin{subarray}{c}x\sim X\\ \hat{\theta}\sim L(x)\end{subarray}}\big[\text{mem}^{K}_{U}(x_{i},\hat{\theta},\theta)]\big]-\text{mem}_{U}(X_{i},\hat{\Theta},\theta)\Big|\leq\epsilon. for some constant ϵ \epsilon independent of ℓ , ℓ ′ \ell,\ell^{\prime} and n n .
### 2.3 Estimating Kolmogorov with compression