How much do language models memorize? — part 4/4
Cleaned-text code points [47498, 54600). Bibliography omitted here; source.md includes it.
Now we use this lemma to prove the statement of the Proposition. Let f f be a the density function for the joint distribution ( X i ∣ θ , Θ ^ ) (X_{i}\mid\theta,\hat{\Theta}) . That is f i ( x i , θ ^ ) = Pr [ X i = x i ∣ θ and Θ ^ = θ ^ ] f_{i}(x_{i},\hat{\theta})=\Pr[X_{i}=x_{i}\mid\theta\text{~~and~~}\hat{\Theta}=\hat{\theta}] . Note that this function is independent of n n . By definition we have mem U ( X i , Θ ^ , θ ) = I ( X i ∣ θ , Θ ^ ) . \text{mem}_{U}(X_{i},\hat{\Theta},\theta)=I(X_{i}\mid\theta,\hat{\Theta}). Now using Lemma 6 we have I ( X i ∣ θ , Θ ^ ) − H K ( f ) \displaystyle I(X_{i}\mid\theta,\hat{\Theta})-H_{K}(f) ≤ E x i ∼ X i | θ [ I K ( x i , θ ^ ) ] \displaystyle\leq\E_{x_{i}\sim X_{i}\mid\theta}[I_{K}(x_{i},\hat{\theta})] ≤ I ( X i ∣ θ , Θ ^ ) + 2 H K ( f ) . \displaystyle\leq I(X_{i}\mid\theta,\hat{\Theta})+2H_{K}(f). and this concludes the statement of Proposition by setting ϵ = 2 H K ( f ) \epsilon=2H_{K}(f) ∎
### A.8 Limitations
Our efforts to measure language model memorization come from a line of recent research to discover whether models have analyzed certain texts, and if so, how much. However, our main experimental contributions relate to the practice of training and evaluating language models, including a new perspective on the phenomenon of grokking ( Nakkiran et al., 2019 ) and a new measurement of capacity. Our results are specific to the environment proposed and do not necessarily generalize to other datasets, architectures, or training setups.
## Appendix B Discussion of other notions of memorization
In this section we list multiple other notions of memorization and compare it with our definition. We specifically focus on why these notions do not satisfy all of our requirements. • Stability-based notions of memorization. There are notions of privacy and memorization that deal with “stability” of the training algorithm to small changes in the training set. Most notably, differential privacy Dwork (2006) considers the worst-cast drift of the model distribution when a single data point changes. Another notion of memorization in Feldman (2020) is based on the change of the model prediction on a point x x , when we add the labeled pair ( x , y ) (x,y) to the training set of a classification/regression model. Both of these notions are crucially relying on the learning algorithm and how it behaves. Moreover, the definition of differential privacy is not ideal for our case because it is a worst-case definition and cannot be applied at sample/model level. While the notion of memorization in Feldman (2020) does not have this particular issue, it suffers from the fact that it only applies to classification models and mostly deals with the memorization of the association between the label ( y y ) and input ( x x ), and not the memorization of x x itself. These issues make these notions not ideal for our case.
• Extraction-based memorization. There are multiple works in the literature ( Carlini et al., 2019 ; Mireshghallah et al., 2022 ; Nasr et al., 2023 ; Zhang et al., 2023 ; Carlini et al., 2023b ; Schwarzschild et al., 2024 ) that define memorization of samples in language models based on how easy it is to extract that sample. Specifically, when trying to understand the extent of memorization of a sample x x in a model θ \theta they measure some notion of complexity for the task of eliciting the model to output x x . Although these notions are great in that they only take a model θ \theta and a sample x x , they still do not account for generalization. Considering our running example of the following training sample: ”What is 2 100 2^{100} ? (A: 1,267,650,600,228 , 229 , 401 , 496 , 703 , 205 , 376 1,267,650,600,228,229,401,496,703,205,376 )”, this will be identified as highly memorized by almost all of the extraction based notions of memorization. Another issue with these definitions are that they are heavily dependent on the details of decoding algorithm. This is not ideal as we do not expect the memorization of a sample x x in a model θ \theta to depend on the detailed parameters we use to generate samples using θ \theta . The work of Schwarzschild et al. (2024) in this category is the closest to ours. This work which is based on prompt-optimization, optimizes a short prompt p p to make the model elicit x x , then it calls the sample x x memorized, if length of p p is less than x x . Although this definition is close to our definition in using compression, it still does not account for generalization of the model. Moreover, it focuses on a specific way of compression through prompting. We posit that compression through prompting is an inferior compression scheme and can often lead to compression rates greater than 1.
• Membership/attribute inference. Membership inference Shokri et al. (2017) and attribute inference attacks Jayaraman & Evans (2022) have been used for empirically measuring the privacy of machine learning algorithms. These notions which usually aim at approximating the stability notions of memorization are suffering from the same shortcomings. They rely heavily on the learning algorithm and the data distribution. Moreover, they fail at providing a sample level notion of memorization. For example, the obtained accuracy for membership inference attack is only meaningful in the population level. This is because various attack may have different true positives for membership, and the union of all these true positive across different attack may cover the entire training set, rendering it unusable as a sample level notion of memorization.
• Data copying in generative models. There are some interesting notions of memorization designed specifically for generative modeling where a generative model may output a certain portion of training samples ( Bhattacharjee et al., 2023 ; Carlini et al., 2023a ) . These notions are similar to extraction based definition of memorization but they are more lenient in that they only require extraction of part of the training data. However, they still suffer from the same challenges as of extraction based definitions.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .