Memorizing Transformers — part 3/3
Cleaned-text code points [31869, 41080). Bibliography omitted here; source.md includes it.
We want to thank Charles Staats for the many fruitful discussions and detailed comments, Henryk Michalewski for early version of of the memory implementation, Petros Maniatis for his help with our code datasets, Aitor Lewkowycz for his help with larger scale memorizing transformer experiments, Behnam Neyshabur for his comments on finetuning non-memory models, Imanol Schlag for his proofread and detailed comments, and Dennis Lee and Manzil Zaheer for discussions about large-scale attention and retrieval.
## Ethics
The ability to memorize large databases of facts could have potential ramifications for society, especially if those databases include sensitive personal information or copyrighted works. However, one advantage of using an external memory is that the memory can be easily cleared of all such information, as we do at the end of each document that we train on. The same is not true of differentiable model parameters, which is what most existing architectures use to store facts and information that they are trained on.
## Reproducibility
Details of our architecture and training hyperparameters are given in Section 4.2 . The datasets for C4 and PG-19 are publicly available. Our additional datasets, Github, Isabelle, and ArXiv Math are derived from publicly available data buckets, which we link in the main part of the paper. Subsection 4.1 include details on how we constructed the datasets from those datasets. We plan to release our code as open source.
## Appendix A Length of inputs
### A.1 Ablation studies
In the following section, we performed ablation studies to investigate the effects of various hyperparameters. Unless otherwise specified, we carried out these experiments with a memorizing transformer with context size 512, XL cache 512 with a memory size of 8192.
##### Multiple k k NN layers.
We experimented with using two k k NN layers, rather than just one. However, we did not see further benefits brought by more than multiple retrieval layers.
##### k k NN layer index
We experimented with adding the external memory to layer 3 3 , 6 6 , 9 9 and 12 12 in a 12-layer transformer, with results shown in Table 14 . We found that adding memory to the middle of the layer stack will obtain the best result, whereas adding memory to layers either too close to the input or to the output obtained less gains.
Layer index Perplexity 3 2.40 6 2.36 9 2.37 12 2.43
##### Number of neighbors
We studied the effects of the number of neighbors we retrieve from memory, with results shown in Table 15 . We found that even with 32 number of neighbors, we can already obtain a comparable results with 128 or 256 neighbors.
Number of neighbors Perplexity 32 2.38 128 2.37 256 2.37
##### Random seeds
We measured the statistical significant of the results reported. We did 3 runs with 3 random seeds for Transformer XL of size 512, and also a memorizing transformer with memory size 8192. We measured the standard deviation of perplexities after 500K steps of training, shown in Table 16 . We saw the standard deviation between different runs of the same experiment appears to be much smaller than the gap between different models.
Models Perplexity Transformer XL 2.67 ± 0.01 2.67\pm 0.01 Memorizing Transformer 2.37 ± 0.005 2.37\pm 0.005
## Appendix B What does the model retrieve from memory?
##### Retrieving citation names
On arXiv math, several examples are shown in Table 17 , which includes both the retrieved token and its surrounding context. We observe that many of the gains in cross-entropy loss took place when trying to predict the name of bibitems, citations, or references, by looking up the references and citations used previously in the paper. Such lookups usually span over the entire paper, which is much longer than 8192 tokens, providing a plausible explanation for the gain beyond memory size of 8192.
Query index Input Target Surrounding context Retrieved index Retrieved surrounding context 20389 Mon thus bibitem{ ComtetMonthusYor } 2208 Brownian motion \ \backslash cite{ ComtetMonthusYor } 16623 cha kra \ \backslash cite{ chakrabarti }. 4677 ∼ \sim 1.2 of \ \backslash cite{ chakrabarti } 14747 as d \ \backslash eqref{ asdfg } which 3365 begin{equation} \ \backslash n \ \backslash label{ asdfg .1}
##### Retrieving function names from the codebase
As with the arXiv papers, we also studied which tokens the model retrieved from memory. As might be expected, the model is often looking up the names of functions, and variables, as shown in Table 18 .
Query index Input Target Surrounding context Retrieved index Retrieved surrounding context 23837 Fo nte menu_play-> setarFonte 14607 menu_load-> setarFonte 23825 , 35 hscreen/2-50, 50, 200, 35 ); 14599 20, y+40, 200, 35 ) 14546 -> adi panel-> adicionaComponente 5205 panel-> adicionaComponente
### B.1 More Retrieving examples in formal theorem proving corpus
##### Example 1
• Input token index: 64604
• Input token: “_”
• Target token: “pair”
• Surrounding context: )) by (simp add: Fourier_sum_limit_pair [OF f, symmetric] Fourier’
• Name needs to be predicted: Fourier_sum_limit_pair
• Retrieved token: “Four”
• Retrieved token index: 64412
• Retrieved context: 2 * n. Fourier_coefficient f k * trigonometric_set k t)
• Definition of the name: Figure 19: Definition of Fourier_sum_limit_pair .
##### Example 2
• Input token index: 46175
• Input token: “tri”’
• Target token: “gon”
• Surrounding context: <le>n. a k * trigonometric_set k x)
• Name needs to be predicted: orthonormal_system_trigonometric_set
• Retrieved token: “gon”
• Retrieved token index: 35457
• Retrieved context: lemma orthonormal_system_trigonometric_set: \ \backslash n "orthonormal_system
• Definition of the name: Figure 20: Definition of orthonormal_system_trigonometric_set .
##### Example 3
• Input token index: 49760
• Input token: “sum”’
• Target token: “m”
• Surrounding context: nusing Fourier_series_square_summable [OF assms, of’
• Name needs to be predicted: Fourier_series_square_summable
• Retrieved token: “sum”
• Retrieved token index: 35457
• Retrieved context: lemma Fourier_series_square_summable \ \backslash n assumes:
• Definition of the name: Figure 21: Definition of Fourier_series_square_summable .
##### Example 4
• Input token index: 49697
• Input token: “_”’
• Target token: “system”
• Surrounding context: lemma Riemann_lebesgue_square_integrable: nassumes "orthonormal_system S w
• Name needs to be predicted: orthonormal_system
• Retrieved token: “system”
• Retrieved token index: 28052
• Retrieved context: definition orthonormal_system :: " \ \backslash ’a::euclidean’
• Definition of the name: Figure 22: Definition of orthonormal_system .
##### Example 5
• Input token index: 34817
• Input token: “.”’
• Target token: “b”
• Surrounding context: shows "integrable (lebesgue_on {a..b})
• Retrieved token 1: “.”
• Retrieved token index 1: 2416
• Retrieved context 1: lebesgue_on {a..b}) f i
• Retrieved token 2: “-”
• Retrieved token index 2: 2445
• Retrieved context 2: (lebesgue_on {a-c..b-c}) (
• Retrieved token 3: “-”
• Retreived token index 3: 6479
• Retrieved context 3: (lebesgue_on {-pi..pi}) (
##### Example 6
• Input token index: 49759
• Input token: “_”’
• Target token: “sum”
• Surrounding context: 0" \ \backslash n using Fourier_series_square_summable [OF assms
• Retrieved token 1: “set”
• Retrieved token index 1: 35044
• Retrieved context 1: definition trigonometric_set :: "nat \ \backslash <Rightarrow>
• Retrieved token 2: “ier”
• Retrieved token index 2: 47272
• Retrieved context 2: definition Fourier_coefficient \ \backslash nwhere
• Retrieved token 3: “ine”
• Retrieved token index 3: 18160
• Retrieved context 3: lemma Schwartz_inequality_strong: \ \backslash nassumes “f’
• Retrieved token 4: “system”
• Retrieved token index 4: 28052
• Retrieved context 4: definition orthonormal_system :: “ \ \backslash ’a::euclidean’
• Retrieved token 5: “<”
• Retrieved token index 5: 47241
• Retrieved context 5: subsection \ \backslash <open>Convergence wrt the L’
• Retrieved token 6: “n”
• Retrieved token index 6: 40835
• Retrieved context 6: \ \backslash n subsection \ \backslash <open>A bit of extra’
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .