Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws — part 2/8

Cleaned-text code points [15536, 31195). Bibliography omitted here; source.md includes it.
The complexity of a knowledge set is determined not only by the number of knowledge pieces but also by the length of the value string v v , the diversity of the vocabulary, and other factors. For instance, if the attribute a = a= “passport number,” then the value v v contains more bits of knowledge compared with a = a= “gender,” because the former has significantly higher diversity . If the attribute a = a= “birth date,” then the value v v could consist of 3 chunks : ( 10 , 2 , 1996 ) (10,2,1996) .

Considering these examples, we propose a set of hyperparameters that may influence the complexity of knowledge: 1. N N — the number of (distinct) names n n , denoted by 𝒩 \mathcal{N} .

2. K K — the number of attributes a a , with 𝒜 \mathcal{A} representing the set of attributes. For simplicity, we assume | 𝒜 | = K |\mathcal{A}|=K is fixed.

3. T T — the number of tokens T T , where every character in v v belongs to 𝒯 \mathcal{T} for some | 𝒯 | = T |\mathcal{T}|=T . For example, we can think of T T as “vocab size” in a tokenizer.

4. C C and L L — the number of chunks and the length of each chunk for the value: each value v ∈ ( 𝒯 L ) C v\in(\mathcal{T}^{L})^{C} can be expressed as v = ( v 1 , v 2 , ⋯ , v C ) v=(v_{1},v_{2},\cdots,v_{C}) , where v i ∈ 𝒯 L v_{i}\in\mathcal{T}^{L} .

5. D D — the diversity of chunks: for each piece of knowledge ( n , a , v ) (n,a,v) and i ∈ [ C ] i\in[C] , the chunk v i v_{i} belongs to 𝒟 a ⊂ 𝒯 L \mathcal{D}_{a}\subset\mathcal{T}^{L} , for some set with cardinality D = def | 𝒟 a | ≪ T L D\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}|\mathcal{D}_{a}|\ll T^{L} .

###### Remark 2.1 .

For notation simplicity, we have assumed that all chunks within an attribute a ∈ 𝒜 a\in\mathcal{A} share the same diversity set 𝒟 a \mathcal{D}_{a} , and all chunks are of equal length, etc. This enables us to more easily demonstrate the influence of each hyperparameter on a model’s capacity. In practice, different attributes may have different diversity sets or value lengths — e.g., 𝒟 passport \mathcal{D}_{\textrm{passport}} could be much larger than 𝒟 gender \mathcal{D}_{\textrm{gender}} . Our theoretical results do apply to these settings, albeit with more complex notation.

In our theoretical result, we introduce a dataset bioD  ( N , K , C , D , L , T ) \textsf{bioD}(N,K,C,D,L,T) defined as follows:

###### Definition 2.2 ( bioD data generation) .

Consider a fixed set of K K attributes, such as a set 𝒜 = { “ID 1”  …  “ID K ” } \mathcal{A}=\big\{\textrm{``ID 1''}\dots\textrm{``ID $K$''}\} , and a fixed set 𝒩 0 \mathcal{N}_{0} of candidate names (with N 0 = def | 𝒩 0 | ≫ N N_{0}\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}|\mathcal{N}_{0}|\gg N ). 1. Generate N N names uniformly at random (without replacement) from 𝒩 0 \mathcal{N}_{0} to form 𝒩 \mathcal{N}

2. For each attribute a ∈ 𝒜 a\in\mathcal{A} , generate D D distinct strings w 1 , a , ⋯ , w D , a ∈ 𝒯 L w_{1,a},\cdots,w_{D,a}\in\mathcal{T}^{L} uniformly at random (without replacement) to form the diversity set 𝒟 a \mathcal{D}_{a} .

3. For each name n ∈ 𝒩 n\in\mathcal{N} and attribute a ∈ 𝒜 a\in\mathcal{A} , generate value v ⋆  ( n , a ) = ( v 1 , v 2 , ⋯ , v C ) v^{\star}(n,a)=(v_{1},v_{2},\cdots,v_{C}) by sampling each v i ∈ 𝒟 a v_{i}\in\mathcal{D}_{a} uniformly at random.

Let 𝒵 = def { ( n , a , v ⋆ ( n , a ) } n ∈ 𝒩 , a ∈ 𝒜 \mathcal{Z}\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\big\{(n,a,v^{\star}(n,a)\big\}_{n\in\mathcal{N},a\in\mathcal{A}} be the knowledge set.

###### Proposition 2.3 (trivial, bit complexity upper bound) .

Given 𝒩 0 \mathcal{N}_{0} and 𝒜 \mathcal{A} and 𝒯 \mathcal{T} , to describe a knowledge set generated in Definition 2.2 , one needs at most the following number of bits: log 2 ⁡ ( | 𝒩 0 | N ) + N  K  C  log 2  D + K  log 2  ( T L D ) ≈ N  log 2  | 𝒩 0 | N + N  K  C  log 2  D + K  D  log 2  T L D . \displaystyle\log_{2}\binom{|\mathcal{N}_{0}|}{N}+NKC\log_{2}D+K\log_{2}\binom{T^{L}}{D}\approx N\log_{2}\frac{|\mathcal{N}_{0}|}{N}+NKC\log_{2}D+KD\log_{2}\frac{T^{L}}{D}\kern 5.0pt.

(The approximation is valid when | 𝒩 0 | ≫ N |\mathcal{N}_{0}|\gg N and T L ≫ D T^{L}\gg D .) We will present a bit complexity lower bound in Section 3 .

### 2.2 Knowledge (Empirical Setting)

We utilize both the synthetic bioD dataset, generated as per Definition 2.2 , and several human biography datasets to evaluate language model scaling laws.

Allen-Zhu and Li [3] introduced a synthetic biography dataset comprising N N individuals, each characterized by six attributes: birth date, birth city, university, major, employer, and working city. 9 9 9 All attributes, except for the working city (determined by the employer’s headquarters), are chosen uniformly and independently at random. There are N 0 = 400 × 400 × 1000 N_{0}=400\times 400\times 1000 possible person names, 12 × 28 × 200 12\times 28\times 200 birth dates, 200 200 birth cities, 300 300 universities, 100 100 majors, and 263 263 employers. Additionally, a random pronoun with 2 possibilities is chosen for each person. To translate these tuples into natural language, in their bioS dataset, each individual is described by six randomly selected English sentence templates corresponding to their attributes. We direct readers to their paper for more details but provide an example below: Anya Briar Forger was born on October 2, 1996 . She spent her early years in Princeton, NJ . She received mentorship and guidance from faculty members at Massachusetts Institute of Technology . She completed her education with a focus on Communications . She had a professional role at Meta Platforms . She was employed in Menlo Park, CA . (2.1)

In this paper, we explore three variations of such datasets: • bioS  ( N ) \textsf{bioS}(N) represents an online dataset for N N individuals, where each biography is generated with new randomness for the selection and ordering of six sentence templates on-the-fly .

• bioS simple  ( N ) \textsf{bioS}^{\textsf{simple}}(N) denotes a similar dataset, but here, each biography is generated once with a fixed random selection and ordering of the sentence templates.

• bioR  ( N ) \textsf{bioR}(N) refers to the same dataset, but with each biography written 40 times by LLaMA2 [ 33 ] to increase realism and diversity.

These datasets correspond to the bioS multi+permute , bioS single+permute , and bioR multi data types discussed in [ 3 ] , albeit with minor differences. While their study focused on N = 100  K N=100K , we expand our scope for bioS to consider N N up to 20  M 20M ; for bioR , we limit N N to 1  M 1M , which already yields a dataset size of 22GB.

As introduced in Section 1 , if each knowledge piece is seen 1000 times during training, we call this 1000 exposures. For bioS  ( N ) \textsf{bioS}(N) , 1000 exposures will unlikely include identical biography data because there are 50 sentence templates for each attribute and a total of 50 6 × 6 ! 50^{6}\times 6! possible biographies per person. For bioS simple  ( N ) \textsf{bioS}^{\textsf{simple}}(N) , 1000 exposures mean 1000 passes of the data. For bioR  ( N ) \textsf{bioR}(N) , 1000/100 exposures mean only 25/2.5 passes of the training data.

For the bioD dataset, we define 𝒩 0 \mathcal{N}_{0} to be identical to bioS , with | 𝒩 0 | = 400 × 400 × 1000 |\mathcal{N}_{0}|=400\times 400\times 1000 . We encapsulate a person’s attributes within a single paragraph, employing random sentence orderings and a consistent sentence template. For example: Anya Briar Forger’s ID 7 is v 7 , 1 , … , v 7 , C v_{7,1},\dots,v_{7,C} . Her ID 2 is v 2 , 1 , … , v 2 , C v_{2,1},\dots,v_{2,C} . […] Her ID 5 is v 5 , 1 , … , v 5 , C v_{5,1},\dots,v_{5,C} . In this paper, we primarily utilize bioS . To illustrate broader applicability and to better connect to theoretical bounds , we also present results for bioS simple \textsf{bioS}^{\textsf{simple}} , bioR , and bioD .

### 2.3 Models and Training

GPT2 was introduced in [ 26 ] . Due to its limitations from the absolute positional embedding [ 2 ] , we adopt its modern variant, rotary positional embedding [ 31 , 7 ] , which we still refer to as GPT2 for convenience. Additionally, we disable dropout, which has been shown to improve performance in language models [ 33 ] . We explore a wide range of model sizes while using a fixed dimension-per-head of 64. The notation GPT2- ℓ \ell - h h represents ℓ \ell layers, h h heads, and 64  h 64h dimensions; for example, GPT2-small corresponds to GPT2-12-12. The default GPT2Tokenizer is used, converting people’s names and most attributes into tokens of variable lengths. In examining the impact of model architectures on scaling laws in Section 7 , we will also use LLaMA/Mistral architectures [ 32 , 19 ] .

Training. We train language models from scratch (i.e., random initialization) using the specified datasets. Knowledge paragraphs about individuals are randomly concatenated, separated by <EOS> tokens, and then randomly segmented into 512-token windows. The standard autoregressive loss is employed for training. Unless specified otherwise, training utilizes the default AdamW optimizer and mixed-precision fp16. Learning rates and weight decays are moderately tuned (see appendix).

## 3 Bit Complexity Lower Bound

When assessing the knowledge stored in a model, we cannot simply rely on the average, word-by-word cross-entropy loss. For example, the phrase “received mentorship and guidance from faculty members” in ( 2.1 ) does not constitute useful knowledge. We should instead focus on the sum of the loss for exactly the knowledge tokens.

Consider a model F F with weight parameters W ∈ 𝒲 W\in\mathcal{W} . Assume F F is trained on a bioD  ( N , K , C , D , L , T ) \textsf{bioD}(N,K,C,D,L,T) dataset 𝒵 \mathcal{Z} as defined in Definition 2.2 using any optimizer; this process is represented as W = W ⁡ ( 𝒵 ) W=W(\mathcal{Z}) (the model’s weight is trained as a function of the training dataset 𝒵 \mathcal{Z} ). During the evaluation phase, we express F F through two functions: F ⊤  ( W , R ) F^{\top}(W,R) , which generates names, and F ⊥  ( W , n , a , R ) F^{\bot}(W,n,a,R) , which generates values given ( n , a ) (n,a) , where R R denotes the randomness used in generation. Let F 1 ⊥  ( W ⁡ ( 𝒵 ) , n , a , R ) F_{1}^{\bot}(W(\mathcal{Z}),n,a,R) represent the first chunk of F ⊥  ( W ⁡ ( 𝒵 ) , n , a , R ) F^{\bot}(W(\mathcal{Z}),n,a,R) . We evaluate F F by calculating the following three cross-entropy losses: 10 10 10 We use 𝔼 n \operatornamewithlimits{\mathbb{E}}_{n} or 𝔼 n , a \operatornamewithlimits{\mathbb{E}}_{n,a} to denote uniform random selection of n ∈ 𝒩 , a ∈ 𝒜 n\in\mathcal{N},a\in\mathcal{A} . 𝐥𝐨𝐬𝐬 n  a  m  e  ( 𝒵 ) \displaystyle\mathbf{loss}_{name}(\mathcal{Z}) = def 𝔼 n ∈ 𝒩 − log 𝐏𝐫 R [ F ⊤ ( W ( 𝒵 ) , R ) = n ] \displaystyle\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\operatornamewithlimits{\mathbb{E}}_{n\in\mathcal{N}}-\log\operatornamewithlimits{\mathbf{Pr}}_{R}\big[F^{\top}(W(\mathcal{Z}),R)=n\big] 𝐥𝐨𝐬𝐬 v  a  l  u  e  1  ( 𝒵 ) \displaystyle\mathbf{loss}_{value1}(\mathcal{Z}) = def 𝔼 n ∈ 𝒩 , a ∈ 𝒜 − log 𝐏𝐫 R [ F 1 ⊤ ( W ( 𝒵 ) , n , a , R ) = v 1 ⋆ ( n , a ) ] \displaystyle\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\operatornamewithlimits{\mathbb{E}}_{n\in\mathcal{N},a\in\mathcal{A}}-\log\operatornamewithlimits{\mathbf{Pr}}_{R}\big[F_{1}^{\top}(W(\mathcal{Z}),n,a,R)=v^{\star}_{1}(n,a)\big] 𝐥𝐨𝐬𝐬 v  a  l  u  e  ( 𝒵 ) \displaystyle\mathbf{loss}_{value}(\mathcal{Z}) = def 𝔼 n ∈ 𝒩 , a ∈ 𝒜 − log 𝐏𝐫 R [ F ⊥ ( W ( 𝒵 ) , n , a , R ) = v ⋆ ( n , a ) ] \displaystyle\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\operatornamewithlimits{\mathbb{E}}_{n\in\mathcal{N},a\in\mathcal{A}}-\log\operatornamewithlimits{\mathbf{Pr}}_{R}\big[F^{\bot}(W(\mathcal{Z}),n,a,R)=v^{\star}(n,a)\big]

###### Remark 3.1 .

For a language model, such quantities can be computed from its auto-regressive cross-entropy loss. For instance, when evaluating the model on the sentence “Anya Briar Forger’s ID 7 is v 7 , 1 , … , v 7 , C v_{7,1},\dots,v_{7,C} ,” summing up (not averaging!) the loss over the tokens in “Anya Briar Forger” yields exactly − log 𝐏𝐫 R [ F ⊤ ( W ( 𝒵 ) , R ) = n ] -\log\operatornamewithlimits{\mathbf{Pr}}_{R}\big[F^{\top}(W(\mathcal{Z}),R)=n\big] for n = “Anya Briar Forger” n=\text{``Anya Briar Forger''} ; summing up the loss over the token v 7 , 1 v_{7,1} results in − log 𝐏𝐫 R [ F 1 ⊤ ( W ( 𝒵 ) , n , a , R ) = v 7 , 1 ] -\log\operatornamewithlimits{\mathbf{Pr}}_{R}\big[F_{1}^{\top}(W(\mathcal{Z}),n,a,R)=v_{7,1}\big] for this n n and a = “ID 7” a=\text{``ID 7''} ; and summing up the loss over the entire sequence v 7 , 1 , … , v 7 , C v_{7,1},\dots,v_{7,C} gives − log ⁡ 𝐏𝐫 R [ F ⊤  ( W ⁡ ( 𝒵 ) , n , a , R ) = v 7 , 1 , … , v 7 , C ] -\log\operatornamewithlimits{\mathbf{Pr}}_{R}\big[F^{\top}(W(\mathcal{Z}),n,a,R)=v_{7,1},\dots,v_{7,C}\big] . This holds regardless of the tokenizer or value length.

###### Theorem 3.2 (bit complexity lower bound) .

Suppose N ≥ Ω ⁡ ( D  log ⁡ N ) N\geq\Omega(D\log N) . We have log 2 ⁡ | 𝒲 | \displaystyle\log_{2}|\mathcal{W}| ≥ 𝔼 𝒵 [ N  log 2  N 0 − N e 𝐥𝐨𝐬𝐬 n  a  m  e  ( 𝒵 ) + N  K  log 2  D C e 𝐥𝐨𝐬𝐬 v  a  l  u  e  ( 𝒵 ) + K  D  log 2  T L − D D  e ( 1 + o ⁡ ( 1 ) )  𝐥𝐨𝐬𝐬 v  a  l  u  e  1  ( 𝒵 ) − o ⁡ ( K  D ) ] \displaystyle\geq\operatornamewithlimits{\mathbb{E}}_{\mathcal{Z}}\Big[N\log_{2}\frac{N_{0}-N}{e^{\mathbf{loss}_{name}(\mathcal{Z})}}+NK\log_{2}\frac{D^{C}}{e^{\mathbf{loss}_{value}(\mathcal{Z})}}+KD\log_{2}\frac{T^{L}-D}{De^{(1+o(1))\mathbf{loss}_{value1}(\mathcal{Z})}}-o(KD)\Big] = N  log 2  N 0 − N e 𝔼 𝒵 𝐥𝐨𝐬𝐬 n  a  m  e  ( 𝒵 ) + N  K  log 2  D C e 𝔼 𝒵 𝐥𝐨𝐬𝐬 v  a  l  u  e  ( 𝒵 ) + K  D  log 2  T L − D D  e ( 1 + o ⁡ ( 1 ) )  𝔼 𝒵 𝐥𝐨𝐬𝐬 v  a  l  u  e  1  ( 𝒵 ) − o ⁡ ( K  D ) \displaystyle=N\log_{2}\frac{N_{0}-N}{e^{\operatornamewithlimits{\mathbb{E}}_{\mathcal{Z}}\mathbf{loss}_{name}(\mathcal{Z})}}+NK\log_{2}\frac{D^{C}}{e^{\operatornamewithlimits{\mathbb{E}}_{\mathcal{Z}}\mathbf{loss}_{value}(\mathcal{Z})}}+KD\log_{2}\frac{T^{L}-D}{De^{(1+o(1))\operatornamewithlimits{\mathbb{E}}_{\mathcal{Z}}\mathbf{loss}_{value1}(\mathcal{Z})}}-o(KD)

The goal of the paper is to study how the number of model parameters competes with this bound.

###### Corollary 3.3 (no-error case) .

In the ideal case, if for every data 𝒵 \mathcal{Z} , F F can generate a name from 𝒩 \mathcal{N} with exact 1 / N 1/N probability each, then 𝐥𝐨𝐬𝐬 n  a  m  e  ( 𝒵 ) = log ⁡ N \mathbf{loss}_{name}(\mathcal{Z})=\log N ; and if F F can 100% accurately generate values given ( n , a ) (n,a) pairs, then 𝐥𝐨𝐬𝐬 v  a  l  u  e  ( 𝒵 ) = 𝐥𝐨𝐬𝐬 v  a  l  u  e  1  ( 𝒵 ) = 0 \mathbf{loss}_{value}(\mathcal{Z})=\mathbf{loss}_{value1}(\mathcal{Z})=0 . In such a case, log 2 | 𝒲 | ≥ N  log 2  N 0 − N N + N  K  C  log 2  D + K  D  log 2  T L − D D − o ⁡ ( K  D ) \log_{2}|\mathcal{W}|\geq N\log_{2}\frac{N_{0}-N}{N}+NKC\log_{2}D+KD\log_{2}\frac{T^{L}-D}{D}-o(KD) asymptotically matching the upper bound Proposition 2.3 .

###### Remark 3.4 (why “sum of 3”) .

It is essential to obtain a lower bound that is the sum of the three components; neglecting any may result in a suboptimal bound (see examples in Appendix A.4 ).

###### Remark 3.5 (why “random data”) .

Studying a lower bound for a fixed dataset 𝒵 \mathcal{Z} is impossible — a model could hard-code 𝒵 \mathcal{Z} into its architecture even without any trainable parameter. Therefore, it is necessary to consider a lower bound with respect to a distribution over datasets.