Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws — part 7/8

Cleaned-text code points [94760, 110104). Bibliography omitted here; source.md includes it.

• For Figure 8(c) which is Case (a) of 300-exposure: – For N = 50 ​ K N=50K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 96;

– For N = 100 ​ K N=100K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 192;

– For N = 200 ​ K N=200K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 192;

– For N = 500 ​ K N=500K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 192.

• For Figure 8(d) which is Case (a) of 600-exposure: – For N = 50 ​ K N=50K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 384;

– For N = 100 ​ K N=100K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 384;

– For N = 200 ​ K N=200K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 384;

– For N = 500 ​ K N=500K , we use w ​ d = 0.002 wd=0.002 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 768.

• For Figure 8(e) which is Case (a) of 1000-exposure: – For N = 50 ​ K N=50K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0005 / 0.001 lr=0.0005/0.001 , and batch size 384;

– For N = 100 ​ K N=100K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0005 / 0.001 lr=0.0005/0.001 , and batch size 768;

– For N = 200 ​ K N=200K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0005 / 0.001 lr=0.0005/0.001 , and batch size 1536.

• For Figure 8(f) which is Case (b) of 100-exposure: – For N = 50 ​ K N=50K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 lr=0.0003/0.0005 , and batch size 12;

– For N = 100 ​ K N=100K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 lr=0.0003/0.0005 , and batch size 24;

– For N = 200 ​ K N=200K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 96;

– For N = 500 ​ K N=500K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 lr=0.0003/0.0005 , and batch size 192;

– For N = 1 ​ M N=1M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 lr=0.0003 , and batch size 192.

• For Figure 8(g) which is Case (c) of 100-exposure: – For N = 50 ​ K N=50K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 12;

– For N = 100 ​ K N=100K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 24;

– For N = 200 ​ K N=200K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0002 / 0.0003 / 0.0005 / 0.001 lr=0.0002/0.0003/0.0005/0.001 , and batch size 96;

– For N = 500 ​ K N=500K , we use w ​ d = 0.005 wd=0.005 , l ​ r = 0.0002 / 0.0003 / 0.0005 lr=0.0002/0.0003/0.0005 , and batch size 192;

– For N = 1 ​ M N=1M , we use w ​ d = 0.005 wd=0.005 , l ​ r = 0.0002 / 0.0003 / 0.0005 lr=0.0002/0.0003/0.0005 , and batch size 192.

• For Figure 8(h) which is Case (c) of 300-exposure: – For N = 50 ​ K N=50K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 96;

– For N = 100 ​ K N=100K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 192;

– For N = 200 ​ K N=200K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 192;

– For N = 500 ​ K N=500K , we use w ​ d = 0.005 wd=0.005 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 384.

## Appendix F Proof of Theorem 3.2

We present a crucial lemma that establishes the bit complexity required to encode random variables based on the probability that these variables match specific reference values.

###### Lemma F.1 .

Let 𝒬 1 , … , 𝒬 k \mathcal{Q}_{1},\dots,\mathcal{Q}_{k} be fixed sets (we call domains ), and assume that for each i ∈ [ k ] i\in[k] , Q i Q_{i} is independently and randomly chosen from its corresponding domain 𝒬 i \mathcal{Q}_{i} . Denote Q = ( Q 1 , … , Q k ) Q=(Q_{1},\dots,Q_{k}) and view Q Q as the training data .

Assume there exists a function W ⁡ ( Q ) ∈ 𝒲 W(Q)\in\mathcal{W} , which we regard as the parameters of a model computed (i.e., trained) from the training data Q Q .

Furthermore, consider an evaluation function F i F_{i} that predicts ∀ i ∈ [ k ] : P i = F i ( W ( Q ) , Q 1 , Q 2 , ⋯ , Q i − 1 , R ) with p i ( Q ) = def 𝐏𝐫 R [ P i = Q i ∣ Q ] . \forall i\in[k]\colon\hskip 20.00003ptP_{i}=F_{i}(W(Q),Q_{1},Q_{2},\cdots,Q_{i-1},R)\hskip 10.00002pt\text{with }\hskip 10.00002ptp_{i}(Q)\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\operatornamewithlimits{\mathbf{Pr}}_{R}[P_{i}=Q_{i}\mid Q]\kern 5.0pt. Here, F F is parameterized by W ⁡ ( Q ) W(Q) and may rely on previous data Q 1 , … , Q i − 1 Q_{1},\dots,Q_{i-1} , and new randomness R R . Then, it follows that log ⁡ | 𝒲 | ≥ ∑ i ∈ [ k ] log ⁡ ( 𝔼 Q [ p i ​ ( Q ) ] × | 𝒬 i | ) ≥ 𝔼 Q [ ∑ i ∈ [ k ] log ⁡ ( p i ​ ( Q ) × | 𝒬 i | ) ] . \displaystyle\log|\mathcal{W}|\geq\sum_{i\in[k]}\log\big(\operatornamewithlimits{\mathbb{E}}_{Q}[p_{i}(Q)]\times|\mathcal{Q}_{i}|\big)\geq\operatornamewithlimits{\mathbb{E}}_{Q}\Big[\sum_{i\in[k]}\log\big(p_{i}(Q)\times|\mathcal{Q}_{i}|\big)\Big]\kern 5.0pt. (F.1)

###### Proof of Lemma F.1 .

Since the second inequality of ( F.1 ) trivially comes from Jensen’s inequality, we only prove the first one.

When i = 1 i=1 , we have P 1 = F 1 ​ ( W ⁡ ( Q ) , R ) P_{1}=F_{1}(W(Q),R) and one can prove the lemma by a simple counting argument, using the property that ∀ R \forall R , P 1 = F 1 ​ ( W ⁡ ( Q ) , R ) P_{1}=F_{1}(W(Q),R) has at most | 𝒲 | |\mathcal{W}| choices of values.

When i ≥ 2 i\geq 2 , we can merge data points Q 1 , Q 2 Q_{1},Q_{2} to be a new data point Q ′ Q^{\prime} with domain 𝒬 ′ = 𝒬 1 × 𝒬 2 \mathcal{Q}^{\prime}=\mathcal{Q}_{1}\times\mathcal{Q}_{2} . We can construct P ′ = ( P 1 , P 2 ) P^{\prime}=(P_{1},P_{2}) from function F 1 , F 2 F_{1},F_{2} by sampling R 1 R_{1} to generate P 1 = F 1 ​ ( W ⁡ ( Q ) , R ) P_{1}=F_{1}(W(Q),R) , and then sample independent R 2 R_{2} to generate P 2 = F 2 ​ ( W ⁡ ( Q ) , Q 1 , R ) P_{2}=F_{2}(W(Q),Q_{1},R) . We know that 𝐏𝐫 R 1 , R 2 [ P ′ = Q ′ ∣ Q ] = 𝐏𝐫 R 1 [ P 1 = Q 1 ∣ Q ] ​ 𝐏𝐫 R 2 [ P 2 = Q 2 ∣ Q ] = p 1 ​ ( Q ) ⋅ p 2 ​ ( Q ) \operatornamewithlimits{\mathbf{Pr}}_{R_{1},R_{2}}[P^{\prime}=Q^{\prime}\mid Q]=\operatornamewithlimits{\mathbf{Pr}}_{R_{1}}[P_{1}=Q_{1}\mid Q]\operatornamewithlimits{\mathbf{Pr}}_{R_{2}}[P_{2}=Q_{2}\mid Q]=p_{1}(Q)\cdot p_{2}(Q) . The lemma now follows using the following identity: log ⁡ ( p 1 ​ ( Q ) ​ | 𝒬 1 | ) + log ⁡ ( p 2 ​ ( Q ) ​ | 𝒬 2 | ) = log ⁡ ( p 1 ​ ( Q ) ​ p 2 ​ ( Q ) ​ | 𝒬 1 | ​ | 𝒬 2 | ) . ∎ \log(p_{1}(Q)|\mathcal{Q}_{1}|)+\log(p_{2}(Q)|\mathcal{Q}_{2}|)=\log(p_{1}(Q)p_{2}(Q)|\mathcal{Q}_{1}||\mathcal{Q}_{2}|)\kern 5.0pt.\qed

### F.1 Warmup Examples

Let us first see two warmup applications of Lemma F.1 .

Value-only. Let g 1 , … , g N ∈ [ T ] g_{1},\dots,g_{N}\in[T] , where each g i g_{i} is i.i.d. uniformly chosen at random from [ T ] [T] . Think of these as values . Suppose a model, parameterized by W W , is trained on the training data 𝒵 = ( g 1 , … , g N ) \mathcal{Z}=\big(g_{1},...,g_{N}\big) . Assume this model, for a given index i ∈ [ N ] i\in[N] , can generate a random prediction f i f_{i} corresponding to g i g_{i} . We can represent this model as f i ​ ( W ​ ( 𝒵 ) , R ) f_{i}(W(\mathcal{Z}),R) , where R R denotes the randomness. The cross-entropy loss for this scenario (averaged over all possible training data) is expressed as 𝐥𝐨𝐬𝐬 = def 𝔼 g [ 𝐥𝐨𝐬𝐬 ( g ) ] = def 𝔼 g [ 1 N ∑ i ∈ [ N ] − log 𝐏𝐫 f i [ f i = g i ] ] ≥ 0 \mathbf{loss}\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\operatornamewithlimits{\mathbb{E}}_{g}[\mathbf{loss}(g)]\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\operatornamewithlimits{\mathbb{E}}_{g}\Big[\frac{1}{N}\sum_{i\in[N]}-\log\operatornamewithlimits{\mathbf{Pr}}_{f_{i}}[f_{i}=g_{i}]\Big]\geq 0 Now we apply Lemma F.1 by setting 𝒬 1 = … ​ 𝒬 N = [ T ] \mathcal{Q}_{1}=...\mathcal{Q}_{N}=[T] , Q i = g i Q_{i}=g_{i} , and P i = f i P_{i}=f_{i} . We have log | 𝒲 | ≥ 𝔼 g [ ∑ i ∈ [ N ] log 𝐏𝐫 f i [ f i = g i ] + log T ] = N log T − N 𝔼 g 𝐥𝐨𝐬𝐬 ( g ) = 𝔼 g N log T e 𝐥𝐨𝐬𝐬 ⁡ ( g ) . \displaystyle\log|\mathcal{W}|\geq\operatornamewithlimits{\mathbb{E}}_{g}\Big[\sum_{i\in[N]}\log\operatornamewithlimits{\mathbf{Pr}}_{f_{i}}[f_{i}=g_{i}]+\log T\Big]=N\log T-N\operatornamewithlimits{\mathbb{E}}_{g}\mathbf{loss}(g)=\operatornamewithlimits{\mathbb{E}}_{g}N\log\frac{T}{e^{\mathbf{loss}(g)}}\kern 5.0pt. Changing the base immediately yields a bit complexity lower bound of log 2 ⁡ | 𝒲 | ≥ N ​ log 2 ​ T e 𝐥𝐨𝐬𝐬 \log_{2}|\mathcal{W}|\geq N\log_{2}\frac{T}{e^{\mathbf{loss}}} . As the loss approaches zero, this matches the bit complexity upper bound.

Name-only. Let g 1 , … , g N ∈ [ N 0 ] g_{1},\dots,g_{N}\in[N_{0}] be N N distinct elements from [ N 0 ] [N_{0}] , sampled uniformly at random without replacement, and considered as names . Suppose a model f f , parameterized by W W , is trained on the dataset 𝒵 = ( g 1 , … , g N ) \mathcal{Z}=\big(g_{1},\dots,g_{N}\big) to predict a name. We denote this as f ⁡ ( W ⁡ ( 𝒵 ) , R ) f(W(\mathcal{Z}),R) , where R R represents randomness. The cross-entropy loss for this scenario is defined as 𝐥𝐨𝐬𝐬 = def 𝔼 g [ 𝐥𝐨𝐬𝐬 ( g ) ] = def 𝔼 g [ 1 N ∑ i ∈ [ N ] − log 𝐏𝐫 f [ f = g i ] ] ≥ 0 \mathbf{loss}\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\operatornamewithlimits{\mathbb{E}}_{g}[\mathbf{loss}(g)]\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\operatornamewithlimits{\mathbb{E}}_{g}\Big[\frac{1}{N}\sum_{i\in[N]}-\log\operatornamewithlimits{\mathbf{Pr}}_{f}[f=g_{i}]\Big]\geq 0

To apply Lemma F.1 , we define 𝒬 1 = [ N 0 ] \mathcal{Q}_{1}=[N_{0}] , 𝒬 2 = [ N 0 − 1 ] \mathcal{Q}_{2}=[N_{0}-1] , and continue until 𝒬 N = [ N 0 − N + 1 ] \mathcal{Q}_{N}=[N_{0}-N+1] . After uniformly randomly generating Q 1 , … , Q N Q_{1},\dots,Q_{N} from 𝒬 1 , … , 𝒬 N \mathcal{Q}_{1},\dots,\mathcal{Q}_{N} , we construct ( g 1 , … , g N ) ∈ [ N ] N (g_{1},\dots,g_{N})\in[N]^{N} as follows: set g 1 = Q 1 g_{1}=Q_{1} ; for g 2 g_{2} , set it to Q 2 Q_{2} if Q 2 < Q 1 Q_{2}<Q_{1} , otherwise g 2 = Q 2 + 1 g_{2}=Q_{2}+1 ; and in general, define g i g_{i} as the Q i Q_{i} -th smallest element in [ N 0 ] ∖ { g 1 , … , g i − 1 } [N_{0}]\setminus\{g_{1},\dots,g_{i-1}\} . This method provides an alternative way to generate 𝒵 = ( g 1 , … , g N ) \mathcal{Z}=(g_{1},\dots,g_{N}) , denoted as 𝒵 ⁡ ( Q ) \mathcal{Z}(Q) . For each i ∈ [ N ] i\in[N] , we define P i P_{i} as follows: first, generate f = f ⁡ ( 𝒲 ⁡ ( 𝒵 ) , R i ) f=f(\mathcal{W}(\mathcal{Z}),R_{i}) using fresh randomness R i R_{i} . Set P i = def s P_{i}\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}s if f f is the s s -th smallest element in [ N 0 ] ∖ { g 1 , … , g i − 1 } [N_{0}]\setminus\{g_{1},\dots,g_{i-1}\} , or a special symbol such as ∅ \varnothing if f f is among { g 1 , … , g i − 1 } \{g_{1},\dots,g_{i-1}\} . ( Note importantly , this definition of P i P_{i} necessitates knowledge of g 1 , … , g i − 1 g_{1},\dots,g_{i-1} ; however, this is permissible as Lemma F.1 allows P i P_{i} to depend on Q 1 , … , Q i − 1 Q_{1},\dots,Q_{i-1} .) For every fixed Q Q (and thus fixed g g ), ∑ i ∈ [ N ] log ( p i ( Q ) ) = ∑ i ∈ [ N ] log ( 𝐏𝐫 P i [ P i = Q i ] ) = ∑ i ∈ [ N ] log ( 𝐏𝐫 R i [ f ( 𝒲 ( 𝒵 ) , R i ) = g i ] ) = − N 𝐥𝐨𝐬𝐬 ( g ) \displaystyle\sum_{i\in[N]}\log\big(p_{i}(Q)\big)=\sum_{i\in[N]}\log\big(\operatornamewithlimits{\mathbf{Pr}}_{P_{i}}[P_{i}=Q_{i}]\big)=\sum_{i\in[N]}\log\big(\operatornamewithlimits{\mathbf{Pr}}_{R_{i}}[f(\mathcal{W}(\mathcal{Z}),R_{i})=g_{i}]\big)=-N\mathbf{loss}(g) Applying Lemma F.1 we have log | 𝒲 | ≥ 𝔼 Q [ ∑ i ∈ [ N ] log ⁡ ( p i ​ ( Q ) × | N 0 − i + 1 | ) ] ≥ 𝔼 Q [ N ​ log ​ N 0 − N e 𝐥𝐨𝐬𝐬 ⁡ ( g ) ] = N ​ log ​ N 0 − N e 𝔼 Q 𝐥𝐨𝐬𝐬 ​ ( g ) . \displaystyle\log|\mathcal{W}|\geq\operatornamewithlimits{\mathbb{E}}_{Q}\Big[\sum_{i\in[N]}\log\big(p_{i}(Q)\times|N_{0}-i+1|\big)\Big]\geq\operatornamewithlimits{\mathbb{E}}_{Q}\Big[N\log\frac{N_{0}-N}{e^{\mathbf{loss}(g)}}\Big]=N\log\frac{N_{0}-N}{e^{\operatornamewithlimits{\mathbb{E}}_{Q}\mathbf{loss}(g)}}\kern 5.0pt. Ideally, if the model f f can perfectly memorize the entire training set { Z } = ( g 1 , … , g N ) \{Z\}=(g_{1},\dots,g_{N}) , its best possible loss 𝐥𝐨𝐬𝐬 ⁡ ( g ) = log ⁡ N \mathbf{loss}(g)=\log N is achieved. Thus, if the model can perfectly learn this training set, the bit complexity lower bound satisfies log ⁡ | 𝒲 | ≥ N ​ log ⁡ N 0 − N N ≥ ( 1 − o ⁡ ( 1 ) ) ​ N ​ log ⁡ N 0 N \log|\mathcal{W}|\geq N\log\frac{N_{0}-N}{N}\geq(1-o(1))N\log\frac{N_{0}}{N} when N ≪ N 0 N\ll N_{0} .

### F.2 Main Proof

We recommend that readers first review the warmup examples in Section F.1 before proceeding with this proof.

###### Proof of Theorem 3.2 .

Let us first construct the domains 𝒬 i \mathcal{Q}_{i} ’s in Lemma F.1 . 1. Let 𝒬 1 = [ N 0 ] \mathcal{Q}_{1}=[N_{0}] , 𝒬 2 = [ N 0 − 1 ] ⋯ 𝒬 N = [ N 0 − N + 1 ] \mathcal{Q}_{2}=[N_{0}-1]\cdots\mathcal{Q}_{N}=[N_{0}-N+1] .

2. Let ( 𝒬 N + j ​ D + 1 , … ​ 𝒬 N + j ​ D + D ) = ( [ T L ] , [ T L − 1 ] , … , [ T L − D + 1 ] ) \big(\mathcal{Q}_{N+jD+1},\dots\mathcal{Q}_{N+jD+D}\big)=\big([T^{L}],[T^{L}-1],\dots,[T^{L}-D+1]\big) for every j = 0 , … , K − 1 j=0,\dots,K-1 .

3. Let 𝒬 N + K ​ D + 1 = ⋯ = 𝒬 N + K ​ D + N ​ K = [ D C ] \mathcal{Q}_{N+KD+1}=\cdots=\mathcal{Q}_{N+KD+NK}=[D^{C}] .

Recall that each Q i Q_{i} is independently and uniformly generated at random from 𝒬 i \mathcal{Q}_{i} . We now present an alternative method for generating the training dataset 𝒵 ⁡ ( Q ) \mathcal{Z}(Q) .

1. Construct 𝒩 = ( n 1 , … , n N ) \mathcal{N}=(n_{1},\dots,n_{N}) as follows: Let n 1 n_{1} be the Q 1 Q_{1} -th name from 𝒩 0 \mathcal{N}_{0} ; for i > 1 i>1 , let n i n_{i} be the Q i Q_{i} -th name from 𝒩 0 ∖ { n 1 , … , n i − 1 } \mathcal{N}_{0}\setminus\{n_{1},\dots,n_{i-1}\} .

2. For each a ′ ∈ [ K ] a^{\prime}\in[K] , let a a be the a ′ a^{\prime} -th attribute in 𝒜 \mathcal{A} . Construct 𝒟 a = ( w 1 , … , w D ) \mathcal{D}_{a}=(w_{1},\dots,w_{D}) as follows: Let w 1 w_{1} be the Q N + ( a ′ − 1 ) ​ D + 1 Q_{N+(a^{\prime}-1)D+1} -th element in 𝒯 L \mathcal{T}^{L} ; for i > 1 i>1 , let w i w_{i} be the Q N + ( a ′ − 1 ) ​ D + i Q_{N+(a^{\prime}-1)D+i} -th element in 𝒯 L ∖ { w 1 , … , w i − 1 } \mathcal{T}^{L}\setminus\{w_{1},\dots,w_{i-1}\} .

3. For the n ′ n^{\prime} -th name n n and the a ′ a^{\prime} -th attribute a a , assign its value v ⋆ ​ ( n , a ) = ( v 1 , … , v C ) ∈ ( 𝒟 a ) C v^{\star}(n,a)=(v_{1},\dots,v_{C})\in(\mathcal{D}_{a})^{C} by setting each v i v_{i} as the s i s_{i} -th element in 𝒟 a \mathcal{D}_{a} , where the integer sequence ( s 1 , … , s C ) := Q N + K ​ D + ( n ′ − 1 ) ​ K + a ′ ∈ [ D C ] (s_{1},\dots,s_{C}):=Q_{N+KD+(n^{\prime}-1)K+a^{\prime}}\in[D^{C}] .

It is easy to verify that this gives the same dataset distribution as Definition 2.2 . Next, consider Q Q being fixed (thus the dataset 𝒵 \mathcal{Z} being fixed), we construct P 1 , P 2 , ⋯ , P N + K ​ D + N ​ K P_{1},P_{2},\cdots,P_{N+KD+NK} using the given model functions F ⊤ ​ ( W ​ ( 𝒵 ) , R ) F^{\top}(W(\mathcal{Z}),R) and F ⊥ ​ ( W ⁡ ( 𝒵 ) , n , a , R ) F^{\bot}(W(\mathcal{Z}),n,a,R) .