MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters — part 5/10

Cleaned-text code points [51651, 66080). Bibliography omitted here; source.md includes it.

The meta update based on the Adam algorithm is as follows, 𝒎 ¯ t + 1 = ρ ¯ 𝒎 ¯ t + ℋ t T ∇ f t ( 𝒘 t ) , 𝒗 ¯ t + 1 = λ ¯ 𝒗 t + ( ℋ t T ∇ f t ( 𝒘 t ) ) 2 , μ ¯ t = ( 1 − ρ ¯ 1 − ρ ¯ t ) / 1 − λ ¯ 1 − λ ¯ t , 𝜷 t + 1 = 𝜷 t − η ​ μ ¯ t ​ 𝒎 ¯ t 𝒗 ¯ t \begin{split}\bar{\boldsymbol{m}}_{t+1}&=\bar{\rho}\,\bar{\boldsymbol{m}}_{t}\,+\,\mathcal{H}_{t}^{T}\nabla f_{t}(\boldsymbol{w}_{t}),\\ \vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t+1}&=\bar{\lambda}\,\boldsymbol{v}_{t}\,+\,\big(\mathcal{H}_{t}^{T}\nabla f_{t}(\boldsymbol{w}_{t})\big)^{2},\\ \bar{\mu}_{t}&=\left(\frac{1-\bar{\rho}}{{\color[rgb]{0.75,0.75,0.75}1-\bar{\rho}^{t}}}\right)\,/\,\sqrt{\frac{1-\bar{\lambda}}{1-\bar{\lambda}^{t}}},\\ \boldsymbol{\beta}_{t+1}&=\boldsymbol{\beta}_{t}\,-\,\eta\,\bar{\mu}_{t}\frac{\bar{\boldsymbol{m}}_{t}}{\sqrt{\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t}}}\end{split} (27) where 𝒎 ¯ t \bar{\boldsymbol{m}}_{t} is the momentum vector, 𝒗 ¯ t \vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t} is the trace of squared surrogate-meta-gradient. Since Adam algorithm needs to keep track of 𝜷 t \boldsymbol{\beta}_{t} , 𝒎 ¯ t \bar{\boldsymbol{m}}_{t} , and 𝒗 ¯ t \vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t} , we have 𝒚 t = [ 𝜷 t 𝒎 ¯ t 𝒗 ¯ t ] . \boldsymbol{y}_{t}=\left[\begin{array}[]{c}\boldsymbol{\beta}_{t}\\ \bar{\boldsymbol{m}}_{t}\\ \vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t}\end{array}\right]. (28) Recall the following notation convention at the end of the Introduction section: for any k ≥ 1 k\geq 1 , and any k k -dimensional vector 𝒗 = [ v 1 , … , v k ] \boldsymbol{v}=[v_{1},\ldots,v_{k}] , we denote the the corresponding diagonal matrix by [ 𝒗 ] \left[\boldsymbol{v}\right] : [ 𝒗 ] ​ = def ​ [ v 1 ⋯ 0 ⋮ ⋱ ⋮ 0 ⋯ v k ] . \left[\boldsymbol{v}\right]\overset{\text{\tiny def}}{=}\left[\begin{array}[]{ccc}v_{1}&\cdots&0\\ \vdots&\ddots&\vdots\\ 0&\cdots&v_{k}\end{array}\right]. (29) Consequently, from ( 21 ), we obtain G t meta = [ d ​ 𝒚 t + 1 d ​ 𝒚 t d ​ 𝒚 t + 1 d ​ 𝒘 t 0 d ​ 𝒚 t + 1 d ​ 𝒉 t ] = [ d ​ 𝜷 t + 1 d ​ 𝜷 t d ​ 𝜷 t + 1 d ​ 𝒎 ¯ t d ​ 𝜷 t + 1 d ​ 𝒗 ¯ t d ​ 𝜷 t + 1 d ​ 𝒘 t 0 d ​ 𝜷 t + 1 d ​ 𝒉 t d ​ 𝒎 ¯ t + 1 d ​ 𝜷 t d ​ 𝒎 ¯ t + 1 d ​ 𝒎 ¯ t d ​ 𝒎 ¯ t + 1 d ​ 𝒗 ¯ t d ​ 𝒎 ¯ t + 1 d ​ 𝒘 t 0 d ​ 𝒎 ¯ t + 1 d ​ 𝒉 t d ​ 𝒗 ¯ ​ t + 1 d ​ 𝜷 t d ​ 𝒗 ¯ ​ t + 1 d ​ 𝒎 ¯ t d ​ 𝒗 ¯ ​ t + 1 d ​ 𝒗 ¯ t d ​ 𝒗 ¯ ​ t + 1 d ​ 𝒘 t 0 d ​ 𝒗 ¯ ​ t + 1 d ​ 𝒉 t ] = [ I − η ​ μ ¯ t ​ [ 1 𝒗 ¯ t ] η ​ μ ¯ t 2 ​ [ 𝒎 ¯ t 𝒗 ¯ t 1.5 ] 0 0 0 0 ρ ¯ ​ I 0 ℋ t T ​ ∇ 2 f t 0 d ​ 𝒎 ¯ t + 1 d ​ 𝒉 t 0 0 λ ¯ ​ I 2 [ ℋ t T ∇ f t ] ℋ t T ∇ 2 f t 0 d ​ 𝒗 ¯ t + 1 d ​ 𝒉 t ] , \begin{split}G^{\textrm{meta}}_{t}&=\left[\begin{array}[]{c|cc|c}\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{y}_{t+1}}{\mathrm{d}\,\boldsymbol{y}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{y}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{y}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right]\\ &=\left[\begin{array}[]{ccc|cc|c}\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{\beta}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{\beta}_{t+1}}{\mathrm{d}\,\bar{\boldsymbol{m}}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{\beta}_{t+1}}{\mathrm{d}\,\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{\beta}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{\beta}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\bar{\boldsymbol{m}}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}{t+1}}{\mathrm{d}\,\bar{\boldsymbol{m}}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}{t+1}}{\mathrm{d}\,\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right]\\ &=\left[\begin{array}[]{ccc|cc|c}I&-\eta\bar{\mu}_{t}\Big[\frac{1}{\sqrt{\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t}}}\Big]&\frac{\eta\bar{\mu}_{t}}{2}\Big[\frac{\bar{\boldsymbol{m}}_{t}}{\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t}^{1.5}}\Big]&0&0&0\\ 0&\bar{\rho}I&0&\mathcal{H}_{t}^{T}\nabla^{2}f_{t}&0&\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}\\ 0&0&\bar{\lambda}I&2\big[\mathcal{H}_{t}^{T}\nabla f_{t}\big]\mathcal{H}_{t}^{T}\nabla^{2}f_{t}&0&\frac{\mathrm{d}\,\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}\end{array}\right],\end{split} (30) where the last equality follows by calculating derivatives of ( 27 ). For the two remaining terms in the last column of G t G_{t} , we have d ​ 𝒎 ¯ t + 1 d ​ 𝒉 t = d d ​ 𝒉 t ( ℋ t T ∇ f t ( 𝒘 t ) ) = η d d ​ 𝒉 t ( L t 𝒉 t ) = η L t . \frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}\,=\,\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{h}_{t}}\big(\mathcal{H}_{t}^{T}\nabla f_{t}(\boldsymbol{w}_{t})\big)\,=\,\eta\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{h}_{t}}\big(L_{t}\boldsymbol{h}_{t}\big)\,=\eta\,L_{t}. (31) where the first equality follows from the update of 𝒎 ¯ t + 1 \bar{\boldsymbol{m}}_{t+1} in ( 27 ), and the second equality is due to ( 23 ). In the same vein, d ​ 𝒗 ¯ t + 1 d ​ 𝒉 t = d d ​ 𝒉 t ( ℋ t T ∇ f t ( 𝒘 t ) ) 2 = d d ​ 𝒉 t ( L t 𝒉 t ) 2 = 2 [ L t 𝒉 t ] d d ​ 𝒉 t ( L t 𝒉 t ) = 2 [ L t 𝒉 t ] L t = 2 [ ℋ t T ∇ f t ( 𝒘 t ) ] L t , \frac{\mathrm{d}\,\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}\,=\,\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{h}_{t}}\big(\mathcal{H}_{t}^{T}\nabla f_{t}(\boldsymbol{w}_{t})\big)^{2}\,=\,\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{h}_{t}}\big(L_{t}\boldsymbol{h}_{t}\big)^{2}\,=\,2\big[L_{t}\boldsymbol{h}_{t}\big]\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{h}_{t}}\big(L_{t}\boldsymbol{h}_{t}\big)\,=\,2\big[L_{t}\boldsymbol{h}_{t}\big]\,L_{t}\,=\,2\big[\mathcal{H}_{t}^{T}\,\nabla f_{t}(\boldsymbol{w}_{t})\big]\,L_{t}, (32) where the first equality follows from the update of 𝒗 ¯ t + 1 \vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t+1} in ( 27 ), the second equality is due to ( 23 ), and the last equality is again from ( 23 ).

Plugging ( 31 ) and ( 32 ) into ( 30 ), we obtain G t meta = [ I − η ​ μ ¯ t ​ [ 1 𝒗 ¯ t ] η ​ μ ¯ t 2 ​ [ 𝒎 ¯ t 𝒗 ¯ t 1.5 ] 0 0 0 0 ρ ¯ ​ I 0 ℋ t T ​ ∇ 2 f t 0 η ​ L t 0 0 λ ¯ ​ I 2 [ ℋ t T ∇ f t ] ℋ t T ∇ 2 f t 0 2 [ ℋ t T ∇ f t ] L t ] . G^{\textrm{meta}}_{t}=\left[\begin{array}[]{ccc|cc|c}I&-\eta\bar{\mu}_{t}\Big[\frac{1}{\sqrt{\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t}}}\Big]&\frac{\eta\bar{\mu}_{t}}{2}\Big[\frac{\bar{\boldsymbol{m}}_{t}}{\vphantom{\tilde{\zeta}}\bar{\boldsymbol{v}}_{t}^{1.5}}\Big]&0&0&0\\ 0&\bar{\rho}I&0&\mathcal{H}_{t}^{T}\nabla^{2}f_{t}&0&\eta\,L_{t}\\ 0&0&\bar{\lambda}I&2\big[\mathcal{H}_{t}^{T}\nabla f_{t}\big]\mathcal{H}_{t}^{T}\nabla^{2}f_{t}&0&2\big[\mathcal{H}_{t}^{T}\,\nabla f_{t}\big]\,L_{t}\end{array}\right]. (33)

#### A.1.3 Meta Lion

The meta update based on the lion algorithm is as follows 𝒎 ¯ t + 1 = ρ ​ 𝒎 ¯ t + ( 1 − ρ ) ​ ∇ 𝜷 F ^ t , \displaystyle\bar{\boldsymbol{m}}_{t+1}=\rho\,\bar{\boldsymbol{m}}_{t}+(1-\rho)\,\widehat{\nabla_{\boldsymbol{\beta}}F}_{t}, (34) 𝜷 t + 1 = 𝜷 t − η ​ Sign ⁡ ( c ​ 𝒎 ¯ t + ( 1 − c ) ​ ∇ 𝜷 F ^ t ) , \displaystyle\boldsymbol{\beta}_{t+1}=\boldsymbol{\beta}_{t}-\eta\operatorname{Sign}\big(c\,\bar{\boldsymbol{m}}_{t}+(1-c)\widehat{\nabla_{\boldsymbol{\beta}}F}_{t}\big), (35) where η \eta is a scalar, called the meta step size , and ρ , c ∈ [ 0 , 1 ) \rho,c\in[0,1) . Note that the meta algorithm operates on a low dimensional space. Therefore, we drop the regularizers like weight-decay in the meta updates, as they are primarily aimed to resolve the overfitting problem in high dimensional problems. Substituting ∇ 𝜷 F ^ t \widehat{\nabla_{\boldsymbol{\beta}}F}_{t} with ℋ t T ∇ f t ( 𝒘 t ) \mathcal{H}_{t}^{T}\nabla f_{t}(\boldsymbol{w}_{t}) we obtain the following meta updates 𝒎 ¯ t + 1 = ρ 𝒎 ¯ t + ( 1 − ρ ) ℋ t T ∇ f t ( 𝒘 t ) , \displaystyle\bar{\boldsymbol{m}}_{t+1}=\rho\,\bar{\boldsymbol{m}}_{t}+(1-\rho)\,\mathcal{H}_{t}^{T}\nabla f_{t}(\boldsymbol{w}_{t}), (36) 𝜷 t + 1 = 𝜷 t − η Sign ( c 𝒎 ¯ t + ( 1 − c ) ℋ t T ∇ f t ( 𝒘 t ) ) . \displaystyle\boldsymbol{\beta}_{t+1}=\boldsymbol{\beta}_{t}-\eta\operatorname{Sign}\big(c\,\bar{\boldsymbol{m}}_{t}+(1-c)\mathcal{H}_{t}^{T}\nabla f_{t}(\boldsymbol{w}_{t})\big). (37) In this case, 𝒚 t = [ 𝜷 t 𝒎 ¯ t ] , \boldsymbol{y}_{t}=\left[\begin{array}[]{c}\boldsymbol{\beta}_{t}\\ \bar{\boldsymbol{m}}_{t}\end{array}\right], and G t meta = [ d ​ 𝒚 t + 1 d ​ 𝒚 t d ​ 𝒚 t + 1 d ​ 𝒘 t 0 d ​ 𝒚 t + 1 d ​ 𝒉 t ] = [ d ​ 𝜷 t + 1 d ​ 𝜷 t d ​ 𝜷 t + 1 d ​ 𝒎 ¯ t d ​ 𝜷 t + 1 d ​ 𝒘 t 0 d ​ 𝜷 t + 1 d ​ 𝒉 t d ​ 𝒎 ¯ t + 1 d ​ 𝜷 t d ​ 𝒎 ¯ t + 1 d ​ 𝒎 ¯ t d ​ 𝒎 ¯ t + 1 d ​ 𝒘 t 0 d ​ 𝒎 ¯ t + 1 d ​ 𝒉 t ] = [ I 0 0 0 0 d ​ 𝒎 ¯ t + 1 d ​ 𝜷 t d ​ 𝒎 ¯ t + 1 d ​ 𝒎 ¯ t d ​ 𝒎 ¯ t + 1 d ​ 𝒘 t 0 d ​ 𝒎 ¯ t + 1 d ​ 𝒉 t ] , \begin{split}G^{\textrm{meta}}_{t}&=\left[\begin{array}[]{ccccc}\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{y}_{t+1}}{\mathrm{d}\,\boldsymbol{y}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{y}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{y}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right]\\ &=\left[\begin{array}[]{ccccc}\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{\beta}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{\beta}_{t+1}}{\mathrm{d}\,\bar{\boldsymbol{m}}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{\beta}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{\beta}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\bar{\boldsymbol{m}}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right]\\ &=\left[\begin{array}[]{ccccc}\hskip-7.0pt\begin{array}[]{c}\\ {I}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {0}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {0}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {0}\\ \\ \end{array}\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\bar{\boldsymbol{m}}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right],\end{split} (38) where the last equality follows from ( 37 ). Consider the following block representation of Y t Y_{t} : Y t = [ B t Y t m ¯ ] . Y_{t}=\left[\begin{array}[]{c}B_{t}\\ Y^{\bar{m}}_{t}\end{array}\right]. (39) Since the base algorithm, does not take 𝒎 ¯ \bar{\boldsymbol{m}} as input, as we will see in ( 41 ) and ( 42 ) of next subsection (Appendix A.2 ), d ​ 𝒎 ¯ t + 1 d ​ 𝒎 ¯ t \frac{\mathrm{d}\,\bar{\boldsymbol{m}}_{t+1}}{\mathrm{d}\,\bar{\boldsymbol{m}}_{t}} is the only non-zero block of G t G_{t} in its column of blocks (i.e., d ​ s t + 1 d ​ 𝒎 ¯ t = 0 \frac{\mathrm{d}\,s_{t+1}}{\mathrm{d}\,\bar{\boldsymbol{m}}_{t}}=0 for every variable s s other than 𝒎 ¯ \bar{\boldsymbol{m}} ). Consequently, it follows from ( 14 ) that Y t m ¯ Y^{\bar{m}}_{t} as defined in ( 39 ), has no impact on the update of X t + 1 X_{t+1} , B t + 1 B_{t+1} , and Q t + 1 Q_{t+1} . Therefore, we can zero-out the rows and columns of G meta G^{\textrm{meta}} that correspond to derivative of 𝒎 ¯ \bar{\boldsymbol{m}} . As such we obtain the following equivalent of G meta G^{\textrm{meta}} in ( 38 ) from an algorithmic perspective: G t meta ≡ [ I m × m 0 0 0 ] . \begin{split}G^{\textrm{meta}}_{t}\equiv\left[\begin{array}[]{cc}\hskip-7.0pt\begin{array}[]{c}\\ {I_{m\times m}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {0}\\ \\ \end{array}\\ \hskip-7.0pt\begin{array}[]{c}\\ {0}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {0}\\ \\ \end{array}\end{array}\right].\end{split} (40) As a result, we get B t = I B_{t}=I for all times t t .

### A.2 Derivation of G base G^{\textrm{base}} for Different Base Updates

We now turn our focus to computation of G base G^{\textrm{base}} . Let us start by simplifying G base G^{\textrm{base}} , and introducing some notations.