MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters — part 6/10

Cleaned-text code points [66080, 80289). Bibliography omitted here; source.md includes it.

Note that the base update has no dependence on internal variables, 𝒚 ~ \tilde{\boldsymbol{y}} , of the meta update. As a result, d ​ 𝒙 t + 1 d ​ 𝒚 ~ t = 0 . \frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\tilde{\boldsymbol{y}}_{t}}=0. (41) Moreover, it follows from the definition of ℋ t \mathcal{H}_{t} in ( 7 ) that d ​ ℋ t + 1 d ​ 𝒚 ~ t = ( 1 − γ ) ​ ∑ t = 0 t γ t − τ ​ d d ​ 𝒚 ~ t ​ ( d ​ 𝒘 t + 1 d ​ 𝜷 τ ) = ( 1 − γ ) ​ ∑ t = 0 t γ t − τ ​ d d ​ 𝜷 τ ​ ( d ​ 𝒘 t + 1 d ​ 𝒚 ~ t ) = ( 1 − γ ) ​ ∑ t = 0 t γ t − τ ​ d d ​ 𝜷 τ ​ ( 0 ) = 0 , \frac{\mathrm{d}\,\mathcal{H}_{t+1}}{\mathrm{d}\,\tilde{\boldsymbol{y}}_{t}}=(1-\gamma)\sum_{t=0}^{t}\gamma^{t-\tau}\frac{\mathrm{d}\,}{d\tilde{\boldsymbol{y}}_{t}}\,\left(\frac{d\boldsymbol{w}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{\tau}}\right)=(1-\gamma)\sum_{t=0}^{t}\gamma^{t-\tau}\frac{\mathrm{d}\,}{d\boldsymbol{\beta}_{\tau}}\,\left(\frac{d\boldsymbol{w}_{t+1}}{\mathrm{d}\,\tilde{\boldsymbol{y}}_{t}}\right)=(1-\gamma)\sum_{t=0}^{t}\gamma^{t-\tau}\frac{\mathrm{d}\,}{d\boldsymbol{\beta}_{\tau}}\,\left(0\right)=0, where the third equality follows from ( 41 ). Therefore, d ​ 𝒉 t + 1 d ​ 𝒚 ~ t = 0 . \frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\tilde{\boldsymbol{y}}_{t}}=0. (42) Note also that Alg base \operatorname{Alg}_{\textrm{base}} does not take ℋ t \mathcal{H}_{t} as input, and therefore, d ​ 𝒙 t + 1 d ​ 𝒉 t = 0 . \frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}=0. (43) Consequently, we can simplify G t base G^{\textrm{base}}_{t} as follows, G t base = [ d ​ 𝒙 t + 1 d ​ 𝒚 t d ​ 𝒙 t + 1 d ​ 𝒙 t d ​ 𝒙 t + 1 d ​ 𝒉 t d ​ 𝒉 t + 1 d ​ 𝒚 t d ​ 𝒉 t + 1 d ​ 𝒙 t d ​ 𝒉 t + 1 d ​ 𝒉 t ] = [ d ​ 𝒙 t + 1 d ​ 𝜷 t d ​ 𝒙 t + 1 d ​ 𝒚 ~ t d ​ 𝒙 t + 1 d ​ 𝒙 t d ​ 𝒙 t + 1 d ​ 𝒉 t d ​ 𝒉 t + 1 d ​ 𝜷 t d ​ 𝒉 t + 1 d ​ 𝒚 ~ t d ​ 𝒉 t + 1 d ​ 𝒙 t d ​ 𝒉 t + 1 d ​ 𝒉 t ] = [ d ​ 𝒙 t + 1 d ​ 𝜷 t 0 d ​ 𝒙 t + 1 d ​ 𝒙 t 0 d ​ 𝒉 t + 1 d ​ 𝜷 t 0 d ​ 𝒉 t + 1 d ​ 𝒙 t d ​ 𝒉 t + 1 d ​ 𝒉 t ] , G^{\textrm{base}}_{t}=\left[\begin{array}[]{ccc}\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\boldsymbol{y}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\boldsymbol{x}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{y}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{x}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right]=\left[\begin{array}[]{cccc}\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\tilde{\boldsymbol{y}}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\boldsymbol{x}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\tilde{\boldsymbol{y}}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{x}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right]=\left[\begin{array}[]{cccc}\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\boldsymbol{x}_{t}}}\\ \\ \end{array}&0\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{x}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right], (44) where the last equality is due to ( 41 ), ( 42 ), and ( 43 ).

On an independent note, consider the following block representation of Y t Y_{t} , Y t = [ B t − 1 − γ γ ​ I Y ~ t ] , Y_{t}=\left[\begin{array}[]{c}B_{t}-\frac{1-\gamma}{\gamma}I\\ \tilde{Y}_{t}\end{array}\right], (45) Therefore, γ ​ Y t + ( 1 − γ ) ​ [ I 0 ] = γ ​ [ B t Y ~ t ] \gamma Y_{t}+(1-\gamma)\left[\begin{array}[]{c}I\\ 0\end{array}\right]=\gamma\,\left[\begin{array}[]{c}B_{t}\\ \tilde{Y}_{t}\end{array}\right] It then follows from ( 19 ) and ( 14 ) that [ X t + 1 Q t + 1 ] = γ ​ G t base ​ [ [ B t Y ~ t ] X t Q t ] . \left[\begin{array}[]{c}X_{t+1}\\ Q_{t+1}\end{array}\right]=\gamma\,G^{\textrm{base}}_{t}\left[\begin{array}[]{c}\left[\begin{array}[]{c}B_{t}\\ \tilde{Y}_{t}\end{array}\right]\\ X_{t}\\ Q_{t}\end{array}\right]. (46) Moreover, from the definition of Y t Y_{t} in ( 11 ), we have d ​ B t d ​ 𝒙 t = ( 1 − γ ) ​ d d ​ 𝒙 t ​ ∑ τ = 0 t γ t − τ ​ d ​ 𝜷 t d ​ 𝜷 τ = ( 1 − γ ) ​ ∑ τ = 0 t γ t − τ ​ d d ​ 𝜷 τ ​ ( d ​ 𝜷 t d ​ 𝒙 t ) = ( 1 − γ ) ​ ∑ τ = 0 t γ t − τ ​ d d ​ 𝜷 τ ​ ( 0 ) = 0 , d ​ B t d ​ 𝜷 t = ( 1 − γ ) ​ d d ​ 𝜷 t ​ ∑ τ = 0 t γ t − τ ​ d ​ 𝜷 t d ​ 𝜷 τ = ( 1 − γ ) ​ ∑ τ = 0 t γ t − τ ​ d d ​ 𝜷 τ ​ ( d ​ 𝜷 t d ​ 𝜷 t ) = ( 1 − γ ) ​ ∑ τ = 0 t γ t − τ ​ d d ​ 𝜷 τ ​ ( I ) = 0 , d ​ B t d ​ 𝒉 t = ( 1 − γ ) ​ d d ​ 𝒉 t ​ ∑ τ = 0 t γ t − τ ​ d ​ 𝜷 t d ​ 𝜷 τ = ( 1 − γ ) ​ ∑ τ = 0 t γ t − τ ​ d d ​ 𝜷 τ ​ ( d ​ 𝜷 t d ​ 𝒉 t ) = ( 1 − γ ) ​ ∑ τ = 0 t γ t − τ ​ d d ​ 𝜷 τ ​ ( 0 ) = 0 . \begin{split}\frac{\mathrm{d}\,B_{t}}{\mathrm{d}\,\boldsymbol{x}_{t}}&=(1-\gamma)\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{x}_{t}}\sum_{\tau=0}^{t}\gamma^{t-\tau}\frac{\mathrm{d}\,\boldsymbol{\beta}_{t}}{\mathrm{d}\,\boldsymbol{\beta}_{\tau}}=(1-\gamma)\sum_{\tau=0}^{t}\gamma^{t-\tau}\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{\beta}_{\tau}}\left(\frac{\mathrm{d}\,\boldsymbol{\beta}_{t}}{\mathrm{d}\,\boldsymbol{x}_{t}}\right)=(1-\gamma)\sum_{\tau=0}^{t}\gamma^{t-\tau}\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{\beta}_{\tau}}\big(0\big)=0,\\ \frac{\mathrm{d}\,B_{t}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}&=(1-\gamma)\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{\beta}_{t}}\sum_{\tau=0}^{t}\gamma^{t-\tau}\frac{\mathrm{d}\,\boldsymbol{\beta}_{t}}{\mathrm{d}\,\boldsymbol{\beta}_{\tau}}=(1-\gamma)\sum_{\tau=0}^{t}\gamma^{t-\tau}\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{\beta}_{\tau}}\left(\frac{\mathrm{d}\,\boldsymbol{\beta}_{t}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}\right)=(1-\gamma)\sum_{\tau=0}^{t}\gamma^{t-\tau}\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{\beta}_{\tau}}\big(I\big)=0,\\ \frac{\mathrm{d}\,B_{t}}{\mathrm{d}\,\boldsymbol{h}_{t}}&=(1-\gamma)\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{h}_{t}}\sum_{\tau=0}^{t}\gamma^{t-\tau}\frac{\mathrm{d}\,\boldsymbol{\beta}_{t}}{\mathrm{d}\,\boldsymbol{\beta}_{\tau}}=(1-\gamma)\sum_{\tau=0}^{t}\gamma^{t-\tau}\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{\beta}_{\tau}}\left(\frac{\mathrm{d}\,\boldsymbol{\beta}_{t}}{\mathrm{d}\,\boldsymbol{h}_{t}}\right)=(1-\gamma)\sum_{\tau=0}^{t}\gamma^{t-\tau}\frac{\mathrm{d}\,}{\mathrm{d}\,\boldsymbol{\beta}_{\tau}}\big(0\big)=0.\end{split} (47)

Finally, recall the definition σ ′ ​ ( 𝜷 t ) ​ = def ​ d ​ 𝜶 t d ​ 𝜷 t \sigma^{\prime}(\boldsymbol{\beta}_{t})\overset{\text{\tiny def}}{=}\frac{\mathrm{d}\,\boldsymbol{\alpha}_{t}}{\mathrm{d}\,\boldsymbol{\beta}_{t}} (48) as the Jacobian of 𝜶 t \boldsymbol{\alpha}_{t} with respect to 𝜷 t \boldsymbol{\beta}_{t} .

We now proceed to derivation of G base G^{\textrm{base}} for different choices of Alg base \operatorname{Alg}_{\textrm{base}} .

### A.3 Base SGD

Base SGD algorithm makes the following base update in each iteration: 𝒘 t + 1 = 𝒘 t − 𝜶 t ∇ f t ( 𝒘 t ) . \boldsymbol{w}_{t+1}=\boldsymbol{w}_{t}-\boldsymbol{\alpha}_{t}\nabla f_{t}(\boldsymbol{w}_{t}). (49) In this case, 𝒙 t = 𝒘 t \boldsymbol{x}_{t}=\boldsymbol{w}_{t} and X t = ℋ t X_{t}=\mathcal{H}_{t} . Then, G t base G^{\textrm{base}}_{t} in ( 44 ) can be simplified to G t base = [ d ​ 𝒙 t + 1 d ​ 𝜷 t 0 d ​ 𝒙 t + 1 d ​ 𝒙 t 0 d ​ 𝒉 t + 1 d ​ 𝜷 t 0 d ​ 𝒉 t + 1 d ​ 𝒙 t d ​ 𝒉 t + 1 d ​ 𝒉 t ] = [ d ​ 𝒘 t + 1 d ​ 𝜷 t 0 d ​ 𝒘 t + 1 d ​ 𝒘 t 0 d ​ 𝒉 t + 1 d ​ 𝜷 t 0 d ​ 𝒉 t + 1 d ​ 𝒘 t d ​ 𝒉 t + 1 d ​ 𝒉 t ] = [ − [ ∇ f t ​ ( 𝒘 t ) ] ​ σ ′ ​ ( 𝜷 t ) 0 I − [ 𝜶 t ] ​ ∇ 2 f t ​ ( 𝒘 t ) 0 d ​ 𝒉 t + 1 d ​ 𝜷 t 0 d ​ 𝒉 t + 1 d ​ 𝒘 t d ​ 𝒉 t + 1 d ​ 𝒉 t ] , \begin{split}G^{\textrm{base}}_{t}&=\left[\begin{array}[]{cccc}\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\boldsymbol{x}_{t}}}\\ \\ \end{array}&0\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{x}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right]\\ &=\left[\begin{array}[]{cccc}\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{w}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{w}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&0\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right]\\ &=\left[\begin{array}[]{cccc}-\left[\nabla f_{t}(\boldsymbol{w}_{t})\right]\sigma^{\prime}(\boldsymbol{\beta}_{t})&0&I-\left[\boldsymbol{\alpha}_{t}\right]\nabla^{2}f_{t}(\boldsymbol{w}_{t})&0\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right],\end{split} (50) where the last equality follows by computing simple derivatives of 𝒘 t + 1 \boldsymbol{w}_{t+1} in ( 49 ).

We proceed to compute the three remaining entries of G t base G^{\textrm{base}}_{t} , i.e., d ​ 𝒉 t + 1 / d ​ 𝜷 t {\mathrm{d}\,\boldsymbol{h}_{t+1}}/{\mathrm{d}\,\boldsymbol{\beta}_{t}} , d ​ 𝒉 t + 1 / d ​ 𝒘 t {\mathrm{d}\,\boldsymbol{h}_{t+1}}/{\mathrm{d}\,\boldsymbol{w}_{t}} , and d ​ 𝒉 t + 1 / d ​ 𝒉 t {\mathrm{d}\,\boldsymbol{h}_{t+1}}/{\mathrm{d}\,\boldsymbol{h}_{t}} . Note that by plugging the first row of G t base G^{\textrm{base}}_{t} , given in ( 50 ), into ( 46 ), and noting that ℋ t = X t \mathcal{H}_{t}=X_{t} , we obtain ℋ t + 1 = γ ⁡ ( I − [ 𝜶 t ] ​ ∇ 2 f t ​ ( 𝒘 t ) ) ​ ℋ t − γ ⁡ [ ∇ f t ​ ( 𝒘 t ) ] ​ σ ′ ​ ( 𝜷 t ) ​ B t , \mathcal{H}_{t+1}=\gamma\big(I-\left[\boldsymbol{\alpha}_{t}\right]\nabla^{2}f_{t}(\boldsymbol{w}_{t})\big)\mathcal{H}_{t}\,-\,\gamma\left[\nabla f_{t}(\boldsymbol{w}_{t})\right]\sigma^{\prime}(\boldsymbol{\beta}_{t})\,B_{t}, (51) for all t ≥ 0 t\geq 0 . By vectorizing both sides of ( 51 ) we obtain 𝒉 t + 1 = γ ⁡ [ ( I − [ 𝜶 t ] ​ ∇ 2 f t ) ​ ℋ t [ 1 ] − [ ∇ f t ] ​ σ ′ ​ ( 𝜷 t ) ​ B t [ 1 ] ( I − [ 𝜶 t ] ​ ∇ 2 f t ) ​ ℋ t [ 2 ] − [ ∇ f t ] ​ σ ′ ​ ( 𝜷 t ) ​ B t [ 2 ] ⋮ ( I − [ 𝜶 t ] ​ ∇ 2 f t ) ​ ℋ t [ m ] − [ ∇ f t ] ​ σ ′ ​ ( 𝜷 t ) ​ B t [ m ] ] . \boldsymbol{h}_{t+1}\,=\,\gamma\left[\begin{array}[]{c}\hskip-7.0pt\begin{array}[]{c}\\ {\big(I-\left[\boldsymbol{\alpha}_{t}\right]\nabla^{2}f_{t}\big)\,\mathcal{H}_{t}^{[1]}\,-\,\left[\nabla f_{t}\right]\,\sigma^{\prime}(\boldsymbol{\beta}_{t})\,B_{t}^{[1]}}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\big(I-\left[\boldsymbol{\alpha}_{t}\right]\nabla^{2}f_{t}\big)\,\mathcal{H}_{t}^{[2]}\,-\,\left[\nabla f_{t}\right]\,\sigma^{\prime}(\boldsymbol{\beta}_{t})\,B_{t}^{[2]}}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\vdots}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\big(I-\left[\boldsymbol{\alpha}_{t}\right]\nabla^{2}f_{t}\big)\,\mathcal{H}_{t}^{[m]}\,-\,\left[\nabla f_{t}\right]\,\sigma^{\prime}(\boldsymbol{\beta}_{t})\,B_{t}^{[m]}}\\ \\ \end{array}\end{array}\right]. (52) Note that for any pair of same-size vectors 𝒂 \boldsymbol{a} and 𝒃 \boldsymbol{b} , we have [ 𝒂 ] ​ 𝒃 = [ 𝒃 ] ​ 𝒂 \left[\boldsymbol{a}\right]\boldsymbol{b}=\left[\boldsymbol{b}\right]\boldsymbol{a} where [ 𝒂 ] \left[\boldsymbol{a}\right] and [ 𝒃 ] \left[\boldsymbol{b}\right] are diagonal matrices of 𝒂 \boldsymbol{a} and 𝒃 \boldsymbol{b} , respectively. Therefore, ( 52 ) can be equivalently written in the following form 𝒉 t + 1 = γ ⁡ [ ( I − [ 𝜶 t ] ∇ 2 f t ) ℋ t [ 1 ] − [ σ ′ ( 𝜷 t ) B t [ 1 ] ] ∇ f t ⋮ ( I − [ 𝜶 t ] ∇ 2 f t ) ℋ t [ m ] − [ σ ′ ( 𝜷 t ) B t [ m ] ] ∇ f t ] . \boldsymbol{h}_{t+1}\,=\,\gamma\left[\begin{array}[]{c}\hskip-7.0pt\begin{array}[]{c}\\ {\big(I-\left[\boldsymbol{\alpha}_{t}\right]\nabla^{2}f_{t}\big)\,\mathcal{H}_{t}^{[1]}\,-\,\big[\sigma^{\prime}(\boldsymbol{\beta}_{t})\,B_{t}^{[1]}\big]\,\nabla f_{t}}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\vdots}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\big(I-\left[\boldsymbol{\alpha}_{t}\right]\nabla^{2}f_{t}\big)\,\mathcal{H}_{t}^{[m]}\,-\,\big[\sigma^{\prime}(\boldsymbol{\beta}_{t})\,B_{t}^{[m]}\big]\,\nabla f_{t}}\\ \\ \end{array}\end{array}\right]. (53)