MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters — part 8/10
Cleaned-text code points [92232, 101927). Bibliography omitted here; source.md includes it.
We proceed to compute the terms in the last row of the G t base G^{\textrm{base}}_{t} above. Consider the following block representation of X t X_{t} , X t = [ ℋ t X t m X t v ] , X_{t}=\left[\begin{array}[]{c}\mathcal{H}_{t}\\ \hfill X^{m}_{t}\\ X^{v}_{t}\end{array}\right], (60) Plugging the first row of G t base G^{\textrm{base}}_{t} , given in ( 59 ), into ( 46 ), implies that ℋ t + 1 = − γ μ t [ 𝒎 t 𝒗 t + κ 𝒘 t ] σ ′ ( 𝜷 t ) B t + γ ( I − κ [ 𝜶 t ] ) ℋ t − γ μ t [ 𝜶 t 𝒗 t ] X t m + γ μ t 2 [ 𝜶 t 𝒎 t 𝒗 t 1.5 ] X t v . \mathcal{H}_{t+1}=-\gamma\mu_{t}\Big[\frac{\boldsymbol{m}_{t}}{\sqrt{\boldsymbol{v}_{t}}}+\kappa\boldsymbol{w}_{t}\Big]\sigma^{\prime}(\boldsymbol{\beta}_{t})B_{t}\,+\,\gamma\big(I-\kappa\left[\boldsymbol{\alpha}_{t}\right]\big)\mathcal{H}_{t}\,-\,\gamma\mu_{t}\Big[\frac{\boldsymbol{\alpha}_{t}}{\sqrt{\boldsymbol{v}_{t}}}\Big]\,X^{m}_{t}\,+\,\gamma\frac{\mu_{t}}{2}\Big[\frac{\boldsymbol{\alpha}_{t}\boldsymbol{m}_{t}}{\boldsymbol{v}_{t}^{1.5}}\Big]\,X^{v}_{t}. (61) for all t ≥ 0 t\geq 0 . Note that for any pair of same-size vectors 𝒂 \boldsymbol{a} and 𝒃 \boldsymbol{b} , we have [ 𝒂 ] 𝒃 = [ 𝒃 ] 𝒂 \left[\boldsymbol{a}\right]\boldsymbol{b}=\left[\boldsymbol{b}\right]\boldsymbol{a} where [ 𝒂 ] \left[\boldsymbol{a}\right] and [ 𝒃 ] \left[\boldsymbol{b}\right] are diagonal matrices of 𝒂 \boldsymbol{a} and 𝒃 \boldsymbol{b} , respectively. Therefore, the i i th column in the matrix equation ( 61 ) can be equivalently written as ℋ t + 1 [ i ] = − γ μ t [ σ ′ ( 𝜷 t ) B t [ i ] ] 𝒎 t 𝒗 t + κ 𝒘 t + γ ( I − κ [ 𝜶 t ] ) ℋ t [ i ] − γ μ t [ X t m [ i ] ] 𝜶 t 𝒗 t + γ μ t 2 [ X t v [ i ] ] 𝜶 t 𝒎 t 𝒗 t 1.5 , \mathcal{H}^{[i]}_{t+1}=-\gamma\mu_{t}\Big[\sigma^{\prime}(\boldsymbol{\beta}_{t})B^{[i]}_{t}\Big]\frac{\boldsymbol{m}_{t}}{\sqrt{\boldsymbol{v}_{t}}}+\kappa\boldsymbol{w}_{t}\,+\,\gamma\big(I-\kappa\left[\boldsymbol{\alpha}_{t}\right]\big)\mathcal{H}^{[i]}_{t}\,-\,\gamma\mu_{t}\Big[X^{m\,[i]}_{t}\Big]\,\frac{\boldsymbol{\alpha}_{t}}{\sqrt{\boldsymbol{v}_{t}}}\,+\,\gamma\frac{\mu_{t}}{2}\Big[X^{v\,[i]}_{t}\Big]\,\frac{\boldsymbol{\alpha}_{t}\boldsymbol{m}_{t}}{\boldsymbol{v}_{t}^{1.5}}, (62) where B t [ i ] B^{[i]}_{t} , ℋ t [ i ] \mathcal{H}^{[i]}_{t} , X t m [ i ] X^{m\,[i]}_{t} , and X t v [ i ] X^{v\,[i]}_{t} stand for the i i th columns of B t B_{t} , ℋ t \mathcal{H}_{t} , X t m X^{m}_{t} , and X t v X^{v}_{t} , respectively. Following similar arguments as in ( 47 ), it is easy to show that d X t m d 𝜷 t = d X t v d 𝜷 t = 0 , d X t m d 𝒘 t = d X t v d 𝒘 t = 0 , d X t m d 𝒎 t = d X t v d 𝒎 t = 0 , d X t m d 𝒗 t = d X t v d 𝒗 t = 0 , d X t m d 𝒉 t = d X t v d 𝒉 t = 0 . \begin{split}&\frac{\mathrm{d}\,X^{m}_{t}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}=\frac{\mathrm{d}\,X^{v}_{t}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}=0,\\ &\frac{\mathrm{d}\,X^{m}_{t}}{\mathrm{d}\,\boldsymbol{w}_{t}}=\frac{\mathrm{d}\,X^{v}_{t}}{\mathrm{d}\,\boldsymbol{w}_{t}}=0,\\ &\frac{\mathrm{d}\,X^{m}_{t}}{\mathrm{d}\,\boldsymbol{m}_{t}}=\frac{\mathrm{d}\,X^{v}_{t}}{\mathrm{d}\,\boldsymbol{m}_{t}}=0,\\ &\frac{\mathrm{d}\,X^{m}_{t}}{\mathrm{d}\,\boldsymbol{v}_{t}}=\frac{\mathrm{d}\,X^{v}_{t}}{\mathrm{d}\,\boldsymbol{v}_{t}}=0,\\ &\frac{\mathrm{d}\,X^{m}_{t}}{\mathrm{d}\,\boldsymbol{h}_{t}}=\frac{\mathrm{d}\,X^{v}_{t}}{\mathrm{d}\,\boldsymbol{h}_{t}}=0.\end{split} (63) Note that 𝒉 t \boldsymbol{h}_{t} is an n m nm -dimensional vector derived from stacking the columns of ℋ t \mathcal{H}_{t} . Therefore, we consider a block representation of 𝒉 t \boldsymbol{h}_{t} consisting of m m blocks, each of which corresponds to a column of ℋ t \mathcal{H}_{t} . By taking the derivative of ( 61 ) with respect to 𝒉 t \boldsymbol{h}_{t} , and using ( 63 ), we obtain d 𝒉 t + 1 d 𝒉 t = γ [ I − κ [ 𝜶 t ] 0 0 0 0 I − κ [ 𝜶 t ] 0 0 0 0 ⋱ 0 0 0 0 I − κ [ 𝜶 t ] ] ← 1 st ← 2 nd ⋮ ← m th . \frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}=\gamma\,\left[\begin{array}[]{c|c|c|c}\begin{array}[]{c}I-\kappa\left[\boldsymbol{\alpha}_{t}\right]\\ \\ \end{array}&0&0&0\\ \hline\cr 0&\begin{array}[]{c}\\ I-\kappa\left[\boldsymbol{\alpha}_{t}\right]\\ \\ \end{array}&0&0\\ \hline\cr 0&0&\begin{array}[]{c}\\ \ddots\\ \\ \end{array}&0\\ \hline\cr 0&0&0&\begin{array}[]{c}\\ I-\kappa\left[\boldsymbol{\alpha}_{t}\right]\end{array}\end{array}\right]\,\,\begin{array}[]{l}\begin{array}[]{c}{\color[rgb]{0.75,0.75,0.75}\leftarrow 1\textrm{st}}\\ \\ \end{array}\\ \begin{array}[]{c}\\ {\color[rgb]{0.75,0.75,0.75}\leftarrow 2\textrm{nd}}\\ \\ \end{array}\\ \begin{array}[]{c}\\ \,\,{\color[rgb]{0.75,0.75,0.75}\vdots}\\ \\ \end{array}\\ \begin{array}[]{c}\\ {\color[rgb]{0.75,0.75,0.75}\leftarrow m\textrm{th}}\end{array}\end{array}. (64) Let β t [ i ] \beta_{t}[i] and w t [ j ] w_{t}[j] denote the i i th and j j th entries of 𝜷 t \boldsymbol{\beta}_{t} and 𝒘 t \boldsymbol{w}_{t} , for i = 1 , … , m i=1,\ldots,m and j = 1 , … , n j=1,\ldots,n , respectively. Note that d 𝒉 t + 1 / d 𝜷 t {\mathrm{d}\,\boldsymbol{h}_{t+1}}/{\mathrm{d}\,\boldsymbol{\beta}_{t}} is a block matrix, in the form of an m × m m\times m array of n × 1 n\times 1 blocks, d 𝒉 t + 1 d 𝜷 t [ i , j ] = def d ℋ t + 1 [ i ] d β t [ j ] \frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}[i,j]\overset{\text{\tiny def}}{=}\frac{\mathrm{d}\,\mathcal{H}_{t+1}^{[i]}}{\mathrm{d}\,\beta_{t}[j]} , for i , j = 1 , … , m i,j=1,\ldots,m . It then follows from ( 61 ) and ( 63 ) that, for i , j = 1 , … , m i,j=1,\ldots,m , d 𝒉 t + 1 d 𝜷 t [ i , j ] = d ℋ t + 1 [ i ] d β t [ j ] = − γ μ t [ 𝒎 t 𝒗 t + κ 𝒘 t ] ( ∂ σ ′ ( 𝜷 t ) ∂ β t [ j ] ) B t [ i ] + γ ( I − κ [ d 𝜶 t d β t [ j ] ] ) ℋ t [ i ] − γ μ t [ 1 𝒗 t ] [ d 𝜶 t d β t [ j ] ] X t m [ i ] + γ μ t 2 [ 𝒎 t 𝒗 t 1.5 ] [ d 𝜶 t d β t [ j ] ] X t v [ i ] , \begin{split}\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}[i,j]=&\frac{\mathrm{d}\,\mathcal{H}_{t+1}^{[i]}}{\mathrm{d}\,\beta_{t}[j]}\\ =&-\gamma\mu_{t}\Big[\frac{\boldsymbol{m}_{t}}{\sqrt{\boldsymbol{v}_{t}}}+\kappa\boldsymbol{w}_{t}\Big]\left(\frac{\partial\,\sigma^{\prime}(\boldsymbol{\beta}_{t})}{\partial\,\beta_{t}[j]}\right)B^{[i]}_{t}\,+\,\gamma\left(I-\kappa\Big[\frac{\mathrm{d}\,\boldsymbol{\alpha}_{t}}{\mathrm{d}\,\beta_{t}[j]}\Big]\right)\mathcal{H}^{[i]}_{t}\\ \,&-\,\gamma\mu_{t}\Big[\frac{1}{\sqrt{\boldsymbol{v}_{t}}}\Big]\Big[\frac{\mathrm{d}\,\boldsymbol{\alpha}_{t}}{\mathrm{d}\,\beta_{t}[j]}\Big]\,X^{m\,[i]}_{t}\,+\,\gamma\frac{\mu_{t}}{2}\Big[\frac{\boldsymbol{m}_{t}}{\boldsymbol{v}_{t}^{1.5}}\Big]\,\Big[\frac{\mathrm{d}\,\boldsymbol{\alpha}_{t}}{\mathrm{d}\,\beta_{t}[j]}\Big]\,X^{v\,[i]}_{t},\end{split} (65) where ∂ ∂ β \frac{\partial}{\partial\beta} stands for the entry-wise partial derivative of a matrix with respect to a scalar variable β \beta .
In the same vein, it follows from ( 62 ) and ( 63 ) that d 𝒉 t + 1 d 𝒘 t = − γ μ t κ [ [ σ ′ ( 𝜷 t ) B t [ 1 ] ] ⋮ [ σ ′ ( 𝜷 t ) B t [ m ] ] ] , \frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}=-\gamma\mu_{t}\kappa\left[\begin{array}[]{c}\hskip-7.0pt\begin{array}[]{c}\\ {\Big[\sigma^{\prime}(\boldsymbol{\beta}_{t})B^{[1]}_{t}\Big]}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\vdots}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\Big[\sigma^{\prime}(\boldsymbol{\beta}_{t})B^{[m]}_{t}\Big]}\\ \\ \end{array}\end{array}\right], (66) d 𝒉 t + 1 d 𝒎 t = γ μ t [ [ 𝜶 t X t v [ 1 ] 2 𝒗 t 1.5 − σ ′ ( 𝜷 t ) B t [ 1 ] 𝒗 t ] ⋮ [ 𝜶 t X t v [ m ] 2 𝒗 t 1.5 − σ ′ ( 𝜷 t ) B t [ m ] 𝒗 t ] ] , \frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{m}_{t}}=\gamma\mu_{t}\left[\begin{array}[]{c}\hskip-7.0pt\begin{array}[]{c}\\ {\Big[\frac{\boldsymbol{\alpha}_{t}X^{v\,[1]}_{t}}{2\,\boldsymbol{v}_{t}^{1.5}}-\frac{\sigma^{\prime}(\boldsymbol{\beta}_{t})B^{[1]}_{t}}{\sqrt{\boldsymbol{v}_{t}}}\Big]}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\vdots}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\Big[\frac{\boldsymbol{\alpha}_{t}X^{v\,[m]}_{t}}{2\,\boldsymbol{v}_{t}^{1.5}}-\frac{\sigma^{\prime}(\boldsymbol{\beta}_{t})B^{[m]}_{t}}{\sqrt{\boldsymbol{v}_{t}}}\Big]}\\ \\ \end{array}\end{array}\right], (67) d 𝒉 t + 1 d 𝒗 t = γ μ t 2 [ [ 1 𝒗 t 1.5 ] [ ( σ ′ ( 𝜷 t ) B t [ 1 ] ) 𝒎 t + 𝜶 t X t m [ 1 ] − 3 𝜶 t 𝒎 t X t v [ 1 ] 2 𝒗 t ] ⋮ [ 1 𝒗 t 1.5 ] [ ( σ ′ ( 𝜷 t ) B t [ m ] ) 𝒎 t + 𝜶 t X t m [ m ] − 3 𝜶 t 𝒎 t X t v [ m ] 2 𝒗 t ] ] . \frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{v}_{t}}=\frac{\gamma\mu_{t}}{2}\left[\begin{array}[]{c}\hskip-7.0pt\begin{array}[]{c}\\ {\Big[\frac{1}{\boldsymbol{v}_{t}^{1.5}}\Big]\,\Big[\big(\sigma^{\prime}(\boldsymbol{\beta}_{t})B^{[1]}_{t}\big)\boldsymbol{m}_{t}+\boldsymbol{\alpha}_{t}\,X^{m\,[1]}_{t}\,-\,\frac{3\boldsymbol{\alpha}_{t}\boldsymbol{m}_{t}X^{v\,[1]}_{t}}{2\,\boldsymbol{v}_{t}}\Big]}\\ \\ \end{array}\par\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\vdots}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\Big[\frac{1}{\boldsymbol{v}_{t}^{1.5}}\Big]\,\Big[\big(\sigma^{\prime}(\boldsymbol{\beta}_{t})B^{[m]}_{t}\big)\boldsymbol{m}_{t}+\boldsymbol{\alpha}_{t}\,X^{m\,[m]}_{t}\,-\,\frac{3\boldsymbol{\alpha}_{t}\boldsymbol{m}_{t}X^{v\,[m]}_{t}}{2\,\boldsymbol{v}_{t}}\Big]}\\ \\ \end{array}\end{array}\right]. (68)
Finally, G t base G^{\textrm{base}}_{t} is obtained by plugging ( 64 ), ( 65 ), ( 66 ), ( 67 ), and ( 68 ) into ( 59 ).
#### A.3.2 Base Lion