MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters — part 9/10

Cleaned-text code points [101927, 116009). Bibliography omitted here; source.md includes it.

The lion algorithm, when used for base update, is as follows 𝒎 t + 1 = ρ 𝒎 t + ( 1 − ρ ) ∇ f t ( 𝒘 t ) , \displaystyle\boldsymbol{m}_{t+1}=\rho\,\boldsymbol{m}_{t}+(1-\rho)\,\nabla f_{t}(\boldsymbol{w}_{t}), (69) 𝒘 t + 1 = 𝒘 t − 𝜶 t Sign ( c 𝒎 t + ( 1 − c ) ∇ f t ) − κ 𝜶 t 𝒘 t , \displaystyle\boldsymbol{w}_{t+1}=\boldsymbol{w}_{t}-\boldsymbol{\alpha}_{t}\operatorname{Sign}\big(c\,\boldsymbol{m}_{t}+(1-c)\nabla f_{t}\big)-\kappa\boldsymbol{\alpha}_{t}\boldsymbol{w}_{t}, (70) where 𝒎 t \boldsymbol{m}_{t} is called the momentum, κ > 0 \kappa>0 is the weight-decay parameter, ρ , c ∈ [ 0 , 1 ) \rho,c\in[0,1) are constants, and Sign ⁡ ( ⋅ ) \operatorname{Sign}(\cdot) is a function that computes entry-wise sign of a vector. Let 𝒙 t = [ 𝒘 t 𝒎 t ] . \boldsymbol{x}_{t}=\left[\begin{array}[]{c}\boldsymbol{w}_{t}\\ \hfill\boldsymbol{m}_{t}\end{array}\right]. (71) It then follows from ( 44 ) that G t base = [ d ​ 𝒙 t + 1 d ​ 𝜷 t 0 d ​ 𝒙 t + 1 d ​ 𝒙 t 0 d ​ 𝒉 t + 1 d ​ 𝜷 t 0 d ​ 𝒉 t + 1 d ​ 𝒙 t d ​ 𝒉 t + 1 d ​ 𝒉 t ] = [ d ​ 𝒘 t + 1 d ​ 𝜷 t 0 d ​ 𝒘 t + 1 d ​ 𝒘 t d ​ 𝒘 t + 1 d ​ 𝒎 t 0 d ​ 𝒎 t + 1 d ​ 𝜷 t 0 d ​ 𝒎 t + 1 d ​ 𝒘 t d ​ 𝒎 t + 1 d ​ 𝒎 t 0 d ​ 𝒉 t + 1 d ​ 𝜷 t 0 d ​ 𝒉 t + 1 d ​ 𝒘 t d ​ 𝒉 t + 1 d ​ 𝒎 t d ​ 𝒉 t + 1 d ​ 𝒉 t ] = [ − [ Sign ( c 𝒎 t + ( 1 − c ) ∇ f t ) + κ 𝒘 t ] σ ′ ( 𝜷 t ) 0 I − κ ⁡ [ 𝜶 t ] 0 0 d ​ 𝒎 t + 1 d ​ 𝜷 t 0 d ​ 𝒎 t + 1 d ​ 𝒘 t d ​ 𝒎 t + 1 d ​ 𝒎 t 0 d ​ 𝒉 t + 1 d ​ 𝜷 t 0 d ​ 𝒉 t + 1 d ​ 𝒘 t d ​ 𝒉 t + 1 d ​ 𝒎 t d ​ 𝒉 t + 1 d ​ 𝒉 t ] \begin{split}G^{\textrm{base}}_{t}&=\left[\begin{array}[]{cccc}\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{x}_{t+1}}{\mathrm{d}\,\boldsymbol{x}_{t}}}\\ \\ \end{array}&0\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{x}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right]\\ &=\left[\begin{array}[]{ccccc}\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{w}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{w}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&\frac{\mathrm{d}\,\boldsymbol{w}_{t+1}}{\mathrm{d}\,\boldsymbol{m}_{t}}&0\\ \frac{\mathrm{d}\,\boldsymbol{m}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}&0&\frac{\mathrm{d}\,\boldsymbol{m}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}&\frac{\mathrm{d}\,\boldsymbol{m}_{t+1}}{\mathrm{d}\,\boldsymbol{m}_{t}}&0\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{m}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right]\\ &=\left[\begin{array}[]{ccccc}-\hskip-7.0pt\begin{array}[]{c}\\ {\big[\operatorname{Sign}\big(c\,\boldsymbol{m}_{t}+(1-c)\nabla f_{t}\big)+\kappa\boldsymbol{w}_{t}\big]\sigma^{\prime}(\boldsymbol{\beta}_{t})}\\ \\ \end{array}&\,0&\hskip-7.0pt\begin{array}[]{c}\\ {I-\kappa\left[\boldsymbol{\alpha}_{t}\right]}\\ \\ \end{array}&0&0\\ \frac{\mathrm{d}\,\boldsymbol{m}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}&0&\frac{\mathrm{d}\,\boldsymbol{m}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}&\frac{\mathrm{d}\,\boldsymbol{m}_{t+1}}{\mathrm{d}\,\boldsymbol{m}_{t}}&0\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&0&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{m}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\right]\end{split} (72) where the second equality is due to ( 71 ) and the last equality follows from ( 70 ). Consider the following block representation of X t X_{t} , X t = [ ℋ t X t m ] . X_{t}=\left[\begin{array}[]{c}\mathcal{H}_{t}\\ \hfill X^{m}_{t}\end{array}\right]. (73) Plugging the first row of G t base G^{\textrm{base}}_{t} , given in ( 72 ), into ( 46 ), implies that ℋ t + 1 = − γ [ Sign ( c 𝒎 t + ( 1 − c ) ∇ f t ) + κ 𝒘 t ] σ ′ ( 𝜷 t ) B t + γ ( I − κ [ 𝜶 t ] ) ℋ t \mathcal{H}_{t+1}=-\gamma\big[\operatorname{Sign}\big(c\,\boldsymbol{m}_{t}+(1-c)\nabla f_{t}\big)+\kappa\boldsymbol{w}_{t}\big]\sigma^{\prime}(\boldsymbol{\beta}_{t})\,B_{t}\,+\,\gamma\big(I-\kappa\left[\boldsymbol{\alpha}_{t}\right]\big)\mathcal{H}_{t} (74) For simplicity of notation, we define the diagonal matrix S t S_{t} as S t = def [ Sign ( c 𝒎 t + ( 1 − c ) ∇ f t ) + κ 𝒘 t ] . S_{t}\overset{\text{\tiny def}}{=}\big[\operatorname{Sign}\big(c\,\boldsymbol{m}_{t}+(1-c)\nabla f_{t}\big)+\kappa\boldsymbol{w}_{t}\big]. (75) Then, 𝒉 t + 1 = γ ⁡ [ − S t ​ σ ′ ​ ( 𝜷 t ) ​ B t [ 1 ] + γ ⁡ ( I − κ ⁡ [ 𝜶 t ] ) ​ ℋ t [ 1 ] ⋮ − S t ​ σ ′ ​ ( 𝜷 t ) ​ B t [ m ] + γ ⁡ ( I − κ ⁡ [ 𝜶 t ] ) ​ ℋ t [ m ] . ] \boldsymbol{h}_{t+1}=\gamma\left[\begin{array}[]{c}\hskip-7.0pt\begin{array}[]{c}\\ {-S_{t}\,\sigma^{\prime}(\boldsymbol{\beta}_{t})\,B_{t}^{[1]}\,+\,\gamma\big(I-\kappa\left[\boldsymbol{\alpha}_{t}\right]\big)\mathcal{H}_{t}^{[1]}}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\vdots}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {-S_{t}\,\sigma^{\prime}(\boldsymbol{\beta}_{t})\,B_{t}^{[m]}\,+\,\gamma\big(I-\kappa\left[\boldsymbol{\alpha}_{t}\right]\big)\mathcal{H}_{t}^{[m]}}\\ \\ \end{array}\hskip-15.0pt\phantom{a}.\end{array}\right] (76) It follows that d ​ 𝒉 t + 1 d ​ 𝒎 t = 0 , \frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{m}_{t}}=0, (77) and d ​ 𝒉 t + 1 d ​ 𝒘 t = − γ ⁡ [ [ 𝒆 1 ] ​ σ ′ ​ ( 𝜷 t ) ​ B t [ 1 ] ⋯ [ 𝒆 n ] ​ σ ′ ​ ( 𝜷 t ) ​ B t [ 1 ] ⋮ ⋱ ⋮ [ 𝒆 1 ] ​ σ ′ ​ ( 𝜷 t ) ​ B t [ m ] ⋯ [ 𝒆 n ] ​ σ ′ ​ ( 𝜷 t ) ​ B t [ m ] ] , \frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}=-\gamma\left[\begin{array}[]{c|c|c}\hskip-7.0pt\begin{array}[]{c}\\ {\left[\boldsymbol{e}_{1}\right]\,\sigma^{\prime}(\boldsymbol{\beta}_{t})\,B_{t}^{[1]}}\\ \\ \end{array}&\cdots&\hskip-7.0pt\begin{array}[]{c}\\ {\left[\boldsymbol{e}_{n}\right]\,\sigma^{\prime}(\boldsymbol{\beta}_{t})\,B_{t}^{[1]}}\\ \\ \end{array}\\ \hline\cr\vdots&\hskip-7.0pt\begin{array}[]{c}\\ {\ddots}\\ \\ \end{array}&\vdots\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\left[\boldsymbol{e}_{1}\right]\,\sigma^{\prime}(\boldsymbol{\beta}_{t})\,B_{t}^{[m]}}\\ \\ \end{array}&\cdots&\hskip-7.0pt\begin{array}[]{c}\\ {\left[\boldsymbol{e}_{n}\right]\,\sigma^{\prime}(\boldsymbol{\beta}_{t})\,B_{t}^{[m]}}\\ \\ \end{array}\end{array}\right], (78) where 𝒆 i \boldsymbol{e}_{i} is the i i th unit vector (i.e., an n n -dimensional vector whose i i th entry is 1 1 and all other entries are zero). Let β t ​ [ i ] \beta_{t}[i] and ℋ t [ i ] \mathcal{H}_{t}^{[i]} be the i i th entry of 𝜷 t \boldsymbol{\beta}_{t} and i i th column of ℋ t \mathcal{H}_{t} , respectively, for i = 1 , … , m i=1,\ldots,m . Then, d ​ 𝒉 t + 1 d ​ 𝜷 t = − γ ⁡ [ γ ​ κ ​ [ d ​ 𝜶 t d ​ β t ​ [ 1 ] ] ​ ℋ t [ 1 ] + S t ​ ∂ σ ′ ​ ( 𝜷 t ) ∂ β t ​ [ 1 ] ​ B t [ 1 ] ⋯ γ ​ κ ​ [ d ​ 𝜶 t d ​ β t ​ [ m ] ] ​ ℋ t [ 1 ] + S t ​ ∂ σ ′ ​ ( 𝜷 t ) ∂ β t ​ [ m ] ​ B t [ 1 ] ⋮ ⋱ ⋮ γ ​ κ ​ [ d ​ 𝜶 t d ​ β t ​ [ 1 ] ] ​ ℋ t [ m ] + S t ​ ∂ σ ′ ​ ( 𝜷 t ) ∂ β t ​ [ 1 ] ​ B t [ m ] ⋯ γ ​ κ ​ [ d ​ 𝜶 t d ​ β t ​ [ m ] ] ​ ℋ t [ m ] + S t ​ ∂ σ ′ ​ ( 𝜷 t ) ∂ β t ​ [ m ] ​ B t [ m ] ] , \frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}=-\gamma\left[\begin{array}[]{c|c|c}\hskip-7.0pt\begin{array}[]{c}\\ {\gamma\kappa\Big[\frac{\mathrm{d}\,\boldsymbol{\alpha}_{t}}{\mathrm{d}\,\beta_{t}[1]}\Big]\,\mathcal{H}_{t}^{[1]}\,+\,S_{t}\,\frac{\partial\,\sigma^{\prime}(\boldsymbol{\beta}_{t})}{\partial\,\beta_{t}[1]}\,B_{t}^{[1]}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\quad\cdots\quad}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\gamma\kappa\Big[\frac{\mathrm{d}\,\boldsymbol{\alpha}_{t}}{\mathrm{d}\,\beta_{t}[m]}\Big]\,\mathcal{H}_{t}^{[1]}\,+\,S_{t}\,\frac{\partial\,\sigma^{\prime}(\boldsymbol{\beta}_{t})}{\partial\,\beta_{t}[{m}]}\,B_{t}^{[1]}}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\vdots}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\ddots}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\vdots}\\ \\ \end{array}\\ \hline\cr\hskip-7.0pt\begin{array}[]{c}\\ {\gamma\kappa\Big[\frac{\mathrm{d}\,\boldsymbol{\alpha}_{t}}{\mathrm{d}\,\beta_{t}[1]}\Big]\,\mathcal{H}_{t}^{[m]}\,+\,S_{t}\,\frac{\partial\,\sigma^{\prime}(\boldsymbol{\beta}_{t})}{\partial\,\beta_{t}[1]}\,B_{t}^{[m]}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\cdots}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\gamma\kappa\Big[\frac{\mathrm{d}\,\boldsymbol{\alpha}_{t}}{\mathrm{d}\,\beta_{t}[m]}\Big]\,\mathcal{H}_{t}^{[m]}\,+\,S_{t}\,\frac{\partial\,\sigma^{\prime}(\boldsymbol{\beta}_{t})}{\partial\,\beta_{t}[{m}]}\,B_{t}^{[m]}}\\ \\ \end{array}\end{array}\right], (79) and d ​ 𝒉 t + 1 d ​ 𝒉 t = γ ⁡ [ I − κ ⁡ [ 𝜶 t ] 0 0 0 0 I − κ ⁡ [ 𝜶 t ] 0 0 0 0 ⋱ 0 0 0 0 I − κ ⁡ [ 𝜶 t ] ] ​ ← 1 ​ st ← 2 ​ nd ⋮ ← m ​ th , . \frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}=\gamma\,\left[\begin{array}[]{c|c|c|c}\begin{array}[]{c}I-\kappa\left[\boldsymbol{\alpha}_{t}\right]\\ \\ \end{array}&0&0&0\\ \hline\cr 0&\begin{array}[]{c}\\ I-\kappa\left[\boldsymbol{\alpha}_{t}\right]\\ \\ \end{array}&0&0\\ \hline\cr 0&0&\begin{array}[]{c}\\ \ddots\\ \\ \end{array}&0\\ \hline\cr 0&0&0&\begin{array}[]{c}\\ I-\kappa\left[\boldsymbol{\alpha}_{t}\right]\end{array}\end{array}\right]\,\,\begin{array}[]{l}\begin{array}[]{c}{\color[rgb]{0.75,0.75,0.75}\leftarrow 1\textrm{st}}\\ \\ \end{array}\\ \begin{array}[]{c}\\ {\color[rgb]{0.75,0.75,0.75}\leftarrow 2\textrm{nd}}\\ \\ \end{array}\\ \begin{array}[]{c}\\ \,\,{\color[rgb]{0.75,0.75,0.75}\vdots}\\ \\ \end{array}\\ \begin{array}[]{c}\\ {\color[rgb]{0.75,0.75,0.75}\leftarrow m\textrm{th}}\end{array},\end{array}. (80)

It follows from ( 21 ), ( 72 ), and ( 77 ) that in the G t G_{t} matrix, d ​ 𝒎 t + 1 d ​ 𝒎 t \frac{\mathrm{d}\,\boldsymbol{m}_{t+1}}{\mathrm{d}\,\boldsymbol{m}_{t}} is the only non-zero block in its corresponding column of blocks. Consequently, it follows from ( 14 ) that X t m X^{m}_{t} , as defined in ( 73 ), has no impact on the update of ℋ t + 1 \mathcal{H}_{t+1} , Y t + 1 Y_{t+1} , and Q t + 1 Q_{t+1} . Therefore, the rows and columns of G base G^{\textrm{base}} that correspond to derivative of 𝒎 \boldsymbol{m} can be completely removed from G base G^{\textrm{base}} . By removing these rows and columns from G t G^{t} , the matrix update ( 14 ) simplifies to [ Y t + 1 ℋ t + 1 Q t + 1 ] = γ ​ [ d ​ 𝒚 t + 1 d ​ 𝒚 t [ − [ Sign ( c 𝒎 t + ( 1 − c ) ∇ f t ) ] σ ′ ( 𝜷 t ) 0 d ​ 𝒉 t + 1 d ​ 𝜷 t 0 ] d ​ 𝒚 t + 1 d ​ 𝒘 t d ​ 𝒚 t + 1 d ​ 𝒉 t I − κ ⁡ [ 𝜶 t ] 0 d ​ 𝒉 t + 1 d ​ 𝒘 t d ​ 𝒉 t + 1 d ​ 𝒉 t ] ​ ( [ Y t ℋ t Q t ] + ( 1 − γ ) ​ [ [ I 0 ] 0 0 ] ) , \left[\begin{array}[]{c}Y_{t+1}\\ \mathcal{H}_{t+1}\\ Q_{t+1}\end{array}\right]=\gamma\,\left[\begin{array}[]{cc}\begin{array}[]{c}\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{y}_{t+1}}{\mathrm{d}\,\boldsymbol{y}_{t}}}\\ \\ \end{array}\\ \hskip-7.0pt\begin{array}[]{c}\\ {\left[\begin{array}[]{cc}\hskip-7.0pt\begin{array}[]{c}\\ {-\big[\operatorname{Sign}\big(c\,\boldsymbol{m}_{t}+(1-c)\nabla f_{t}\big)\big]\sigma^{\prime}(\boldsymbol{\beta}_{t})}\\ \\ \end{array}&\,\,\,\,\hskip-7.0pt\begin{array}[]{c}\\ {0}\\ \\ \end{array}\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{\beta}_{t}}}\\ \\ \end{array}&\,\,\,\,\hskip-7.0pt\begin{array}[]{c}\\ {0}\\ \\ \end{array}\end{array}\right]}\\ \\ \end{array}\end{array}&\begin{array}[]{cc}\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{y}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{y}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\\ \hskip-7.0pt\begin{array}[]{c}\\ {I-\kappa\left[\boldsymbol{\alpha}_{t}\right]}\\ \\ \end{array}&0\\ \hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{w}_{t}}}\\ \\ \end{array}&\hskip-7.0pt\begin{array}[]{c}\\ {\frac{\mathrm{d}\,\boldsymbol{h}_{t+1}}{\mathrm{d}\,\boldsymbol{h}_{t}}}\\ \\ \end{array}\end{array}\end{array}\right]\,\left(\left[\begin{array}[]{c}Y_{t}\\ \mathcal{H}_{t}\\ Q_{t}\end{array}\right]+(1-\gamma)\left[\begin{array}[]{c}\hskip-10.0pt\begin{array}[]{c}\\ {\left[\begin{array}[]{c}I\\ 0\end{array}\right]}\\ \\ \end{array}\\ 0\\ 0\end{array}\right]\right), (81) where d ​ 𝒉 t + 1 / d ​ 𝜷 t {\mathrm{d}\,\boldsymbol{h}_{t+1}}/{\mathrm{d}\,\boldsymbol{\beta}_{t}} , d ​ 𝒉 t + 1 / d ​ 𝒘 t {\mathrm{d}\,\boldsymbol{h}_{t+1}}/{\mathrm{d}\,\boldsymbol{w}_{t}} , and d ​ 𝒉 t + 1 / d ​ 𝒉 t {\mathrm{d}\,\boldsymbol{h}_{t+1}}/{\mathrm{d}\,\boldsymbol{h}_{t}} are given in ( 79 ), ( 78 ), and ( 80 ), respectively; and the blocks in the first row depend on the meta update.

## Appendix B Exiting Step-size Optimization Algorithms as Special Cases of MetaOptimize

In this appendix we show that some of the existing step-size optimization algorithms are special cases of the MetaOptimize framework. In particular, we first consider the IDBD algorithm ( Sutton, 1992 ) and its extension ( Xu et al., 2018 ) , and then discuss about the HyperGradient algorithm ( Baydin et al., 2017 ) .

### B.1 IDBD and Its Extensions