Nested Learning: The Illusion of Deep Learning Architectures — part 14/14

Cleaned-text code points [196798, 211657). Bibliography omitted here; source.md includes it.

Revisiting the momentum term without using the chain rule in backpropagation, W ℓ t + 1 = W ℓ t + 𝒎 ℓ t + 1 \displaystyle{W_{\ell}}_{{}_{t+1}}={W_{\ell}}_{{}_{t}}+{\bm{m}_{\ell}}_{{}_{t+1}} 𝒎 ℓ t + 1 = α ℓ , t + 1 𝒎 ℓ t − η ℓ , t + 1 ∇ W ℓ t ℒ ( W ℓ t ; 𝒙 t + 1 ) , \displaystyle{\bm{m}_{\ell}}_{{}_{t+1}}=\alpha_{\ell,t+1}{\bm{m}_{\ell}}_{{}_{t}}-\eta_{\ell,t+1}\nabla_{{W_{\ell}}_{{}_{t}}}\mathcal{L}\left({W_{\ell}}_{{}_{t}};\bm{x}_{t+1}\right), (100) one can interpret the momentum as a key or value-less associative memory ( Behrouz et al., 2025b , see Section 5) , where the gradient terms ∇ W ℓ t ℒ ( W ℓ t ; 𝒙 t + 1 ) \nabla_{{W_{\ell}}_{{}_{t}}}\mathcal{L}\left({W_{\ell}}_{{}_{t}};\bm{x}_{t+1}\right) are compressed into the momentum. We expect from a powerful momentum term to perfectly memorize all the past gradients in the training process so it can better model the current update to the weights. To this end, one can start from defining a simple objective as: ℒ ~ t = ∑ i = 1 t ∥ 𝒎 ℓ t ⊙ 𝒈 ℓ i + 1 − 𝑷 ℓ t ∥ 2 2 + λ ℓ ∥ 𝒎 ℓ t ∥ 2 F , \displaystyle\tilde{\mathcal{L}}_{t}=\sum_{i=1}^{t}\|{\bm{m}_{\ell}}_{{}_{t}}\odot{\bm{g}_{\ell}}_{{}_{i+1}}-\bm{P}_{\ell_{t}}\|_{2}^{2}\>\>+\>\>\lambda_{\ell}\|{\bm{m}_{\ell}}_{{}_{t}}\|^{2}_{F}, (101) where 𝒈 ℓ t + 1 = − ∇ W ℓ t ℒ ( W ℓ t ; 𝒙 t + 1 ) {\bm{g}_{\ell}}_{{}_{t+1}}=-\nabla_{{W_{\ell}}_{{}_{t}}}\mathcal{L}\left({W_{\ell}}_{{}_{t}};\bm{x}_{t+1}\right) . This objective aims to find a momentum term that does not simply map the gradients to 1 (which also results in limited memory management in momentum), but it maps gradients to a global property of past data samples. The more expressive this global property is, the more accurately the momentum can incorporate the compressed information from past. Accordingly, the objective in Equation 101 admits an optimal associative memory that maps gradients to 𝑷 ℓ t + 1 \bm{P}_{\ell_{t+1}} as: 𝒎 ℓ , i ( t ) ∗ = [ ( 𝑯 ℓ , i ( t ) + λ ℓ ​ 𝑰 ) − 1 ] ⊙ ( 𝑴 ℓ , i ( t ) ) = [ ( 𝑯 ℓ , i ( t ) + λ ℓ ​ 𝑰 ) − 1 ] ⊙ 𝑴 ~ ℓ , i + 1 ( t ) ⊙ 𝑷 ℓ t , where \displaystyle{\bm{m}^{(t)}_{\ell,i}}^{*}=\left[\left(\bm{H}^{(t)}_{\ell,i}\>+\>\lambda_{\ell}\>\bm{I}\right)^{-1}\right]\odot\left(\bm{M}^{(t)}_{\ell,i}\right)=\left[\left(\bm{H}^{(t)}_{\ell,i}\>+\>\lambda_{\ell}\>\bm{I}\right)^{-1}\right]\odot\tilde{\bm{M}}^{(t)}_{\ell,i+1}\odot\bm{P}_{\ell_{t}},\qquad\text{where} 𝑴 ( t ) ℓ , i + 1 = 𝑴 ( t ) ℓ , i + β 1 𝒈 ℓ i + 1 ⊙ 𝑷 ℓ t = 𝑴 ~ ( t ) ℓ , i + 1 ⊙ 𝑷 ℓ t , \displaystyle\bm{M}^{(t)}_{\ell,i+1}=\bm{M}^{(t)}_{\ell,i}\>+\>\beta_{1}\>{\bm{g}_{\ell}}_{{}_{i+1}}\odot\bm{P}_{\ell_{t}}=\tilde{\bm{M}}^{(t)}_{\ell,i+1}\odot\bm{P}_{\ell_{t}}, 𝑴 ~ ( t ) ℓ , i + 1 = 𝑴 ( t ) ℓ , i + β 1 𝒈 ℓ i + 1 , \displaystyle\tilde{\bm{M}}^{(t)}_{\ell,i+1}=\bm{M}^{(t)}_{\ell,i}\>+\>\beta_{1}\>{\bm{g}_{\ell}}_{{}_{i+1}}, 𝑯 ( t ) ℓ , i + 1 = 𝑯 ( t ) ℓ , i + β 2 𝒈 ℓ i + 1 ⊙ 𝒈 ℓ i + 1 = 𝑯 ( t ) ℓ , i + β 2 𝒈 ℓ i + 1 2 . \displaystyle\bm{H}^{(t)}_{\ell,i+1}=\bm{H}^{(t)}_{\ell,i}+\beta_{2}\>{\bm{g}_{\ell}}_{{}_{i+1}}\odot{\bm{g}_{\ell}}_{{}_{i+1}}=\bm{H}^{(t)}_{\ell,i}+\beta_{2}\>{\bm{g}_{\ell}}_{{}_{i+1}}^{2}. (102) Given this solution, one can write the update step as: W ℓ i + 1 = W ℓ i − η t 𝒎 ℓ , i ( t ) ∗ = W ℓ i − η i [ ( 𝑯 ℓ , i ( t ) + λ ℓ 𝑰 ) − 1 ] ⊙ 𝑴 ~ ℓ , i + 1 ( t ) ⊙ 𝑷 ℓ t . \displaystyle{W_{\ell}}_{{}_{i+1}}={W_{\ell}}_{{}_{i}}-\eta_{t}\>{\bm{m}^{(t)}_{\ell,i}}^{*}={W_{\ell}}_{{}_{i}}-\eta_{i}\>\left[\left(\bm{H}^{(t)}_{\ell,i}\>+\>\lambda_{\ell}\>\bm{I}\right)^{-1}\right]\odot\tilde{\bm{M}}^{(t)}_{\ell,i+1}\odot\bm{P}_{\ell_{t}}. (103) We start with a simple case, where 𝑷 ℓ t \bm{P}_{\ell_{t}} is the summation of the square of past gradients: i.e., 𝑷 ℓ t = ∑ i = 1 t 𝒈 ℓ i + 1 2 \bm{P}_{\ell_{t}}=\sum_{i=1}^{t}\bm{g}_{\ell_{i+1}}^{2} . With the choice of λ → 0 \lambda\rightarrow 0 , Equation 103 recovers simple gradient descent with momentum. That is, let λ = 0 \lambda=0 , we have: W ℓ i + 1 = W ℓ i − η t 𝒎 ℓ , i ( t ) ∗ = W ℓ i − η i ( 𝑯 ℓ , i ( t ) ) − 1 ⊙ 𝑴 ~ ℓ , i + 1 ( t ) ⊙ 𝑷 ℓ t ⏟ 𝑯 ℓ , i ( t ) / β 2 = W ℓ i − η t β 2 𝑴 ~ ℓ , i + 1 ( t ) , \displaystyle{W_{\ell}}_{{}_{i+1}}={W_{\ell}}_{{}_{i}}-\eta_{t}\>{\bm{m}^{(t)}_{\ell,i}}^{*}={W_{\ell}}_{{}_{i}}-\eta_{i}\>\left(\bm{H}^{(t)}_{\ell,i}\right)^{-1}\odot\tilde{\bm{M}}^{(t)}_{\ell,i+1}\odot\underset{\bm{H}^{(t)}_{\ell,i}/\beta_{2}}{\underbrace{\bm{P}_{\ell_{t}}}}={W_{\ell}}_{{}_{i}}-\eta_{t}\beta_{2}\>\tilde{\bm{M}}^{(t)}_{\ell,i+1}, (104) which based on the definition of 𝑴 ~ ℓ , i + 1 ( t ) \tilde{\bm{M}}^{(t)}_{\ell,i+1} , this update rule is equivalent to gradient descent with momentum. Next, we explore a more sophisticated design choice, where we use 𝑷 ℓ t \bm{P}_{\ell_{t}} as the variance of data samples before token t + 1 t+1 . In this case, formally, 𝑷 ℓ t = ∑ i = 1 t 𝒈 ℓ i + 1 2 \bm{P}_{\ell_{t}}=\sqrt{\sum_{i=1}^{t}\bm{g}_{\ell_{i+1}}^{2}} and so the update rule is as follows: W ℓ i + 1 = W ℓ i − η t 𝒎 ℓ , i ( t ) ∗ = W ℓ i − η i [ ( 𝑯 ℓ , i ( t ) + λ ℓ 𝑰 ) − 1 ] ⊙ 𝑴 ~ ℓ , i + 1 ( t ) ⊙ 𝑷 ℓ t ⏟ 𝑯 ℓ , i ( t ) 1 2 / β 2 ≈ W ℓ i − η t β 2 𝑴 ~ ℓ , i ( t ) 𝑯 ℓ , i ( t ) 1 / 2 + ε \displaystyle{W_{\ell}}_{{}_{i+1}}={W_{\ell}}_{{}_{i}}-\eta_{t}\>{\bm{m}^{(t)}_{\ell,i}}^{*}={W_{\ell}}_{{}_{i}}-\eta_{i}\>\left[\left(\bm{H}^{(t)}_{\ell,i}\>+\>\lambda_{\ell}\>\bm{I}\right)^{-1}\right]\odot\tilde{\bm{M}}^{(t)}_{\ell,i+1}\odot\underset{\bm{H}^{(t)^{\frac{1}{2}}}_{\ell,i}/\sqrt{\beta_{2}}}{\underbrace{\bm{P}_{\ell_{t}}}}\approx{W_{\ell}}_{{}_{i}}-\frac{\eta_{t}}{\sqrt{\beta_{2}}}\>\frac{\tilde{\bm{M}}^{(t)}_{\ell,i}}{\bm{H}^{(t)^{1/2}}_{\ell,i}\>+\varepsilon} (105) which is equivalent to the popular Adam optimizer ( Kingma and Ba, 2014 ) . Therefore, Adam is an optimal associative memory given the L 2 L_{2} regression objective that is defined in Equation 101 . In fact, the update component of Adam at each state aims to learn a mapping between the gradients and their variance (as a global property of past data samples). One interesting point about this formulation is about the frequency of elements. Looking at the first and second momentum in Adam optimizer, the frequency of update for both of these memories are the same as both are updated after each sample. Furthermore, the computation of each of them can be done in parallel and so are independent. Accordingly, it is one of the few examples that two components without any internal gradient flow, are placed in the same frequency and so level (see Section 3.2 ).

Going beyond element-wise update, we reformulate Equation 101 with outer-product operation as: ℒ ~ t = ∑ i = 1 t ∥ 𝒎 ℓ t 𝒈 ℓ i + 1 − 𝑷 ℓ t ∥ 2 2 + λ ℓ ∥ 𝒎 ℓ t ∥ 2 F , \displaystyle\tilde{\mathcal{L}}_{t}=\sum_{i=1}^{t}\|{\bm{m}_{\ell}}_{{}_{t}}{\bm{g}_{\ell}}_{{}_{i+1}}-\bm{P}_{\ell_{t}}\|_{2}^{2}\>\>+\>\>\lambda_{\ell}\|{\bm{m}_{\ell}}_{{}_{t}}\|^{2}_{F}, (106) where 𝒈 ℓ t + 1 = − ∇ W ℓ t ℒ ( W ℓ t ; 𝒙 t + 1 ) {\bm{g}_{\ell}}_{{}_{t+1}}=-\nabla_{{W_{\ell}}_{{}_{t}}}\mathcal{L}\left({W_{\ell}}_{{}_{t}};\bm{x}_{t+1}\right) . Similar to Equation 102 , finding the optimal solution to the above objective is defined as: 𝒎 ℓ , i ( t ) ∗ = [ ( 𝑯 ℓ , i ( t ) + λ ℓ ​ 𝑰 ) − 1 ] ​ ( 𝑴 ℓ , i ( t ) ) = [ ( 𝑯 ℓ , i ( t ) + λ ℓ ​ 𝑰 ) − 1 ] ​ 𝑴 ~ ℓ , i + 1 ( t ) ​ 𝑷 ℓ t , where \displaystyle{\bm{m}^{(t)}_{\ell,i}}^{*}=\left[\left(\bm{H}^{(t)}_{\ell,i}\>+\>\lambda_{\ell}\>\bm{I}\right)^{-1}\right]\left(\bm{M}^{(t)}_{\ell,i}\right)=\left[\left(\bm{H}^{(t)}_{\ell,i}\>+\>\lambda_{\ell}\>\bm{I}\right)^{-1}\right]\tilde{\bm{M}}^{(t)}_{\ell,i+1}\bm{P}_{\ell_{t}},\qquad\text{where} (107) 𝑴 ( t ) ℓ , i + 1 = 𝑴 ( t ) ℓ , i + β 1 𝑷 ℓ t 𝒈 ℓ i + 1 ⊤ = 𝑷 ℓ t 𝑴 ~ ( t ) ℓ , i + 1 , \displaystyle\bm{M}^{(t)}_{\ell,i+1}=\bm{M}^{(t)}_{\ell,i}\>+\>\beta_{1}\>\bm{P}_{\ell_{t}}\>{\bm{g}_{\ell}}_{{}_{i+1}}^{\top}=\bm{P}_{\ell_{t}}\>\tilde{\bm{M}}^{(t)}_{\ell,i+1}, (108) 𝑴 ~ ( t ) ℓ , i + 1 = 𝑴 ( t ) ℓ , i + β 1 𝒈 ℓ i + 1 ⊤ , \displaystyle\tilde{\bm{M}}^{(t)}_{\ell,i+1}=\bm{M}^{(t)}_{\ell,i}\>+\>\beta_{1}\>{\bm{g}_{\ell}}_{{}_{i+1}}^{\top}, (109) 𝑯 ( t ) ℓ , i + 1 = 𝑯 ( t ) ℓ , i + β 2 𝒈 ℓ i + 1 𝒈 ℓ i + 1 ⊤ . \displaystyle\bm{H}^{(t)}_{\ell,i+1}=\bm{H}^{(t)}_{\ell,i}+\beta_{2}\>{\bm{g}_{\ell}}_{{}_{i+1}}{\bm{g}_{\ell}}_{{}_{i+1}}^{\top}. (110) With a similar choice as Adam, i.e., letting 𝑷 ℓ t \bm{P}_{\ell_{t}} be the variance of the gradients so far, 𝑷 ℓ t = ∑ i = 1 t 𝒈 ℓ i + 1 ​ 𝒈 ℓ i + 1 ⊤ \bm{P}_{\ell_{t}}=\sqrt{\sum_{i=1}^{t}\bm{g}_{\ell_{i+1}}\bm{g}_{\ell_{i+1}}^{\top}} , then the above solution can be simplified as: W ℓ i + 1 = W ℓ i − η t 𝒎 ℓ , i ( t ) ∗ = W ℓ i − η i [ ( 𝑯 ℓ , i ( t ) + λ ℓ 𝑰 ) − 1 ] 𝑷 ℓ t ⏟ 𝑯 ℓ , i ( t ) 1 2 / β 2 𝑴 ~ ℓ , i + 1 ( t ) ≈ W ℓ i − η t β 2 𝑯 ℓ , i ( t ) − 1 / 2 𝑴 ~ ℓ , i ( t ) \displaystyle{W_{\ell}}_{{}_{i+1}}={W_{\ell}}_{{}_{i}}-\eta_{t}\>{\bm{m}^{(t)}_{\ell,i}}^{*}={W_{\ell}}_{{}_{i}}-\eta_{i}\>\left[\left(\bm{H}^{(t)}_{\ell,i}\>+\>\lambda_{\ell}\>\bm{I}\right)^{-1}\right]\>\underset{\bm{H}^{(t)^{\frac{1}{2}}}_{\ell,i}/\sqrt{\beta_{2}}}{\underbrace{\bm{P}_{\ell_{t}}}}\>\tilde{\bm{M}}^{(t)}_{\ell,i+1}\approx{W_{\ell}}_{{}_{i}}-\frac{\eta_{t}}{\sqrt{\beta_{2}}}\>\bm{H}^{(t)^{-1/2}}_{\ell,i}\>\tilde{\bm{M}}^{(t)}_{\ell,i} (111) The above formulation is the AdaGrad with momentum ( Défossez et al., 2022 ) and so generalizes the AdaGrad ( Duchi et al., 2011 ) , i.e., when β 1 = 1 \beta_{1}=1 . Similarly, based on the connection of Adam optimizer ( Kingma and Ba, 2014 ) with other algorithms such as RMSProp ( Hinton et al., 2012 ) , SignSGD and its momentum-based variants ( Bernstein et al., 2018 ) , NAdam ( Dozat, 2016 ) , AMSGrad ( Reddi et al., 2016 ) , RAdam ( Liu et al., 2020 ) , and Lion ( Chen et al., 2023 ) , as well as considering AdaGrad’s connection with optimizers such as Shampoo ( Gupta et al., 2018 ) and Soap ( Vyas et al., 2025 ) –i.e., as the approximation of the preconditioning term–we can conclude that all these optimizers can be re-formulated as associative memory that aims to compress the gradients.

## Appendix C Delta Gradient Descent with Normalization

In Section 4.5 we discussed that gradient descent can be see as an associative memory, and be reformulated as: W t + 1 = arg ⁡ min W ​ ⟨ W ​ 𝒙 t , ∇ y t ℒ ​ ( W t , 𝒙 t ) ⟩ + 1 2 ​ η t ​ ‖ W − W t ‖ 2 2 , \displaystyle W_{t+1}=\arg\min_{W}\>\>\langle W\bm{x}_{t},\nabla_{y_{t}}\mathcal{L}(W_{t};\bm{x}_{t})\rangle+\frac{1}{2\eta_{t}}\>\|W-W_{t}\|^{2}_{2}, (112) where each step aims at learning the negative of the gradient direction. Due to the fact that this learning rule only use update terms that depends on the current gradient, we defined 𝐮 t = − ∇ y t ℒ ​ ( W t , 𝒙 t ) \mathbf{u}_{t}=-\nabla_{y_{t}}\mathcal{L}(W_{t};\bm{x}_{t}) , and extended the above process to Delta Gradient Descent with more expressive objective of ℒ 2 \mathcal{L}_{2} regression loss: W t + 1 = arg ⁡ min W ​ 1 2 ​ ‖ W ​ 𝒙 t − 𝐮 t ‖ 2 2 + 1 2 ​ η t ​ ‖ W − W t ‖ 2 2 . \displaystyle W_{t+1}=\arg\min_{W}\>\>\frac{1}{2}\|W\bm{x}_{t}-\mathbf{u}_{t}\|^{2}_{2}+\frac{1}{2\eta_{t}}\|W-W_{t}\|^{2}_{2}. (113) We assume that 𝒙 t \bm{x}_{t} is normalized (e.g. in normalized memory systems or in neural networks with normalization layers, ‖ 𝒙 t ‖ 2 = λ \|\bm{x}_{t}\|_{2}=\lambda ). Taking gradient to optimize the above objective, 2 ​ ( W t + 1 ​ 𝒙 t − ∇ y t ℒ ​ ( W t , 𝒙 t ) ) ​ 𝒙 t ⊤ + 2 ​ η t ​ ( W t + 1 − W t ) = 0 , \displaystyle 2(W_{t+1}\bm{x}_{t}-\nabla_{y_{t}}\mathcal{L}(W_{t},\bm{x}_{t}))\bm{x}_{t}^{\top}+2\eta_{t}(W_{t+1}-W_{t})=0, (114) which results in: W t + 1 ​ ( 𝒙 t ​ 𝒙 t ⊤ + η t ​ I ) = ∇ y t ℒ ​ ( W t , 𝒙 t ) ​ 𝒙 t ⊤ + η t ​ W t , \displaystyle W_{t+1}(\bm{x}_{t}\bm{x}_{t}^{\top}+\eta_{t}I)=\nabla_{y_{t}}\mathcal{L}(W_{t},\bm{x}_{t})\bm{x}_{t}^{\top}+\eta_{t}W_{t}, (115) ⇒ \displaystyle\Rightarrow\>\> W t + 1 = ( ∇ y t ℒ ​ ( W t , x t ) ​ x t ⊤ + η t ​ W t ) ​ ( 𝒙 t ​ 𝒙 t ⊤ + η t ​ I ) − 1 . \displaystyle W_{t+1}=(\nabla_{y_{t}}\mathcal{L}(W_{t},x_{t})x_{t}^{\top}+\eta_{t}W_{t})(\bm{x}_{t}\bm{x}_{t}^{\top}+\eta_{t}I)^{-1}. (116) To compute ( 𝒙 t ​ 𝒙 t ⊤ + η t ​ I ) − 1 (\bm{x}_{t}\bm{x}_{t}^{\top}+\eta_{t}I)^{-1} term with Sherman-Morrison lemma, we have: ( x t ​ x t ⊤ + η t ​ I ) − 1 = 1 η t ​ ( I − 1 λ 2 + η t ​ 𝒙 t ​ 𝒙 t ⊤ ) , \displaystyle(x_{t}x_{t}^{\top}+\eta_{t}I)^{-1}=\frac{1}{\eta_{t}}(I-\frac{1}{\lambda^{2}+\eta_{t}}\bm{x}_{t}\bm{x}_{t}^{\top}), (117) and so: W t + 1 = ( ∇ y t ℒ ​ ( W t , 𝒙 t ) ​ 𝒙 t ⊤ + η t ​ W t ) ​ 1 η t ​ ( I − 1 λ 2 + η t ​ 𝒙 t ​ 𝒙 t ⊤ ) \displaystyle W_{t+1}=(\nabla_{y_{t}}\mathcal{L}(W_{t},\bm{x}_{t})\bm{x}_{t}^{\top}+\eta_{t}W_{t})\frac{1}{\eta_{t}}(I-\frac{1}{\lambda^{2}+\eta_{t}}\bm{x}_{t}\bm{x}_{t}^{\top}) (118) ⇒ \displaystyle\Rightarrow\>\> W t + 1 = W t ​ ( I − 1 λ 2 + η t ​ 𝒙 t ​ 𝒙 t ⊤ ) + 1 η t ​ ∇ y t ℒ ​ ( W t , 𝒙 t ) ​ 𝒙 t ⊤ − 1 λ 2 ​ η t + η t 2 ​ ∇ y t ℒ ​ ( W t , 𝒙 t ) ​ 𝒙 t ⊤ ​ 𝒙 t ​ 𝒙 t ⊤ ⏟ λ λ 2 ​ η t + η t 2 ​ ∇ y t ℒ ​ ( W t , 𝒙 t ) ​ 𝒙 t ⊤ \displaystyle W_{t+1}=W_{t}\left(I-\frac{1}{\lambda^{2}+\eta_{t}}\bm{x}_{t}\bm{x}_{t}^{\top}\right)+\frac{1}{\eta_{t}}\nabla_{y_{t}}\mathcal{L}(W_{t},\bm{x}_{t})\bm{x}_{t}^{\top}-\underset{\frac{\lambda}{\lambda^{2}\eta_{t}+\eta^{2}_{t}}\nabla_{y_{t}}\mathcal{L}(W_{t},\bm{x}_{t})\bm{x}_{t}^{\top}}{\underbrace{\frac{1}{\lambda^{2}\eta_{t}+\eta^{2}_{t}}\nabla_{y_{t}}\mathcal{L}(W_{t},\bm{x}_{t})\bm{x}_{t}^{\top}\bm{x}_{t}\bm{x}_{t}^{\top}}} (119) ⇒ \displaystyle\Rightarrow\>\> W t + 1 = W t ​ ( I − 1 λ 2 + η t ​ 𝒙 t ​ 𝒙 t ⊤ ) − ( λ λ 2 ​ η t + η t 2 − 1 η t ) ​ ∇ y t ℒ ​ ( W t , 𝒙 t ) ​ 𝒙 t ⊤ \displaystyle W_{t+1}=W_{t}\left(I-\frac{1}{\lambda^{2}+\eta_{t}}\bm{x}_{t}\bm{x}_{t}^{\top}\right)-\left(\frac{\lambda}{\lambda^{2}\eta_{t}+\eta^{2}_{t}}-\frac{1}{\eta_{t}}\right)\nabla_{y_{t}}\mathcal{L}(W_{t},\bm{x}_{t})\bm{x}_{t}^{\top} (120) ⇒ \displaystyle\Rightarrow\>\> W t + 1 = W t ​ ( I − α t ​ 𝒙 t ​ 𝒙 t ⊤ ) − β ​ ∇ y t ℒ ​ ( W t , 𝒙 t ) ​ 𝒙 t ⊤ \displaystyle W_{t+1}=W_{t}\left(I-\alpha_{t}\bm{x}_{t}\bm{x}_{t}^{\top}\right)-\beta\>\>\nabla_{y_{t}}\mathcal{L}(W_{t},\bm{x}_{t})\bm{x}_{t}^{\top} (121)

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .