MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters — part 3/10

Cleaned-text code points [28204, 41599). Bibliography omitted here; source.md includes it.

The learned step-sizes reveal an interesting pattern (Fig. 4 ): • Task adaptation: MetaOptimize increases step-sizes immediately after task switches to enhance adaptation.

• Layer-wise behavior: In blockwise case, early-layer step-sizes decrease over time, indicating convergence to globally useful features, while last-layer step-sizes increase, reflecting the need to adapt to changing labels.

### 7.3 ImageNet dataset

We trained ResNet-18 with batch-size 256 on ImageNet ( Deng et al., 2009 ) . We compared MetaOptimize with scalar step-size against four state-of-the-art hyperparamter optimization algorithms, namely DoG ( Ivgi et al., 2023 ) , gdtuo ( Chandra et al., 2022 ) , Prodigy ( Mishchenko & Defazio, 2023 ) , and mechanic ( Cutkosky et al., 2024 ) , as well as AdamW and Lion baselines with fixed step-sizes, and AdamW with a well-tuned cosine decay learning rate scheduler with a 10k iterations warmup. Learning curves and complexity overheads are shown respectively in Fig. 6 and Table 1 , showcasing the advantage of MetaOptimize algorithms (learning curve of DoG is not depicted due to its relatively poor performance). Unlike CIFAR10, here the blockwise versions of MetaOptimize showed no improvement over the scalar versions. Refer to Appendix D for further details.

### 7.4 Language modeling

For language model experiments, we used the TinyStories dataset ( Eldan & Li, 2023 ) , a synthetic collection of brief stories designed for children aged 3 to 4. This dataset proves effective for training and evaluating language models that are significantly smaller than the current state-of-the-art, and capable of crafting stories that are not only fluent and coherent but also diverse.

We used the implementation in ( Karpathy, 2024 ) for training 15M parameter model with a batch size of 128 on the TinyStories dataset. Two combinations of Hessian-free MetaOptimize with scalar step sizes were tested against Lion and AdamW with well-tuned fixed step sizes, AdamW with a well-tuned cosine decay learning rate scheduler with 1k warmup iterations, and the four state-of-the-art step-size adaptation algorithms mentioned in the previous subsection. According to the learning curves, shown in Fig. 6 , MetaOptimize outperforms all baselines (with an initial delay due to small initial step-sizes), except for the well-tuned learning rate scheduler within 30k iterations.

### 7.5 Sensitivity analysis

Here, we briefly discussion the sensitivity of MetaOptimize to its meta-meta-parameters.

For the meta-stepsize η \eta in MetaOptimize, there is generally no need for tuning, and the default value η = 10 − 3 \eta=10^{-3} works universally well in stationary supervised learning. All experiments in this section used this default value with no sweeping required. The rationale for this choice is that when using Adam, Lion, or RMSProp for meta-updates, the absolute change in β \beta per iteration is approximately η × O ⁡ ( 1 ) ≃ 10 − 3 \eta\times O(1)\simeq 10^{-3} . Unless the current stepsize α \alpha is already near its optimal value, most β \beta updates will consistently move toward the optimal β \beta . Within 1,000 steps, β \beta can change by O ⁡ ( 1 ) O(1) , nearly doubling or halving α = exp ⁡ ( β ) \alpha=\exp(\beta) . Over 10,000 iterations, α \alpha can adjust to stepsizes that are e 10 > 20,000 e^{10}>20,000 times larger or smaller, allowing η ≃ 10 − 3 \eta\simeq 10^{-3} to efficiently track optimal stepsizes while minimizing unnecessary fluctuations in α \alpha .

Regarding the discount factor γ \gamma , we used the γ = 1 \gamma=1 in all stationary experiments and observed minimal sensitivity to γ \gamma for values γ ≥ 0.999 \gamma\geq 0.999 in a series of preliminary tests. However, performance begins to degrade with smaller values of γ \gamma . In the non-stationary CIFAR100, γ = 0.999 \gamma=0.999 performed slightly better than 1 1 .

## 8 Related Works

Automatic adaptation of step sizes, has been an important research topic in the literature of stochastic optimization. Several works aimed to remove the manual tuning of learning rates via adaptations of classical line search ( Rolinek & Martius, 2018 ; Vaswani et al., 2019 ; Paquette & Scheinberg, 2020 ; Kunstner et al., 2023 ) and Polyak step size ( Berrada et al., 2020 ; Loizou et al., 2021 ) , stochastic proximal methods ( Asi & Duchi, 2019 ) , stochastic quadratic approximation ( Schaul et al., 2013 ) , hyper-gradient descent ( Baydin et al., 2017 ) , nested hyper-gradient descent ( Chandra et al., 2022 ) , distance to a solution adaptation ( Ivgi et al., 2023 ; Defazio & Mishchenko, 2023 ; Mishchenko & Defazio, 2023 ) , and online convex learning ( Cutkosky et al., 2024 ) . A limitation of most of these methods is their potential underperformance when their meta-parameters are not optimally configured for specific problems ( Ivgi et al., 2023 ) . Moreover, the primary focus of most of these methods is on minimizing immediate loss rather than considering the long-term effects of step sizes on future loss.

Normalization techniques proposed over past few years, such as AdaGrad ( Duchi et al., 2011 ) , RMSProp, and Adam have significantly enhanced the training process. While these algorithms show promise in the stationary problems, these normalization techniques do not optimize effective step sizes and are prone to have sub-optimal performance especially in the continual learning settings ( Degris et al., 2024 ) .

An early practical step-size optimization method was the Incremental-Delta-Bar-Delta (IDBD) algorithm, introduced in ( Sutton, 1992 ) , which aimed to optimize the step-size vector to minimize a specific form of quadratic loss functions in a continual setting. This algorithm was later extended for neural networks in ( Xu et al., 2018 ; Donini et al., 2019 ) , and further adapted in ( Mahmood et al., 2012 ; Javed, 2020 ; Micaelli & Storkey, 2021 ) for different meta or base updates beyond SGD. However, the development of IDBD and its extensions included some implicit assumptions, notably overlooking the impact of step-size dynamics on the formulation of step-size update rules. These extensions are, in essence, special cases of the L-approximation within the MetaOptimize framework. The current work extends the IDBD research, significantly broadening the framework and establishing a solid basis for the derivations. IDBD and its extensions have been used in various machine learning tasks including independent component analysis ( Schraudolph & Giannakopoulos, 1999 ) , human motion tracking ( Kehl & Van Gool, 2006 ) , classification ( Koop, 2007 ) , and reinforcement learning ( Xu et al., 2018 ; Young et al., 2018 ; Javed et al., 2024 ) . Refer to ( Sutton, 2022 ) for a comprehensive history of step-size optimization.

Hypergradient Descent (HD) ( Baydin et al., 2017 ) adapts learning rates using immediate loss gradients. MADA ( Ozkara et al., 2024 ) extends HD by parameterizing a space of optimizers and navigating it via hypergradient descent. Both focus on short-term effects, whereas MetaOptimize introduces a discount factor γ \gamma to model long-term influences, generalizing HD as a special case when γ = 0 \gamma=0 .

A related line of work is gradient-based bilevel optimization, initially introduced by ( Bengio, 2000 ) and later expanded in ( Maclaurin et al., 2015 ; Pedregosa, 2016 ; Franceschi et al., 2018 ; Gao et al., 2022 ) . Recent advances, such as ( Lorraine et al., 2020 ) , enable the optimization of millions of hyperparameters. While bilevel optimization focuses on tuning hyperparameters to minimize validation loss through repeated full training runs of the base algorithm, MetaOptimize diverges significantly. Designed for continual learning, MetaOptimize optimizes meta-parameters on-the-fly during a single streaming run, without relying on validation loss. Instead, it minimizes online loss (or regret) directly, aligning with the continual learning framework where no validation or test sets exist, and data arrives sequentially.

Another relevant literature is learn to optimize (L2O), which aim to learn optimization strategies from data. Classical L2O methods such as ( Andrychowicz et al., 2016 ) train optimizers offline and deploy them unchanged. Later works, including ( Metz et al., 2020 ; Metz et al., 2022 ) , develop more effective or scalable architectures, often using neural networks to modulate optimizer behavior. While powerful, these methods typically lack the ability to adapt online. In contrast, MetaOptimize updates its meta-parameters continuously during training, which is advantageous in nonstationary or continual learning scenarios.

There is also a line of research on the so-called parameter-free optimization that aims to remove the need for step-size tuning with almost no knowledge of the problem properties. Most of these methods are primarily designed for stochastic convex optimization ( Luo & Schapire, 2015 ; Orabona & Pál, 2016 ) , while more recent ones ( Orabona & Tommasi, 2017 ; Ivgi et al., 2023 ) were applied to supervised learning tasks with small or moderate sample sizes.

## 9 Limitations and Future Works

Our work represents a step toward unlocking the potential of meta-parameter optimization, with substantial room for further exploration, some of which we outline here:

Hessian: We confined our experiments to Hessian-free methods for practicality, though Hessian-based algorithms could offer superior performance. These methods, however, face challenges requiring additional research. The Hessian matrix is notably noisy, impacting ℋ t + 1 \mathcal{H}_{t+1} multiplicatively, necessitating smoothing and clipping techniques. Additionally, the Hessian approximates the loss landscape’s curvature but fails to account for non-differentiable curvatures, such as those from ReLU unit breakpoints, significant at training’s end. From a computational perspective, developing low-complexity methods for approximate Hessian matrix products, especially for adjusting step-sizes at the layer and weight levels, is essential.

More accurate traces: As discussed in Section 3 , accuracy of the backward approximation ( 5 ) may degrade for larger values of the meta-stepsize η \eta . Eligibility traces in RL suffer from a similar problem, to resolve which more-sophisticated traces (e.g., Dutch traces) have been developed (see Chapter 11 of ( Sutton & Barto, 2018 ) ). Developing more accurate backward approximations for meta-parameter optimization can result in considerable improvements in performance and stability.

Blockwise step-sizes: While step sizes can vary much in granularity, our experiments focused on scalar and blockwise step-sizes. While increasing the number of step sizes is anticipated to enhance performance, our experimental findings in Section 7 reveal that this improvement is not consistent across the MetaOptimize approximations evaluated. Further investigation is needed in future research.

Other approximations: We explored a limited set of MetaOptimize’s possible approximations, leaving a comprehensive analysis of various approximations for future research.

Other meta-parameters: Our study was limited to differentiable meta-parameters, not covering discrete ones like batch size or network layer count. We also did not investigate several significant differentiable meta-parameters beyond step-sizes, deferring such exploration to future work.

Automatic Differentiation: While certain versions of MetaOptimize, such as the L-Approximation, could be implemented using standard automatic differentiation software, its applicability to the general case of MetaOptimize remains unclear. Unlike updates for 𝒘 \boldsymbol{w} and 𝜷 \boldsymbol{\beta} (base and meta parameters), the H H matrix lacks an explicit incremental formula that can be easily handled by automatic differentiation. For some versions of MetaOptimize, including the Hessian-free approximations used in our experiments, automatic differentiation is unnecessary, as meta updates do not require additional differentiation. Exploring the scope and applicability of automatic differentiation across different MetaOptimize instances is an interesting direction for future research.

Discount factor γ = 𝟏 \boldsymbol{\gamma=1} : Our backward formulation (Eq. ( 14 )) formally assumes γ < 1 \gamma<1 due to a normalization factor used in the definition of the surrogate gradient. For γ = 1 \gamma=1 , a simple workaround is to remove this scaling factor, which makes the derivation fully valid and consistent—matching our actual implementation and experiments. That said, the case γ = 1 \gamma=1 presents subtle theoretical differences, much like in reinforcement learning and dynamic programming where additional centering is often helpful. Adapting similar techniques for meta-optimization may yield benefits and is a promising direction for future work.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

## Appendix A Step-size Optimization for Different Choices of Base and Meta updates

In this appendix, we derive G t G_{t} defined in ( 9 ) for different choices of algorithms for base and meta updates, and propose corresponding step-size optimization algorithms.