Structured Scaling of AI Discovery Across Diverse Scientific Domains — part 10/12

Cleaned-text code points [140742, 156284). Bibliography omitted here; source.md includes it.

The two searches use the same mission boundaries, physical evaluator, and cost definition, but different prior information. Reference construction may use the historical encounter order and nominal epochs because its purpose is to map that trajectory family into the benchmark. The evolved program receives the broad mission windows and allowed flyby bodies, but not the mission identity, historical order or intermediate epochs, or the optimized reference trajectory and cost. This prevents direct transfer of the reference result. It does not establish independence from knowledge encoded in the pretrained language model: the numerical task specification may itself suggest a familiar encounter sequence.

#### I.5 How evolution changes mission-design search

To examine the search procedure, we replayed the archived initial and evolved programs with lightweight instrumentation. The wrapper records the ordered encounter sequences passed to the program’s internal trajectory evaluations, every Lambert-solver call and its branch parameters, and elapsed execution time, without changing the search logic. The traces are used only to characterize program behavior; the trajectory costs reported in the main text come from independent replay of the checkpoint solutions.

The evolved programs do not uniformly maximize search breadth ( Figure 9 ). They expand the set of encounter sequences substantially for Galileo and Cassini, retain a narrow sequence set for Mariner 10, and search fewer encounter sequences than the initial permutation generator for Voyager 2 and the prospective Jupiter case. Their code stages systematic and stochastic sequence generation, prunes or retains candidate sequences according to early costs, and redirects the remaining time toward promising sequences. Evolution therefore selects task-dependent search schedulers rather than simply making every search wider or deeper.

The middle panel captures a second discrete decision that remains after an encounter sequence has been selected. Each transfer leg may admit short- and long-path, prograde and retrograde, and multi-revolution Lambert solutions, and the useful combinations depend on the continuous encounter epochs. In each of the five tasks for which the initial program reaches this stage, the evolved program explores more branch types. Cassini nevertheless exhibits the complementary strategy: it evaluates many encounter sequences while using only two Lambert branch types. After this screening, the programs combine global exploration with local refinement of encounter epochs and maneuver variables. This division of effort avoids spending the full budget on either exhaustive sequence enumeration or repeated polishing of one transfer basin.

The right panel measures an engineering property of the complete search implementation. All valid evolved replays execute Lambert-solver calls at a higher observed rate than the initial program. The archived programs use caching and candidate pruning to avoid repeated work, while the measured rate also reflects solver choice and other program overhead. A higher throughput allows more transfer candidates to be tested within a wall-clock budget, but is not itself a measure of solution quality; this comparison is not an isolated speed ablation. It instead shows the computational capacity available to the evolved search strategy. Together with adaptive budget allocation, broad branch coverage where useful, and complementary global and local optimization, this capacity allows more effective use of finite computation.

The replay describes the behavior of the final executable programs, not the provenance of every heuristic they contain. Several programs include mission-specific sequence templates or incumbent solutions that may have been introduced during evolution or inferred from pretrained knowledge. The figure therefore supports claims about the resulting search policies, but does not by itself establish how independently each encounter-sequence heuristic was discovered.

### J GPU Kernel Optimization

##### Task definitions and benchmark settings.

TriMul follows the triangular multiplicative update operator used in AlphaFold3-class protein-structure models [ 1 , 140 ] , instantiated through the official GPUMode TriMul benchmark [ 49 ] .

The evaluated benchmark settings are summarized in Table 9 .

Asymmetric matrix multiplication follows the KernelBench Level 1 tail skinny matrix multiplication task [ 103 ] , extended with additional highly unbalanced shapes that arise in modern AI workloads. The evaluated benchmark settings are summarized in Table 10 .

Batched cumsum follows the KernelBench Level 1 cumsum task [ 103 ] , extended with additional LLM sampling-inspired shapes. The evaluated benchmark settings are summarized in Table 11 .

##### TriMul evaluation alignment.

Table 12 reports the TriMul comparison on H100 under Triton 3.4.0 and Triton 3.6.0. We report both versions because TriMul latency is noticeably affected by the Triton compiler version. Triton 3.4.0 is retained for comparability with prior AI-discovery system baselines, including TTT-Discover and K-Search. However, in our local evaluation, Triton 3.6.0 reproduces the public GPUMode leaderboard times more closely. We therefore use Triton 3.6.0 as the leaderboard-aligned setting for the main-text comparison, while reporting Triton 3.4.0 as an additional reference. Each local measurement is repeated three times and reported as mean ± \pm standard deviation. Under both Triton versions, the kernel discovered by SimpleTES achieves the lowest latency among all compared AI methods and public GPUMode Triton submissions.

##### Cross-device performance generalization.

Table 13 evaluates whether the TriMul kernel discovered on NVIDIA H200 transfers to other GPU architectures. The same Triton program is evaluated directly on NVIDIA A100, NVIDIA H100, and AMD MI300, without any platform-specific re-tuning or additional search. Despite being discovered only on H200, the kernel achieves lower latency than the compared AI-discovery system baselines and the top-3 GPUMode Triton submissions available for each target GPU.

### K Scaling Law Discovery

##### Overview.

Scaling law discovery [ 79 , 78 ] studies how machine learning performance changes with scale and aims to identify compact symbolic laws that extrapolate from small-scale experiments to larger regimes. In foundation-model development, such laws are used to predict quantities such as training loss, downstream error, or task-specific metrics from variables including model size, dataset size, vocabulary size, learning rate, batch size, and architectural choices. The central challenge is not merely to fit a curve, but to recover a concise analytic form whose structure is not known a priori and that generalizes across related experimental settings. Each trial consists of a set of input variables, a target quantity, and a control index identifying the experimental context, such as a model family, dataset, or domain. The control index makes it possible to search for a shared symbolic form across settings while allowing the coefficients to vary within each setting. This makes scaling law discovery especially suitable for SimpleTES : candidate laws can be represented as executable programs, evaluated automatically on held-out extrapolation performance, and compared across a large, structured design space involving symbolic form, asymptotic behavior, parameter sharing, and fitting procedures. Compared with standard regression, the emphasis here is on symbolic structure and extrapolation; compared with broader agentic ML benchmarks, the objective is not to engineer an end-to-end workflow, but to discover a compact law that transfers across settings and remains accurate in unseen scaling regimes. We formalize scaling law discovery as follows.

A canonical example is pretraining scaling, where the inputs are model size N N and dataset size D D , the target is training loss L L , and the objective is to discover a law of the form L ≈ f θ ​ ( N , D ) L\approx f_{\theta}(N,D) . The same framework also covers scaling with domain mixture, learning rate, batch size, and related variables. What distinguishes this task from ordinary regression is that success is measured primarily by extrapolation beyond the fitted regime, not by interpolation within it.

##### Experimental setting.

We report results on the four-task SLDBench subset: parallel (36 seen / 12 unseen) [ 26 ] , domain_mix (80 seen / 24 unseen) [ 163 ] , lr&bsz (2,702 seen / 117 unseen) [ 77 ] , and u_shape (389 seen / 127 unseen) [ 157 ] . Following the original SLDBench protocol, the unseen split is always constructed as an extrapolation test set by holding out the largest model sizes, compute regimes, or other extreme settings, rather than by using random interpolation-style splits. The execution environment also follows SLDBench: agents operate in a sandbox terminal with a minimal Python stack ( scikit-learn , pandas , and datasets ) and no network access, and must implement the discovered law and its parameter-fitting subroutine under the required function signature. Final performance is measured by test-set R 2 R^{2} , clipped to [ − 1 , 1 ] [-1,1] , on the hidden extrapolation split, where higher is better. We set C = 16 C=16 , L = 20 L=20 , and K = 16 K=16 for SimpleTES on this task.

To ensure a controlled comparison, our method uses the same initialization, task instruction, and evaluator as SLDAgent. In particular, the initial program is the same baseline program pair used by SLDAgent: a naive power-law scaling_law_func together with a standard BFGS-based fit_scaling_law optimizer. We also adopt the same task-specific instruction template, which asks the model to evolve both the symbolic expression and the fitting routine from this baseline, while emphasizing extrapolation accuracy, cross-setting generalization, parameter efficiency, and numerical/theoretical stability; the instruction additionally provides task context, function signatures, and data characteristics such as feature definitions and value ranges. The evaluator is likewise kept identical to the SLDAgent setting. This alignment is important because SLDAgent is itself an evolution-based method built on top of the OpenEvolve framework, using iterative mutation and evaluation of candidate programs; accordingly, our comparison isolates the effect of the evolutionary strategy rather than differences in initialization, prompting, or evaluation pipeline.

##### Results analysis.

Table 14 shows that scaling law discovery remains a strong discriminator of agent capabilities even on this reduced four-task subset. SimpleTES achieves the best overall average score of 0.674, outperforming the strongest baseline, SLDAgent with GPT-5, which attains 0.572 on the same subset. The tasks also reveal a useful spectrum of difficulty. parallel is close to saturated, with many methods reaching near-perfect extrapolation, while domain_mix is broadly tractable but still differentiates top-performing methods at the margin. By contrast, lr&bsz and especially u_shape remain challenging: many agents still obtain negative test R 2 R^{2} , indicating that symbolic forms that interpolate well in-range often fail to extrapolate reliably into the held-out regime. SimpleTES ’s advantage is most pronounced on these harder extrapolative settings, where it attains the best score on lr&bsz and the best score on u_shape , while also tying for best on parallel and domain_mix .

##### Case study.

The lr&bsz task is a particularly informative case study because it tests whether a discovered scaling law can support an actual decision, rather than merely fitting observed losses. The goal is to predict validation loss as a function of learning rate, batch size, dataset size, and model size, and then use the predicted surface to select a good hyperparameter configuration in an extrapolated regime. Because the held-out split consists of larger-scale configurations, success requires recovering the geometry of the loss basin beyond the observed range.

Our method discovers an explicit symbolic law for the full loss surface, L ^ SimpleTES ​ ( lr , bsz , D , N ) = \displaystyle\hat{L}_{\text{{SimpleTES} }}(\mathrm{lr},\mathrm{bsz},D,N)={} 6.567185 ​ lr − 0.0131 ​ bsz 0.0096 ​ D 0.0346 ​ N − 0.1173 \displaystyle 6.567185\,\mathrm{lr}^{-0.0131}\,\mathrm{bsz}^{0.0096}\,D^{0.0346}\,N^{-0.1173} + 0.408323 ​ lr 0.2807 ​ bsz − 0.4838 ​ D 0.0499 ​ N 0.0995 \displaystyle+0.408323\,\mathrm{lr}^{0.2807}\,\mathrm{bsz}^{-0.4838}\,D^{0.0499}\,N^{0.0995} + 26.840067 ​ lr − 0.0657 ​ bsz 0.0595 ​ D − 0.2274 ​ N 0.0474 \displaystyle+26.840067\,\mathrm{lr}^{-0.0657}\,\mathrm{bsz}^{0.0595}\,D^{-0.2274}\,N^{0.0474} + 66071022.857 ​ lr 0.0844 ​ bsz 1.8723 ​ D − 1.4189 ​ N − 0.0982 \displaystyle+66071022.857\,\mathrm{lr}^{0.0844}\,\mathrm{bsz}^{1.8723}\,D^{-1.4189}\,N^{-0.0982} + 11.145293 ​ lr 0.9783 ​ bsz 0.1254 ​ D − 0.4142 ​ N 0.4786 \displaystyle+11.145293\,\mathrm{lr}^{0.9783}\,\mathrm{bsz}^{0.1254}\,D^{-0.4142}\,N^{0.4786} − 0.1294720520 . \displaystyle-0.1294720520. Given a target regime ( D , N ) (D,N) , we evaluate this law on the full admissible hyperparameter grid 𝒢 \mathcal{G} and choose the point with the minimum predicted loss: ( lr † , bsz † ) = arg ⁡ min ( lr , bsz ) ∈ 𝒢 ​ L ^ SimpleTES ​ ( lr , bsz , D , N ) . (\mathrm{lr}^{\dagger},\mathrm{bsz}^{\dagger})=\arg\min_{(\mathrm{lr},\mathrm{bsz})\in\mathcal{G}}\hat{L}_{\text{{SimpleTES} }}(\mathrm{lr},\mathrm{bsz},D,N). This matches the actual deployment setting: we score every feasible grid point directly, rather than optimizing in a continuous space and then rounding back to the nearest grid point.

For the extrapolation example in Figure 5 e, corresponding to a 1 1 B-parameter model trained on 100 100 B tokens, this procedure selects the blue-star configuration ( 1.953 × 10 − 3 , 512 ) (1.953\times 10^{-3},512) . Its realized validation loss is 2.0774 2.0774 , while the true best grid point is the red-star configuration ( 1.381 × 10 − 3 , 512 ) (1.381\times 10^{-3},512) with loss 2.0762 2.0762 . Thus, the configuration selected by SimpleTES is only 0.058 % 0.058\% above the optimum. For comparison, SLDAgent selects the gray-star point ( 1.953 × 10 − 3 , 384 ) (1.953\times 10^{-3},384) , whose realized loss is 2.0776 2.0776 , or 0.067 % 0.067\% above the optimum. Although the numerical gap is small, SimpleTES more accurately identifies the optimal region of the extrapolated loss surface and lands slightly closer to the empirical optimum.

This comparison is meaningful because the three marked points all lie in the same narrow low-loss valley, so the main challenge is fine-grained recovery of the learning-rate/batch-size trade-off rather than coarse localization. Modeling the full loss surface, rather than only the optimum coordinates, provides richer supervision and makes the discovered law directly usable as a practical selector in unseen regimes.

More broadly, this example shows why high held-out R 2 R^{2} matters in practice: the value of a scaling law lies not only in predictive accuracy, but also in its ability to support extrapolative decisions. On the lr&bsz task, SimpleTES turns a symbolic loss law into a grid-level hyperparameter rule and selects a point that is essentially optimal on the true evaluation landscape.

### L Whole-brain neural-activity forecast on ZAPBench

##### Problem definition and benchmark design.