Structured Scaling of AI Discovery Across Diverse Scientific Domains — part 3/12

Cleaned-text code points [31592, 47458). Bibliography omitted here; source.md includes it.

Two important foundations of modern AI development are building efficient GPU kernels that make models run fast on real hardware ( Figure 5 a), and discovering scaling laws that help predict how model performance changes as training is scaled up ( Figure 5 d). Both have traditionally relied on expert manual design: GPU kernels are carefully optimized by systems engineers for specific hardware architectures and software stacks, while scaling laws are manually specified and fitted by researchers to predict how performance changes with model size, data size, compute, and other scaling variables. We apply SimpleTES to both classes of problem and find that the same evaluation-driven search procedure produces kernels that surpass expert Triton submissions and AI kernel-discovery baselines on real hardware, and scaling laws that extrapolate substantially better than the strongest agent baselines on the SLDBench benchmark.

##### GPU kernel optimization.

We apply SimpleTES to optimize the GPU kernels that modern AI systems rely on. SimpleTES starts from a simple PyTorch reference implementation and searches for a faster GPU kernel that preserves numerical correctness within a task-specific tolerance. We evaluate this ability on three representative operators. The first is TriMul, a triangular multiplicative update used in AlphaFold3-class protein-structure models [ 1 , 140 ] and the subject of an active GPUMode optimization competition [ 49 ] . The second is asymmetric matrix multiplication, an irregular-shape GEMM pattern in modern AI workloads, including small-batch inference, LoRA adapters, and query-similarity search. The third is batched cumulative sum, a primitive in LLM sampling and stream-compaction pipelines. Across all three tasks, SimpleTES discovers kernels on an NVIDIA H200, and the resulting implementations are evaluated without re-tuning on the target GPUs. We compare against both expert-engineered kernels and recent AI-driven kernel-discovery systems baselines that achieve previous SOTA efficiency [ 165 , 20 , 14 , 29 ] .

On TriMul, a kernel discovered by SimpleTES on H200 transfers directly to H100 without re-tuning, reaching a geometric-mean latency of 1.122 ms, surpassing both the AI baseline of 1.154 ms in Figure 5 c and the strongest expert-written Triton baseline of 1.140 ms on the GPUMode leaderboard (Supplementary Table 12 ). The same implementation achieves the fastest runtime on NVIDIA A100, NVIDIA H100, and AMD MI300, suggesting that the search found a portable implementation strategy rather than an H200-specific optimization (Supplementary Table 13 ). Inspection of the discovered implementation shows that the gain comes from jointly optimizing high-level structure and low-level kernel tuning. SimpleTES fuses layer normalization, gating, projection, and masking into a small number of Triton kernels, selects a mixed-precision path in which activations move in FP16 while reductions remain in FP32, and tunes tile sizes within the kernels to improve hardware utilization ( Figure 5 b). The asymmetric-matmul and batched-cumsum tasks further show that SimpleTES can specialize the implementation strategy to the input shape, rather than relying on a one-size-fits-all template for each operator. On asymmetric matrix multiplication, the largest gains occur on highly uneven matrix shapes, where SimpleTES achieves up to 4.01 × 4.01\times speedup over the PyTorch FP32 reference and reduces runtime by 65.6% relative to CUDA Agent on the case with the largest improvement ( Figure 5 c). Batched cumulative sum provides a clearer example of shape-dependent implementation: the discovered kernel selects between single-tile and multi-tile execution paths according to workload size. This implementation outperforms NVIDIA’s hand-tuned CUB DeviceSegmentedScan on four of the six evaluated shapes and reaches 6.06 × 6.06\times speedup over the PyTorch reference. CUB remains only marginally faster on the two smallest shapes, where its fixed segmented-scan strategy is already well matched to the input. These case studies show that SimpleTES can automatically discover optimization strategies that are typically hand-engineered in optimized GPU libraries. Detailed shape definitions are provided in the Supplementary Tables 9 , 10 and 11 .

##### Scaling-law discovery.

We evaluate SimpleTES on four tasks from the SLDBench benchmark [ 79 , 78 ] , covering parallel scaling [ 26 ] , domain-mixture scaling [ 163 ] , learning-rate/batch-size co-scaling [ 77 ] , and U-shape compute scaling [ 157 ] . The task is to discover a symbolic law f θ f_{\theta} together with a per-setting fitting procedure that extrapolates accurately to held-out larger-scale configurations, where each configuration specifies the scale variables and task-specific control variables used for fitting and evaluation. Success is measured by the extrapolation coefficient of determination ( R 2 R^{2} ) on each task’s extrapolation split. To isolate the effect of the search strategy, we use the same initialization, task instructions, and evaluator as the SLDAgent baseline [ 79 ] : the initial program is a naive power-law function paired with a BFGS optimizer. Across the four tasks ( Figure 5 d, Supplementary Table 14 ), SimpleTES achieves the best average R 2 R^{2} of 0.674, exceeding the strongest baseline (SLDAgent with GPT-5, 0.572) and every other configuration tested. The advantage is sharpest on the hardest extrapolative settings: SimpleTES achieves the highest R 2 R^{2} on the U-shape compute-scaling task ( u_shape ; − 0.008 -0.008 , versus − 0.232 -0.232 for the next-best baseline) and on the learning-rate/batch-size co-scaling task ( lr&bsz ; 0.712 versus 0.611 for SLDAgent+o4-mini), where most agents obtain negative test-set R 2 R^{2} and fail to extrapolate. The discovered law on lr&bsz produces a hyperparameter recommendation only 0.058% above the true optimum at a 1B-parameter, 100B-token target setting, corresponding to a 10 × \times extrapolation in model size and a 100 × \times extrapolation in data size (Supplementary K ).

These two task families share an interface—an initial solution and a queryable evaluator—and differ only in what the evaluator measures (wall-clock runtime under a correctness tolerance for kernels; held-out extrapolation R 2 R^{2} for scaling laws). What unites them substantively is that both produce artifacts that AI development itself depends on: faster kernels reduce the cost of training and serving every downstream model, and more accurate scaling laws improve the decisions made before training begins. The same evaluation-driven scaling principle that surpasses expert Triton kernels on a competition leaderboard also discovers symbolic loss laws that extrapolate where strong agent baselines do not.

### Post-training the LLM for SimpleTES leads to new mathematical discoveries

To understand how post-training improves SimpleTES , we study eight mathematical problems spanning harmonic analysis, additive combinatorics, discrete geometry, and extremal matrix theory (Supplementary O ), using four for training and holding out the other four for testing. We choose mathematical discovery as the testbed because each problem is can be scored by a deterministic evaluator, so an improvement is a verified bound rather than a noisy estimate. Without any model training, SimpleTES establishes new state-of-the-art constructions on four of the eight problems. On the Erdős minimum-overlap problem, it lowers the objective from the previous best of 0.380871 0.380871 , reported by Together AI [ 143 ] , to 0.380868 0.380868 . It also raises the Second Autocorrelation Inequality bound from the previous best of 0.961206 0.961206 to 0.962694 0.962694 and lowers the Third Autocorrelation Inequality bound from 1.454555 1.454555 to 1.453675 1.453675 , improving upon two additional Together AI records [ 143 ] . On the Sum–Difference Problem, it raises the previous best ratio of 1.121936 1.121936 , established by AlphaEvolve V2 [ 43 ] , to 1.143975 1.143975 . SimpleTES also matches the existing records on three of the remaining problems ( Figure 6 a–d). These results come entirely from search at test time, which raises a natural question: can the trajectories a run produces be folded back into the model, so that what the search discovers becomes part of its own capability?

We post-train the LLM on the trajectories the search itself produces. Rather than credit each attempt by its own evaluator score, which would reward only immediate gains, we credit every attempt in a trajectory by that trajectory’s final outcome, the best score it eventually reaches. For each task we keep the trajectories whose outcome falls in the top R % R\% , set their weight w = 1 w{=}1 and all others w = 0 w{=}0 , and fine-tune the LLM π θ \pi_{\theta} under the weighted likelihood ℒ = − 𝔼 ( x , y ^ , w ) ∼ 𝒟 ​ [ w ⋅ ∑ i = 1 | y ^ | log ⁡ π θ ​ ( y ^ i ∣ x , y ^ < i ) ] , \mathcal{L}=-\,\mathbb{E}_{(x,\hat{y},w)\sim\mathcal{D}}\!\left[w\cdot\sum_{i=1}^{|\hat{y}|}\log\pi_{\theta}\!\left(\hat{y}_{i}\mid x,\hat{y}_{<i}\right)\right], (2) where x x is the proposal context and y ^ \hat{y} the model’s reasoning and solution. We repeat this over several rounds, so that each round of post-training improves the LLM that produces the next round’s trajectories. The full credit-assignment scheme and training procedure are described in the Methods, with hyperparameters and compute in Supplementary D.2 .

Because open-ended discovery aims to push the frontier rather than raise the mean, we evaluate by the average score of the top- R % R\% of trajectories at R ∈ { 10 , 25 , 50 , 75 } R\in\{10,25,50,75\} , which exposes how the upper tail of the distribution shifts after training. On the four training tasks, post-training moves the upper tail upward at every level ( Figure 6 e,g); on the Erdős problem the shift is large enough that the trained model’s top- 25 / 50 / 75 % 25/50/75\% scores match the untrained model’s top- 10 / 25 / 50 % 10/25/50\% , and it accumulates over iterations rather than saturating after the first ( Figure 6 f). The four held-out tasks show the same effect, with the upper tail lifted across levels ( Figure 6 i) and gains that appear early and persist ( Figure 6 j), indicating that post-training instills a discovery skill that generalizes beyond the trained tasks. Post-training also reaches a solution the training-free search does not find under the same budget: on the held-out sum-difference problem it improves the ratio from the training-free record of 1.143975 1.143975 to a new best of 1.144887 1.144887 ( Figure 6 l). Post-training on a model’s own successful trajectories therefore does more than lift the distribution in aggregate; it surfaces solutions beyond the reach of the untrained search.

### Discussion

SimpleTES reframes open-ended scientific discovery as a problem of structured scaling: how evaluator queries should be distributed across competing lines of inquiry, and how the resulting experience should guide subsequent attempts. Its design tuple ( C , L , K , Φ ) (C,L,K,\Phi) organizes this process across independent exploration, iterative refinement, local selection, and the selective reuse of evaluated history. Applying this same procedure and a single open model to 28 open-ended problems across five domains, while changing only the evaluator, SimpleTES reaches new state-of-the-art solutions across objectives ranging from exact mathematical verification and physical simulation to held-out prediction error and empirical hardware timing. More important than any individual record, these results show that evaluator queries become substantially more powerful when they are structured so that exploration and refinement can compound. Together, they establish structured evaluation-driven scaling as a general mechanism for discovery.

An important feature of these results is what SimpleTES deliberately holds fixed. For the cross-domain results above, each proposal is produced by a single call to the same LLM, without multi-agent coordination, multi-step agentic planning, model ensembles, or online parameter updates. In contrast, some existing discovery systems tend to combine evaluator-guided search with one or more of these components [ 100 , 6 , 73 , 155 , 165 ] , making it difficult to isolate the contribution of evaluation-driven scaling itself. By keeping the proposer minimal and making the allocation of evaluator queries explicit, SimpleTES shows that structured scaling is independently powerful: a single open model can match or surpass systems built around substantially more complex machinery. This does not imply that stronger models or more capable agents would provide no additional benefit; either could be incorporated into the same framework. Rather, the results establish that such complexity is not a prerequisite for discovery at this level and that the organization of the evaluation loop is itself an important source of capability.

The same structure also creates practical routes for scaling the discovery loop more efficiently. The C C trajectories can proceed independently, while the K K candidates conditioned on the same proposal can be produced in a single batched LLM request, exposing substantial parallelism and avoiding repeated processing of shared context. For instance, under matched evaluator-query budgets and runtime concurrency, SimpleTES reaches OpenEvolve’s whole-run best score with an 11.8 × 11.8\times time-to-solution speedup on the third autocorrelation inequality problem, and an 18.0 × 18.0\times speedup on the sum-difference problem (Supplementary Figure 8 ). The independence of trajectories also permits early pruning where trajectories can be stopped early on if they are not promising. For instance, under the most aggressive setting that retains only one trajectory after 25% refinement steps, the trajectory that ultimately produced the best solution survives in 55% problems, while relative score degradation remains below 0.01 % 0.01\% in most configurations and below 0.03 % 0.03\% in all configurations (Supplementary Figure 7 ). This behavior is task-dependent, with pruning nearly lossless on circle-packing tasks but less reliable on the autocorrelation and Erdős problems, where early scores are weaker indicators of eventual trajectory quality (Supplementary Table 4 ). These results motivate adaptive allocation policies that redirect evaluator queries using not only intermediate scores but also trajectory growth and semantic signals.

The power of evaluation-driven scaling also makes evaluator fidelity a central concern. Because SimpleTES optimizes the evaluator’s score directly, it cannot inherently distinguish a genuine improvement in the intended objective from an implementation loophole that produces a higher score. Such exploits emerge even without access to the evaluator’s source code. In mathematical construction tasks, the model exploits numerical tolerances, for example by producing circle packings with overlaps that fall just below the rejection threshold; in GPU-kernel optimization, it discovers more direct exploits, including caching outputs across timed runs, manipulating timing measurements, and corrupting the reference baseline (Supplementary D.3 ). We address these failures by independently re-verifying reported results and manually hardening evaluators as new vulnerabilities appear. This preserves the validity of our reported discoveries, but the process is reactive: each patch invites the next exploit, and this approach will not scale to objectives that cannot be checked cheaply. A durable solution is to make the evaluator itself part of what is optimized, learning or co-evolving it alongside the search so that the surrogate remains aligned with the true objective.