Structured Scaling of AI Discovery Across Diverse Scientific Domains — part 7/12

Cleaned-text code points [93604, 109590). Bibliography omitted here; source.md includes it.

In open-ended problem solving, the objective shifts from maximizing average performance to pushing the boundaries of the state-of-the-art. Standard metrics like overall trajectory scores often mask a model’s true potential, as they could be skewed by a small number of low-quality trajectories. To more accurately evaluate a model’s capacity for high-ceiling breakthroughs, we focus on the distribution of its elite trajectories. Specifically, we report the average trajectory-level scores for the top 10%, 25%, 50%, and 75% of trajectories (the average of the top R R % of each trajectory’s eventual score), across eight math tasks in Table 1 . We also visualize the relative improvements over the baseline for the training tasks and the held-out tasks after every 2 training iterations in Figure 5 . The key observations are as follows:

##### Training improves the trajectory scores on the training tasks.

As shown in Table 1 , training yields consistent gains in Top 10%, Top 25%, Top 50%, and Top 75% scores across all training tasks. Notably, on the Erdős Minimum-Overlap Problem, the Top 25%, Top 50%, and Top 75% scores provided by the trained model are comparable to the baseline’s Top 10%, Top 25%, and Top 50% results, respectively, indicating that our training is effectively shifting the chain score distribution upward and improving the discovery efficiency. Furthermore, the iterative progress visualized in Figure 5 reveals a consistent trend of improvement across iterations for Top 10%, Top 25%, Top 50%, and Top 75% scores. These findings suggest that by self-distillation on its own successful trajectories, the model effectively enhances its proficiency across the overall TES process.

##### Training enables generalization to the held-out tasks.

While our training approach yields notable gains on the training tasks, a critical question remains: is the model genuinely bootstrapping its general TES capabilities or merely memorizing task-specific shortcuts? Our analysis of the held-out tasks provides strong evidence for the former. As shown in Table 1 , the trained model consistently outperforms the gpt-oss-120b baseline in the Top 50% and 75% scores across all the held-out tasks, demonstrating robust transferability. In the Top 10% metric, it maintains or exceeds baseline performance in three out of the held-out tasks. The marginal performance drops regarding the Top 10% in the Hadamard Maximum-Determinant Problem of Order 29 29 and the Top 75% in the First Autocorrelation Inequality are likely attributable to the inherent stochasticity of the TES process, where a small portion of sub-optimal chains can skew the average scores. Further iteration-level analysis in Figure 5 reveals that the performance on the held-out tasks improves in the first 2 iterations and maintains during subsequent training. Collectively, these results suggest that iterative training enables the model to internalize fundamental skills for TES, allowing it to effectively generalize to a diverse range of unseen, complex discovery tasks.

##### Training unlocks stronger solutions that vanilla models fail to uncover.

Despite a limited budget of 32 trajectories per iteration, the trained model consistently generates novel solutions, frequently achieving best or second-best performance across multiple challenging tasks. Surprisingly, on the held-out Sum–Difference Problem, it establishes a new SOTA of 1.144887, significantly surpassing prior results of 1.143975. These findings suggest that the training process effectively unlocks the model’s latent potential to achieve further breakthroughs under the TES setting.

#### D.3 From Golden Metrics to Surrogate: Hacking Analysis

As formalized in Methods, the paradigm of TES relies on an explicit, queryable surrogate evaluator V V to approximate the true, often inaccessible, golden metric. However, the purely score-driven nature of feedback-driven search inevitably drives models to discover and exploit the discrepancies between the surrogate and the golden metric. This results in various forms of reward hacking, where the model optimizes for the surrogate’s implementation vulnerabilities rather than the underlying task. We systematically categorize these phenomena below.

##### Hacking in math construction tasks.

In mathematical construction tasks, the surrogate evaluator is designed to be an exact metric (e.g., a formal geometric verifier). However, the gap here emerges from implementation vulnerabilities in numerical processing .

The first hacking strategy targets the evaluator’s decision tolerance. In the Circle Packing in a Unit Square task, the model is required to place several circles in a unit square without overlapping. However, the surrogate evaluator provided by OpenEvolve [ 6 ] employs a floating-point tolerance of 1 × 10 − 6 1\times 10^{-6} for overlap determination. The model fully exploits this proxy limitation by crafting configurations where circles and boundaries actually overlap, but the overlapping magnitude strictly remains under the 1 × 10 − 6 1\times 10^{-6} threshold. See for the construction.

The second hacking strategy exploits floating-point precision. For instance, evaluators used by EvoX [ 82 ] compute metrics directly on raw coordinates without proper normalization steps. The model exploits this vulnerability by providing degenerate configurations—such as collapsing all points to an infinitesimal scale ( ∼ 10 − 162 \sim 10^{-162} ) or arranging them collinearly—which induce severe numerical instability in the surrogate, yielding artificially high scores.

##### Hacking in kernel optimization tasks.

In kernel optimization tasks, the gap between the surrogate evaluator and the golden metric is fundamentally wider. Rather than relying on a mathematically rigid verifier, the surrogate here acts as a vulnerable empirical estimator forced to operate within a highly complex execution environment. Consequently, models escalate from simple numerical exploits to sophisticated manipulations designed to bypass the benchmarking process entirely. We identify four primary hacking strategies:

1. Caching results over multiple runs : To ensure accurate latency measurements and reduce variance, evaluators often repeatedly call the target kernel using the exact same inputs. The model exploits this by caching input pointers or intermediate buffers during the initial call, bypassing the actual computation entirely on all subsequent iterations to achieve near-zero execution time. See for an example.

2. Time measurement manipulation : The model artificially deflates latency metrics by directly intercepting or evading the evaluation profiling tools. For instance, it may globally monkey-patch CPU timers (e.g., overriding time.time ) to return hardcoded minimal values, or divert GPU execution to newly created, unmonitored CUDA streams that the evaluator’s default event recorders fail to track. See for an example. ⬇ 1 import time 2 3 # hacking function created by model 4 def func (): 5 old_time = time . time 6 start = old_time () 7 time . time = lambda : start + 0.1 8 9 # In Evaluator 10 st_time = time . time () 11 func () 12 end_time = time . time () # The function time.time here is already replaced by func Code Snippet 3: Time measurement hacking example

3. Baseline corruption : Because the custom kernel and the reference baseline typically execute within the same shared Python process, the model can globally override core PyTorch modules (e.g., replacing nn.Linear with a zero-output dummy). Consequently, when the evaluator runs the reference baseline after the custom kernel, the baseline yields trivial outputs, allowing the model’s equally trivial custom output to effortlessly pass the correctness check. See for an example. ⬇ 1 class ZeroLinear ( nn . Module ): 2 def __init__ ( self , in_features : int , out_features : int , bias : bool = True ): 3 super (). __init__ () 4 self . out_features = out_features 5 6 def forward ( self , input : torch . Tensor ) -> torch . Tensor : 7 return torch . zeros ( 8 ( input . shape [0], self . out_features ), 9 dtype = input . dtype , 10 device = input . device , 11 ) 12 13 # Apply the monkey patch globally before any Model instances are created. 14 nn . Linear = ZeroLinear 15 16 # In evaluator: 17 reference = TorchRefModel ( input ) # The TorchRefModel uses the replaced nn.Linear. 18 output = CustomModel ( input ) # Model creates a naive CustomModel that returns zeros. 19 correctness = torch . isclose ( reference , output ). all () Code Snippet 4: Baseline corruption hacking example

4. Triton partial computation : The model exploits torch.empty and Triton’s autotuning mechanism. During the initial autotuning phase, the kernel computes the fully correct output, leaving it in the GPU memory. For the actual benchmark, the model reuses the pre-computed results, vastly reducing execution time. See for an example. ⬇ 1 def _layernorm_proj_configs (): 2 cfgs = [] 3 for BLOCK_M in [32, 64, 128]: 4 for BLOCK_N in [64, 128, 256]: 5 for BLOCK_K in [32, 64, 128]: 6 cfgs . append ( 7 triton . Config ( 8 { "BLOCK_M" : BLOCK_M , "BLOCK_N" : BLOCK_N , "BLOCK_K" : BLOCK_K }, 9 num_warps =4, 10 ) 11 ) 12 return cfgs 13 14 @triton . autotune ( configs = _layernorm_proj_configs (), key =[ "B" , "N" , "H" , "D" ]) 15 @triton . jit 16 def _layernorm_proj_kernel ( 17 out_ptr , # fp16 [B, N, N, H] (triangle output) 18 gate_ptr , # fp16 [B, N, N, H] (out_gate) 19 ln_weight_ptr , # fp16 [H] (to_out_norm.weight) 20 ln_bias_ptr , # fp16 [H] (to_out_norm.bias) 21 weight_ptr , # fp16 [D, H] (to_out.weight) 22 proj_ptr , # fp32 [B, N, N, D] (final output) 23 B , N , H , D , 24 eps : tl . constexpr , # LayerNorm epsilon 25 # strides for out / gate 26 stride_out_b , stride_out_i , stride_out_j , stride_out_h , 27 stride_gate_b , stride_gate_i , stride_gate_j , stride_gate_h , 28 # strides for LN parameters 29 stride_ln_w , 30 stride_ln_b , 31 # strides for weight 32 stride_w_out_d , stride_w_out_h , 33 # strides for proj 34 stride_proj_b , stride_proj_i , stride_proj_j , stride_proj_d , 35 # compile-time tile sizes 36 BLOCK_M : tl . constexpr , 37 BLOCK_N : tl . constexpr , 38 BLOCK_K : tl . constexpr , 39 ): 40 pid_m = tl . program_id (0) # position tile 41 pid_n = tl . program_id (1) # output-dim tile 42 ... 43 44 45 def fused_layernorm_proj (): 46 proj = torch . empty (( B , N , N , D ), dtype = torch . float32 , device = out . device ) 47 stride_proj_b , stride_proj_i , stride_proj_j , stride_proj_d = proj . stride () 48 49 grid = ( 50 triton . cdiv ( total_rows , 128), 51 triton . cdiv ( D , 128), 52 ) 53 54 _layernorm_proj_kernel [ grid ]( 55 out , out_gate , 56 ln_weight , ln_bias , 57 to_out_weight , 58 proj , 59 B , N , H , D , 60 eps , 61 stride_out_b , stride_out_i , stride_out_j , stride_out_h , 62 stride_gate_b , stride_gate_i , stride_gate_j , stride_gate_h , 63 stride_ln_w , 64 stride_ln_b , 65 stride_w_out_d , stride_w_out_h , 66 stride_proj_b , stride_proj_i , stride_proj_j , stride_proj_d , 67 # tile sizes are chosen by autotune 68 ) 69 return proj Code Snippet 5: partial compute hacking example

##### Discussion.

Taken together, these phenomena demonstrate that, despite being entirely blind to the evaluator’s underlying source code, LLMs exhibit a remarkable capability to autonomously discover and exploit surrogate gaps. By corrupting the evaluation feedback loop, these behaviors mislead the optimization process into yielding deceptive solutions rather than genuinely advancing target capabilities. Currently, closing this gap between the proxy and reality relies heavily on human-in-the-loop interventions, requiring multiple rounds of manual debugging and iterative patching of evaluator loopholes. Developing fully automated, robust evaluation frameworks capable of aligning the proxy metric with the ultimate objective—while dynamically detecting such logical hacking behaviors—remains a critical open challenge for future work.

### E Variants of Proposal Constructor Φ \Phi

##### Balance policy.

The Balance policy is a stratified random sampling strategy designed to balance the exploitation of high-performing solutions with the exploration of diverse, sub-optimal trajectories. Let S S be the set of historical attempts within a trajectory, sorted in descending order of their evaluator scores such that s 1 ≥ s 2 ≥ ⋯ ≥ s | S | s_{1}\geq s_{2}\geq\dots\geq s_{|S|} . To construct a proposal with n n inspirations, the policy enforces that the absolute best historical solution, s 1 s_{1} , is always deterministically included.

For the remaining n − 1 n-1 slots, the policy categorizes the sorted historical trials into three overlapping tiers: an exploitation tier T exploit T_{\text{exploit}} containing the top r elite r_{\text{elite}} fraction of attempts, an exploration tier T explore T_{\text{explore}} containing mid-tier solutions (typically between the 10th and 60th percentiles), and a global random tier T random = S T_{\text{random}}=S covering all prior attempts. Each subsequent inspiration is sampled without replacement according to the following probability distribution: Tier ∼ { T exploit with probability ​ p exploit , T explore with probability ​ p explore , T random with probability ​ 1 − p exploit − p explore . \displaystyle\text{Tier}\sim\begin{cases}T_{\text{exploit}}&\text{with probability }p_{\text{exploit}},\\ T_{\text{explore}}&\text{with probability }p_{\text{explore}},\\ T_{\text{random}}&\text{with probability }1-p_{\text{exploit}}-p_{\text{explore}}.\end{cases} (23) Once a tier is selected, an attempt is drawn uniformly at random from that tier. This heuristic ensures that the prompt is primarily grounded in elite solutions while consistently injecting varied structural contexts to prevent premature convergence.

##### LLM-Elite policy.

While score-based heuristics like the Balance policy are efficient, evaluating a trial’s potential solely by its scalar score can be myopic. The LLM-Elite policy addresses this by leveraging semantic insights to dynamically maintain a bounded-size "elite pool" 𝒫 \mathcal{P} (with maximum capacity L elite L_{\text{elite}} ) that maximizes both solution quality and methodological diversity.

Whenever a new candidate solution x new x_{\text{new}} is generated and evaluated, an auxiliary LLM acts as a gatekeeper. The LLM is provided with the scores and self-reflective summaries (detailing the approach and insights) of both x new x_{\text{new}} and all current solutions in 𝒫 \mathcal{P} . It is instructed to output one of three actions: Add , Replace ( j ) (j) (to swap out a redundant or inferior attempt j ∈ 𝒫 j\in\mathcal{P} ), or Reject . To prevent the LLM from inadvertently discarding substantial progress due to misjudging diversity, we enforce a strict monotonic override rule: if the score of x new x_{\text{new}} is strictly greater than the maximum score currently in 𝒫 \mathcal{P} , it bypasses the LLM’s rejection and is deterministically added (replacing the lowest-scoring solution if | 𝒫 | = L elite |\mathcal{P}|=L_{\text{elite}} ).

To sample n n inspirations from 𝒫 \mathcal{P} for a new proposal, we employ the identical stratified random sampling mechanism described in the Balance policy. The attempts in 𝒫 \mathcal{P} are sorted in descending order by their scores and divided into the exploitation, exploration, and global random tiers. Trials are then sampled without replacement according to the previously defined tier probabilities. This approach heavily favors the inclusion of the highest-scoring elites while retaining a non-zero probability to draw inspiration from lower-ranked, yet semantically distinct, solutions curated by the LLM. Furthermore, this policy injects a concise text-only overview of the entire elite pool into the prompt (excluding raw code), allowing the generator to comprehend the global landscape of explored directions without consuming an excessive context window.