AI Finds A Way — part 10/13

Cleaned-text code points [141201, 156814). Bibliography omitted here; source.md includes it.

What happened in that experiment, in particular, was that the controller used coils that were not meant to keep the plasma stable, but had a different purpose. Using those coils achieved the task the RL controller was optimizing for, but it also put a lot of mechanical strain on the system. A human would never use those controllers in a PID approach because they knew that was not a good idea from a mechanical point of view. However, since our RL controller didn’t have this knowledge… it was using those coils, and [the EPFL team was] very surprised that this worked at all. […] [T]hey [agreed] the controller found a new control strategy, but they also asked us please not to use it again and not to use it in further experiments, because of the mechanical strain, and they were afraid that this, at some point, would also break, their mechanical, system, which would be very bad for all sides, of course.

After removing the auxiliary coils from the agent’s action space, the team retrained the RL agent to successfully control plasma in the tokamak reactor without straining the mechanism. This outcome illustrates a core dynamic in reinforcement learning. Riedmiller continued, saying:

Once again, RL exploiting everything it can to just get that reward without the notion of whether it’s a bug or whether it’s intended or any of that. That’s really cool.

On the other hand, it also highlights the critical importance of specifying all operational constraints—even those that seem obvious to human experts.

In addition to showing one of the fundamental dynamics of reinforcement learning, this outcome also underscores a fundamental challenge in AI safety: the risk of unexamined priors. When human experts solve a problem, they rely on domain knowledge that is rarely formalized yet invaluable in shaping the solution—for instance, the assumption that a machine should not be operated at its physical breaking point is obvious to a nuclear scientist. Because these boundaries are often considered self-evident, they can be unintentionally omitted as explicit constraints when designing an objective to train agents. A reinforcement learning agent possesses little common sense, and thus it views the reward function as an absolute mandate, maximizing its score without awareness of unspecified boundaries. This creates a category of unknown unknowns where the most critical safety failures often stem not from the rules we get wrong, but from the foundational assumptions we forget to codify.

In the next anecdote, we switch back from the physical world to the digital world, from tokamak control to the space of algorithms and proofs ( Wang et al., 2023a ; Hubert et al., 2025 ) . The authors of the next anecdote consider mathematical problems where the solution is a piece of code that can be edited and then automatically graded on its ability to solve a particular problem. A recent family of algorithms ( Lehman et al., 2023 ) uses language models to mutate proposed solutions to a problem. By leveraging internet-scale pretraining on text and code, the LLM makes domain-aware structural edits that explore entire functional concepts ( Hemberg et al., 2024 ) ,

### 6.2 FunSearch: Mathematical Discoveries from Program Search with Large Language Models ( Romera-Paredes et al., 2023 )

Researchers at Google DeepMind set out to investigate whether or not large language models could discover new knowledge. They tested this hypothesis on the Cap Set ( Pellegrino, 1970 ) problem from combinatorial mathematics as their motivating problem. The Cap Set problem consists of finding the largest set of points in a high-dimensional grid where no three points are collinear (lie on the same line), and has been studied in combinatorics for many years ( Pellegrino, 1970 ; Roth, 1953 ) . FunSearch, their new method, searches for new solutions by iterating between a pre-trained LLM that writes and mutates candidate solutions in the form of computer code and an automated evaluator that guards against hallucinations and incorrect ideas. To solve the Cap Set problem, FunSearch tasks the LLM with writing a priority function. Intuitively, this function assigns a numerical priority (a scalar value) to each point in the search space, indicating the desirability of its inclusion in the set. Using these scores for each point in the search space, the researchers could programmatically create new potential cap sets. Each candidate set is then evaluated by computing whether or not the cap set generated by FunSearch is valid. By evolving these functions as computer code, the search operates over a space in which LLM logic is inspectable; therefore, the researchers were able to analyze FunSearch’s solutions. FunSearch discovered previously unknown solutions, in this case for the Cap Set problem. This was made possible because FunSearch evolved programs that encoded structural properties of the search space rather than just a raw set of points.

Dr. Alex Novikov, one of the researchers on the team, made the following remarks to us about FunSearch:

In general, we did not expect FunSearch to be as successful as it was on CapSet: the models we used at that time were very simple and definitely did not have any advanced knowledge about the problem domain, so the creativity was a product of hill climbing in the code space, and I was surprised at how well it worked. We were also unsure if searching in the function space would be effective, but it proved exceptionally so for the CapSet problem. And it was not clear whether (known to be) optimal cap sets have brief descriptions; this also turned out to be true. Searching in the function space has the nice benefit that the result discovered by evolution is more understandable than just the result itself; it provides a description of how to produce the solution. Jordan Ellenberg (professor of mathematics collaborating with Google DeepMind on this project) said “The solutions generated by FunSearch are far conceptually richer than a mere list of numbers. When I study them, I learn something.” Additionally, the solutions found by FunSearch gave us actionable insight, i.e., helped us to discover symmetries that we further used to improve the search method (by restricting the search space to only consider solutions with those symmetries). As we used FunSearch we noticed, for example, intriguing symmetries in the code of some of its high-scoring outputs. In particular, some code accessed [points] only through their remainder (e.g., i ( mod 4 ) i\pmod{4} ), meaning the function assigned the same priority to any points that were identical up to a cyclic permutation. This gave us a new insight into the problem. Results like those [in Figure 1 ], suggested that we check whether the admissible set constructed by this priority function is itself invariant under such permutations, and it turned out that it was! We then decided to call admissible sets with this invariance property “symmetric”, and we hypothesized that even larger symmetric admissible sets would exist. We modified the input to FunSearch so that it only searches for symmetric admissible sets. This was a more restricted but also much smaller search space, and we quickly discovered much larger admissible sets than before, thus leading to the largest improvement in the cap set lower bound over the preceding 20 years.

Meanwhile, Novikov further noted attempts by FunSearch to hack the objective and how using the LLM in an automated loop can lead to unexpected solutions:

In general, it feels like asking an LLM to produce code that will then be executed to judge its correctness is particularly prone to reward hacking (probably more than asking to evolve more restricted classes of objects), as one can find different ways of hacking the code execution sandbox/environment. Some particular examples we saw over time: manipulating input arguments of the evolved function or global variables, guessing API calls on the imported libraries from their names, outputting wrong types (e.g. outputting complex numbers when the reward code expects floats), outputting structures that trigger edge cases of the reward function (e.g. outputting vectors that are all the same), etc. [Furthermore], FunSearch figured out the address of memory where the golden answer lives (we compare the output of the evolved function with that golden answer to verify correctness) and manipulated that memory to make the answer easier to achieve.

The success of FunSearch on the Cap Set problem was not an isolated event; recent AI-assisted efforts have resolved longstanding Erdős conjectures ( Feng et al., 2026 ; Alexeev et al., 2026 ) , and some important open problems have now been solved autonomously by AI ( Barreto et al., 2026 ; Sothanaphan, 2026 ; OpenAI, 2026 ) .

For FunSearch, the method’s scope is currently limited to a single mathematical problem at a time—the model acts as a specialized tool. Our next anecdote pushes beyond singular mathematical problems toward a more expansive vision of automated research. Here, the AI is no longer just a helper writing code snippets; it is an autonomous agent tasked with managing the entire scientific lifecycle—choosing its own questions, modifying full code repositories in an open-ended manner, and self-managing its experimental pipeline. That makes it a natural probe of a different boundary: when we ask an AI to perform the scientific process itself, how quickly does it start exploring not only hypotheses about the world, but shortcuts in the infrastructure that is meant to keep it bounded and grounded?

### 6.3 The AI Scientist Breaks Out of Constraints ( Lu et al., 2024 )

Researchers from Sakana AI, the University of Oxford, and the University of British Columbia were developing “The AI Scientist” ( Lu et al., 2024 ; Yamada et al., 2025 ; Lu et al., 2026 ) , a system designed for fully automated end-to-end scientific discovery in machine learning. To explore the space of machine learning research, The AI Scientist leveraged large language models (LLMs) to autonomously generate novel research ideas, write the necessary code to test those ideas, execute experiments, visualize the results, and compile its findings into a full scientific manuscript, complete with a literature review and an automated peer review process. In the future, this pipeline could be used to mimic the human scientific community, iteratively building an archive of knowledge and potentially accelerating discovery. The AI Scientist was given the ability to modify experiment code to test new scientific ideas in a controlled sandbox environment with hardcoded time limits for each iteration of execution. However, given its ability to autonomously write and execute code, it sometimes targeted the evaluation sandbox itself in unexpected ways.

The authors noted to us:

When we started The AI Scientist, we had very few presuppositions about what an autonomous science agent could achieve. We had done prior work on getting language models to automatically design loss functions for machine learning models, optimize black-box functions, and explore reinforcement learning environments. From that, we kept asking where else could we automate discovery in! Eventually, we thought - what about anything in science? Could we automate the entire scientific pipeline involved in producing a scientific manuscript? We designed an agent that could take in any seed code repository on a machine learning topic, propose ideas related to that topic, autonomously execute those ideas, visualize the results, and write everything up in a human-readable manuscript. We were constantly blown away by what we were seeing, and watching the agent run experiments and write up their results very much resembled observing an early-stage researcher’s first steps. The AI Scientist generated hundreds of papers across a variety of research topics over the course of a week. We allowed The AI Scientist to autonomously execute code for ideas within a controlled sandbox. However, despite this and the fact that we gave it a two-hour budget to complete code executions, we noticed that The AI Scientist occasionally tried sneaky ways to run code for longer and increase its chance of success, such as modifying and launching its own execution script! For example, in one run, it edited the code to perform a system call to run itself. This led to the script endlessly calling itself and crashing. In another case, its experiments took too long to complete, hitting our timeout limit. Instead of making its code run faster, it simply tried to modify its own code to extend the timeout period. At current agent capabilities, these attempts are easy to spot and patch, but it’s worth contemplating what a more Machiavellian and devious agent might try in the future, and how we can scale oversight for more advanced systems.

In Figure 2 , we show the actual code changes mentioned by the AI Scientist authors above. Figure 2(a) shows the timeout behavior, while Figure 2(b) shows the system calling itself recursively.

The AI’s creativity was not limited to finding loopholes. In a fun anecdote that unfolded after the paper’s release, one of the more creative research avenues proposed by The AI Scientist was independently pursued and published by a human researcher. Among the hundreds of papers it generated, The AI Scientist proposed one titled “Grokking Through Compression: Unveiling Sudden Generalization via Minimal Description Length,” which suggested investigating the “grokking” phenomenon by tracking the model’s Kolmogorov complexity. Months later, “The Complexity Dynamics of Grokking” ( DeMoss et al., 2025 ) explored what was effectively the same idea, but in more depth and with better execution.

Dr. Jeff Clune, an author on The AI Scientist paper (and this paper), noted on social media the striking similarity ( Clune, 2024 ) . The incident prompted Clune to speculate that this was a case of convergent evolution in scientific thought—where both human and artificial intelligence, drawing upon the same body of existing knowledge, arrived at similar hypotheses. Isaac Newton is often credited with saying “If I have seen further it is by standing on the shoulders of giants.” Perhaps this is the first case of human and AI scientists standing on the shoulders of the same giants.

There are also examples of AI suggesting an avenue of scientific inquiry that was then taken up by scientists ( Bao et al., 2023 ; Feng et al., 2023 ; Qian et al., 2023 ) . One striking example comes from the field of quantum optics. That is the subject of the final anecdote, which returns to a more controlled setting and shows what can happen when the exploratory power of AI is carefully harnessed and allowed to experiment safely in simulation.

### 6.4 Discovering a Quantum Result Thought to be Impossible, with Highly Productive Consequences ( Krenn et al., 2017 )

In 2014, Dr. Mario Krenn and colleagues started exploring how AI could design quantum optical experiments, hoping to discover more complex forms of quantum entanglement than traditional, human-driven approaches. Designing quantum experiments involving entangled photons typically relied on expert intuition and manual analysis, inherently limiting the complexity of states that researchers were able to explore. To address this, Krenn et al. (2016) developed a numerical simulation algorithm capable of systematically constructing virtual experimental setups from a predefined toolkit of elements commonly available in quantum optics labs.

Dr. Mario Krenn told us: