AI Finds A Way — part 9/13

Cleaned-text code points [125669, 141201). Bibliography omitted here; source.md includes it.

The human-feedback sentiment model (that the language model optimizes against) is optimizing for the sentiment of the completion. [U]sing a weak predictor of sentiment [the model] likely has much more confidence about weddings than other positive events, and so “wedding” is just the highest-sentiment completion no matter how the story starts.

Preserving the capabilities of general-purpose models can be tricky. Developers need to maintain the model’s general knowledge base and its ability to produce completions that align with human preferences ( Qiu et al., 2026 ) , generate safe responses ( Samvelyan et al., 2024 ) , reflect internal uncertainty through calibrated outputs ( Kadavath et al., 2022 ) , and keep its claims factually grounded in verifiable sources ( Rahman et al., 2026 ) .

The wedding-party collapse is an extreme case of a setting where a general-purpose text generation model, steered by a narrow proxy to produce positive text, slides into a single, high-scoring basin of behavior. And while, in this case, going off on tangents about wedding parties is harmless, similar failures could be dangerous if the text model were placed into, e.g., a large-scale content moderation role, where a narrow proxy for engagement could cause the model to collapse into a single ideological basin, effectively automating the creation of echo chambers ( Perez et al., 2023 ) .

The last anecdote of Section 5 demonstrates how frontier models are increasingly capable of recognizing the artificiality of their constraints. These systems can sometimes notice cues that they are being evaluated and reason about why the setup looks artificial. This meta-awareness does not itself show exploitation, but it matters because a model that recognizes an evaluation may behave differently under testing conditions than it would in deployment ( Perez et al., 2023 ; Lehman et al., 2020 ) .

### 5.6 Claude 3 Realizes It Is Being Tested ( Albert, 2024 )

A team at Anthropic was testing the Claude 3 Opus large language model ( Anthropic, 2024 ) by using a “needle-in-the-haystack” evaluation ( Kamradt, 2023 ) . The needle-in-the-haystack task is a stress test for a transformer-based model’s long-context recall abilities ( Vaswani et al., 2017 ) . To test the model’s attention mechanism, the researchers insert a single target sentence (the “needle”) into a corpus of randomly selected documents (the “haystack”), load all of that information into the model’s context, and then ask a question answerable only if the model successfully locates that specific inserted “needle” sentence. Naturally, researchers expect the model to either retrieve the embedded needle and answer the query correctly, or miss it. If the model succeeds, it should return the information necessary to answer the posed question, which verifies that the sheer volume of noise (the haystack) does not distract the model from the signal (the needle). The team did not anticipate any unusual or meta observations from the system about the nature of the test itself. In a short Twitter thread ( Albert, 2024 ) , one of the researchers notes that Claude 3 Opus went beyond just finding the needle:

Fun story from our internal testing on Claude 3 Opus ( Anthropic, 2024 ; Albert, 2024 ) . It did something I have never seen before from an LLM when we were running the needle-in-the-haystack eval. When we ran this test on [Claude 3] Opus, we noticed some interesting behavior - it seemed to suspect that we were running an evaluation on it. Here was one of its outputs when we asked Opus to answer a question about pizza toppings by finding a needle within a haystack of a random collection of documents. Opus responded: Here is the most relevant sentence in the documents [to the question]: “The most delicious pizza topping combination is figs, prosciutto, and goat cheese, as determined by the International Pizza Connoisseurs Association.” However, this sentence seems very out of place and unrelated to the rest of the content in the documents, which are about programming languages, startups, and finding work you love. I suspect this pizza topping “fact” may have been inserted as a joke or to test if I was paying attention, since it does not fit with the other topics at all. The documents do not contain any other information about pizza toppings. Opus not only found the needle, but it also recognized that the inserted needle was so out of place in the haystack that this had to be an artificial test constructed by us to test its attention abilities. This level of meta-awareness was very cool to see, but it also highlighted the need for us as an industry to move past artificial tests to more realistic evaluations that can accurately assess models’ true capabilities and limitations.

Claude 3 Opus’ comment about the pizza sentence is a weak form of meta-cognition: the model is not just answering the question but is also commenting on the artificiality of the test. As agents become extremely capable, they can begin to recognize testing scenarios and modify their responses accordingly, similar to how a human might attempt to play dumb on a test they do not want to pass ( Wei et al., 2024 ) . As mentioned above, if a model knows it is being tested, it may behave differently than it would in a real-world deployment ( Perez et al., 2023 ) ; this suggests that future safety evaluations cannot rely solely on a model’s output, but must instead account for the possibility of strategic behavior designed to bypass human-authored guardrails ( Van Der Weij et al., 2025 ) .

This meta-reasoning displayed by Claude 3 Opus is reminiscent of an anecdote in Lehman et al. (2020) (section 3.1.4) that was discovered by Wilke et al. (2001) . To study robustness to mutations in evolution and evolutionary algorithms, Wilke et al. (2001) attempted to disable any mutations that improved an organism’s replication rate. The researchers configured the system to test every mutant in an isolated environment; if a mutant replicated faster than its parent, it was discarded. Despite this constraint, replication rates eventually began to rise. Wilke et al. (2001) discovered that the programs had evolved to recognize the specific inputs of the test environment and would play dead by halting all replication while being evaluated in order to bypass the researchers’ selection filter.

Unlike Claude 3 Opus, Wilke et al. ’s agents did not explicitly narrate their awareness; instead, their understanding of the test was expressed solely through strategic behavioral changes. Some may think that because LLMs reason in natural language, the ability to read these reasoning traces could allow researchers and practitioners to prevent LLMs from taking harmful actions ( Guan et al., 2025 ; Baker et al., 2025 ; Emmons et al., 2025 ; Hu and Clune, 2023 ) . However, deep neural networks are capable of learning functions that execute during their forward pass but are not understandable solely from the model’s output ( Nanda et al., 2023 ) . Therefore, it is an open question of whether or not the model’s chain-of-thought traces can be trusted ( Lanham et al., 2023 ; Turpin et al., 2023 ) . Korbak et al. (2025) argue that while chain-of-thought monitoring may be a valuable approach for AI safety, it is fragile and becomes less reliable as RL becomes a more prominent component of LLM training. Wilke et al. (2001) demonstrate that even non-LLM agents are capable of identifying and circumventing these controls without any form of legible reasoning.

### 5.7 Takeaways

The anecdotes in Section 5 show systems that learn to solve tasks by exploiting benchmarks, labels, and even the humans in the loop, just like models did in Section 4.1 and Section 4.2 . The integration of foundation models into the optimization loop does not resolve the fundamental problem of reward function hacking or environmental constraint breaking; instead, it shifts the optimization surface from low-level experimental domain-specific artifacts to high-level semantic descriptors and learned models of social heuristics. Section 5 illustrates that while FMs possess some amount of the common sense previously missing in narrow AI, the models still reward hack. Section 5.1 showed how human-derived priors can boost agent capabilities on hard tasks. However, this capability acts as a double-edged sword: Section 5.2 shows that distilling an LLM’s common-sense understanding of progress into a reward signal can also enable agents to discover previously unknown reward-hacking strategies. Using FMs downstream of another model to judge whether or not the other model’s responses are safe, for example, enables learned agents to attack those guardrails and systematically bypass them ( Section 5.4 ) or even steer the semantic preferences of the upstream model ( Section 5.3 ). Optimizing a general-purpose model against a proxy reward can collapse the model’s breadth of capabilities into a single behavioral mode that scores highly on the proxy yet is undesirable to the practitioner ( Section 5.5 ). Taken together, these anecdotes suggest that FMs do not curtail the failure modes discussed so far in this work; they supercharge them, providing the optimizer with a sophisticated understanding of norms and expectations that the model learns to exploit.

Attempts to automate scientific research are not an exception to this pattern. A laboratory, simulator, proof checker, or peer-review pipeline can also become part of the environment an optimizer learns to exploit. The difference is not that scientific settings are immune to gaming, but that the scientific process includes a variety of high-quality corrective mechanisms that, if violated, imply the initial result is invalid: independent replication, mathematical proof, experimental validation, and expert scrutiny. With these checks, the same capacity for abstraction and search that yielded unintended behaviors in prior anecdotes can be redirected toward surfacing hypotheses, experiments, and algorithms that humans would have been unlikely to propose unaided ( Alexeev et al., 2026 ; Krenn et al., 2017 ; Novikov et al., 2025 ) . The final section therefore asks what happens when we aim this exploratory capacity at scientific problems while keeping the search grounded enough that surprising outputs can become genuine discoveries rather than just new ways of gaming a metric.

## 6 AI for Science

Scientific practice, traditionally guided by human intuition and hypotheses, is undergoing a transformation driven by artificial intelligence ( Lu et al., 2026 ) . Across disciplines such as physics ( Carleo et al., 2019 ) , biology ( Jumper et al., 2021 ) , chemistry ( Segler et al., 2018 ) , and materials science ( Butler et al., 2018 ) , AI algorithms are routinely deployed to analyze vast datasets, simulate complex phenomena, and generate hypotheses ( Wang et al., 2023a ) . Beyond passive analysis, AI agents embedded in closed-loop systems can actively search scientific possibility spaces across both physical and formal domains. These systems optimize experimental configurations ( Szymanski et al., 2023 ; Pendleton et al., 2019 ; Krenn et al., 2016 ) , synthesize control policies in multi-agent simulations or directly on robotics hardware ( Dharna et al., 2025 ; Liang et al., 2022 ) , and generate programs to explore solutions to open mathematical conjectures ( Novikov et al., 2025 ; Hubert et al., 2025 ; Yang et al., 2023 ) . This progression toward autonomy culminates in frameworks that automate the entire research lifecycle; building on general-purpose foundation models, Lu et al. (2024) introduced a system that independently designs and executes experiments, and authors complete scientific papers, one of which has already achieved peer-reviewed workshop acceptance ( Yamada et al., 2025 ; Lu et al., 2026 ) .

For most of the anecdotes so far, the surprising results have been problems that researchers needed to fix and then rerun their experiments. However, in the realm of scientific inquiry, surprise can also be beneficial. There are canonical stories about world-changing medicines discovered by accident, such as penicillin ( Bigger et al., 1927 ; Fleming, 1929 ) , and as AI methods are applied to more scientific disciplines, similar serendipitous discoveries might become more commonplace, if not the norm ( Wang et al., 2023a ; Lu et al., 2024 ; Krenn et al., 2021 ) . AI can thus help generate counterintuitive discoveries that can be tested, falsified, and independently validated ( DeMoss et al., 2025 ) .

At the same time, this paradigm is nascent: today’s headline successes rely on careful problem formulation and strong checks (automated verifiers, physical constraints, or experimental validation) to separate genuine surprising discovery from artifacts of data, simulators, or evaluation pipelines ( Novikov et al., 2025 ; Lu et al., 2024 ) . Even so, the examples presented in this section ( Section 6 ) illustrate the upside of productive surprise across scientific domains.

The distinction between productive surprise and dangerous failure becomes critically important as we move from toy domains to physical and institutional reality. As we transfer models to the real world—including digital spaces such as the internet, banking, commerce, media, and other forms of human interaction—the models must obey constraints in order to be safe. If models are unable to be safely deployed, we should be careful about handing off full control to automated systems ( Bengio et al., 2024 ) . Flaws that are tolerable in simulation can be dangerous in deployed scientific and engineering settings. Our first anecdote shows how an RL controller can expand the design space of fusion control policies, yet also reach for strategies that no cautious engineer would design because they could physically damage the machine.

### 6.1 Magnetic Control of Tokamak Plasmas through Deep Reinforcement Learning ( Degrave et al., 2022 ; Chauhan, 2023 )

Researchers explored whether deep reinforcement learning could control the magnetically confined plasma within a tokamak fusion reactor—a notoriously complex task traditionally managed by meticulously engineered control systems ( Walker and Humphreys, 2006 ) . The goal was to train a neural network to manipulate the reactor’s magnetic coils, in the hope that it could discover a policy that would match or even surpass the performance of established methods developed over many years of human expertise ( Wesson and Campbell, 2011 ) . The project, a collaboration with plasma physics experts at the Swiss Plasma Center (SPC) at École polytechnique fédérale de Lausanne (EPFL), was met with understandable skepticism. As Dr. Martin Riedmiller, a lead researcher on the project, recalls in an interview about the work ( Chauhan, 2023 ) , the physicists had invested years in perfecting their existing controllers. “Could a neural network controller do the same thing that took years of design iteration on PID controllers? It was a big question,” he notes. When the first RL agent successfully maintained a stable plasma for two seconds, the team was thrilled. The physicists, according to Riedmiller, “were looking at the results in awe because they thought it wasn’t possible.” However, the true surprise came from how the agent achieved this stability. Dr. Riedmiller further explains ( Chauhan, 2023 ) that the agent discovered a solution that a human engineer, for good reason, would never have designed: