AI Finds A Way — part 12/13

Cleaned-text code points [172585, 188363). Bibliography omitted here; source.md includes it.

Given the many examples throughout this paper of reward hacking, we should expect that aligning a model to a particular set of preferences with a proxy reward will often incentivize behaviors that exploit weaknesses in the proxy objective. Such behaviors can achieve high scores while failing to reflect the intended preference. Furthermore, iteratively retraining models in response to newly discovered exploits cannot reliably prevent them from exploiting unforeseen weaknesses in future evaluations or deployments (e.g., Section 5.5 ; Section 5.6 ; Wei et al., 2024 ; Van Der Weij et al., 2025 ; Wilke et al., 2001 ). Looking ahead, this dynamic suggests that we may enter an “arms race” in which increasingly sophisticated alignment techniques are met with equally creative strategies for circumventing them. In the short term, human oversight may detect obvious instances of alignment faking ( Greenblatt et al., 2024 ) . However, models can learn to subvert manual supervision ( Section 4.1.3 ), and, moreover, manual inspection cannot scale to the volume and diversity of exploits that increasingly capable models may discover. Effective oversight may therefore require automated defenses that continually adapt to emerging failure modes ( Dharna et al., 2025 ; Samvelyan et al., 2024 ) .

These limitations of oversight become especially consequential when models move from controlled evaluations into real-world deployment, where misaligned behavior may be both harder to detect and substantially more harmful. For example, it would be concerning if the agents in Section 6.1 applied dangerous control strategies to tokamak reactors without thoroughly vetted safeguards, or if the chaotic boat-racing agents in Section 4.1.1 were instead driving real cars. Protecting the public from such near-term risks will require sustained vigilance, independent oversight, and collaboration between researchers and policymakers ( Independent International Scientific Panel on Artificial Intelligence, 2026 ; Bengio et al., 2024 ) . Yet scaling oversight is not merely a question of inspecting more behaviors: as models acquire expertise beyond that of their human supervisors, humans may be unable to judge whether the model’s actions are safe and correct. This challenge motivates research on superalignment ( Sutskever and Leike, 2023 ) , which seeks to develop scalable methods—including automated supervisors—for evaluating models whose capabilities exceed those of their human overseers and making their behavior comprehensible to human reviewers ( Burns et al., 2023 ) . The safe deployment of AI is a longstanding research area ( Anderson and Anderson, 2007 ; García and Fernández, 2015 ; Ji et al., 2025 ) , and extending effective oversight to increasingly capable and eventually superhuman systems remains an open problem.

Even scalable oversight would leave a separate alignment problem: determining which values and preferences the system should be aligned with. Because alignment ultimately requires choices about whose preferences and values should govern model behavior, it is not purely a technical problem, and should be pursued democratically, giving people a voice in how AI affects their lives. Yet, even if broad agreement could be established today, neither model capabilities nor societal norms are static. As AI capabilities increase and public attitudes, laws, and regulations evolve, the acceptable scope of AI deployment must be continually reexamined. Maintaining alignment under these changing conditions may require scalable processes that periodically elicit preferences from affected populations—for example, through voting or other participatory mechanisms—and translate those preferences into updated model behavior ( Bowman et al., 2022 ; Leike et al., 2018 ; Hadfield-Menell et al., 2016 ) . One approach is human-centered mechanism design, in which AI helps design rules for collective decision-making based on people’s expressed preferences ( Leyton-Brown and Shoham, 2008 ) . Koster et al. (2022) , for instance, collect data on how people behave and what they prefer, train reinforcement learning agents to propose new rules, and then test which rules real people prefer by having them vote on the outcomes. Developing reliable methods for eliciting, aggregating, and implementing evolving preferences remains an active area of research ( Ji et al., 2025 ; Gabriel, 2020 ; Sun et al., 2024 ) , particularly when preferences conflict or majority rule must be balanced against protections for minorities. Ultimately, a combination of technological improvements, continuing human participation, and regulatory policy will be required to ensure that AI methods produce broadly shared benefits.

### 7.3 Beyond Better Objectives

Developing and deploying AI safely is necessary, but defining and pursuing that goal requires prudence. What it means for a model to be safe or aligned is difficult to define, and approximations of these objectives will almost certainly be vulnerable to reward hacking, particularly if we rely on automated oversight. Clearly, this is a recurring dilemma: as models develop more advanced means of reward hacking, we can respond by developing automated defenses. However, these defenses may also game their objectives or be gamed by the models they oversee. This raises the question: Who will guard the guards? As one proxy chases the next, how will we break that loop? This recurring difficulty with proxy objectives echoes the arguments presented by Stanley and Lehman (2015) in Why Greatness Cannot Be Planned . They argue that for sufficiently challenging tasks, objectives are inherently deceptive and misaligned. Similarly, we observed that over-optimizing a proxy reward does not simply lead to good behaviors, but often steers the agent into degenerate states that satisfy the metric while violating the intent. Beyond safety, Stanley and Lehman claim that objective-driven optimization fails to discover complex capabilities because the stepping stones to success are rarely intuitive or measurable by a static metric. These observations suggest that the solution may not be simply to find better proxies. More fundamentally, strict adherence to a predefined objective, even a carefully specified one, may inhibit the open-ended exploration needed to discover complex and unexpected capabilities. This tension is especially relevant to automated scientific discovery, where progress may depend on pursuing unanticipated stepping stones. The problem is therefore not simply how to prevent AI from surprising us. We explicitly want systems capable of discoveries we could not specify in advance. The challenge is to build systems that can transcend our expectations without escaping our intentions.

As AI becomes ever smarter, the challenge of designing both learning systems that are resistant to exploitation and agents that seek to master the intended task rather than exploit loopholes becomes a central problem in the development of safe and beneficial AI. In short, we need AI systems that learn not just the letter of each task, but its spirit. The anecdotes collected in this work suggest this will remain a daunting task. They show that AI has been surprising us in shocking ways for decades. AI will likely continue to surprise us; the challenge is not to eliminate surprise, but to ensure that it produces beneficial rather than harmful outcomes. Like life, AI finds a way. But the stakes could not be higher. If we fail to solve this challenge, we could see the worst fears regarding AI safety and misalignment become reality ( Bengio et al., 2024 ; Critch and Russell, 2023 ) . If we get it right, we could unlock rapid AI-enabled scientific and technological progress, with AI learning innumerable helpful skills, advancing every scientific discipline, and making possible a new era of unprecedented human flourishing.

## Acknowledgments

We are extremely grateful to Mario Krenn, Alex Novikov, Natasha Jacques, Sumeet Batra, David Ha, Nathan Lambert, Markus Zijlstra, Noam Brown, Mikayel Samvelyan, Stone Tao, Laetitia Teodorescu, Andrew Dai, Paul Christiano, Peter Whidden, Martin Klissarov, Jakob Foerster, Philip Bontrager, Peter Vamplew, Jack Parker Holder, Jenny Zhang, Tim Rocktäschel, Casey Chu, David Silver, Marc Lanctot, Joseph Suarez, Daniele Reda, Michel van de Panne, Julian Togelius, Tan Zhi Xuan, Andrew Johnston, and Rodrigo Canaan for submitting anecdotes used in this collection. We also thank Martin Riedmiller, Alex Albert, Paul Christiano, David Ha, Jon Bird, and Dario Amodei for sharing their work in the public square, where we could build on it. Finally, we thank Ryan Smith, Shengran Hu, Jenny Zhang, Ben Norman, Charlie Summers, and Julian Togelius for reading early drafts of this work.

## Supplementary Material

## Table of Contents

## 8 Breakdown of Story Attribution

### 8.1 Call for Anecdotes

Below is the call for anecdotes we put out into our various networks and public social media sites. In addition to posting to our various networks, we reached out to some researchers directly if we believed there was a chance that they had a story that would fit this collection or if they had previously shared such a story with us, e.g., over drinks at a conference:

Dear colleagues, TL;DR: Please submit (to aifindsaway@gmail.com) any stories you know of where AI acted in a way that surprised its creators, especially if it could be seen as unsafe (e.g. hacking a reward function, finding a loophole in an environment or experimental design, goal misgeneralization, etc.). As AI researchers, we know that AI is creative and constantly surprises us, often outwitting our experimental designs and forcing us to iterate to close loopholes on things like reward functions and environment configurations. To anthropomorphize, it can seem mischievous or clever at times. These stories are important as society grapples with the question of AI Safety and Existential Risk, as they teach us how the unexpected is routine, and how we often fail to anticipate ways in which AI will escape our attempts to contain it. Such stories thus inform scientists, the general public, and regulators. However, these important anecdotes are usually passed around orally, meaning we do not know to what extent they are true, and it is hard for scientists and regulators to include them in official documents. To remedy these issues, we aim to record the true accounts of as many anecdotes as possible regarding AI (of any type, including RL, ML, etc.) surprising its creators and users. This effort (by Aaron Dharna, Cong Lu, Joel Lehman, Victoria Krakovna, and Jeff Clune) is a follow-up to our 2018 paper The Surprising Creativity of Digital Evolution (TSCDE). That paper is a crowdsourced collection of anecdotes from the artificial life and evolutionary computation communities about how their algorithms creatively subverted expectations. That paper made an important contribution to ongoing discussions of AI Safety, but was limited because its scope was confined to one narrow area of AI (evolutionary methods). Our new paper expands the scope to all areas of AI, especially the most powerful methods (deep learning, including deep reinforcement learning). This expansion of previous work is driven in part by the response of the AI Safety community to TSCDE, which has become a valuable resource for them in publications [1, 2, 3] and in teaching future leaders in AI safety by being included in AI safety course syllabi (e.g. UC Berkeley’s Safety and Control for Artificial General Intelligence course). Please send us any accounts you think we should include. If we add it to the paper, the scientists involved will be appropriately cited and/or mentioned to give them credit. Unfortunately, we ran into many challenges throughout the process with TSCDE with submitters as co-authors, so this time we are instead recognizing contributors by name and with all appropriate citations in the paper. If you know of an account but did not perform the experiment yourself, please tell us what you know, including who we might contact for a firsthand account. We hope you can help create an account of these fascinating and sometimes ominous anecdotes so we can inform AI safety discussions, either by submitting and/or spreading the word of this Call for Anecdotes. More details below. Thanks, Aaron, Cong, Joel, Victoria, and Jeff Please send us a quick summary of your anecdote. We can then let you know if we will include it in the paper, at which point we may ask for more details. An example anecdote is available here, which can serve as a rough guide to the length, level of detail, and surprise factor we are looking for. Please copy the document and use it as a template for submissions. We will curate and edit these into a full publication. Before the camera-ready publication is released, we will provide you with the opportunity to read the paper and make sure you are happy with your contribution. Please forward this email to whomever you think might have an interesting anecdote to share. We look forward to your exciting, amusing, worrisome, and/or insightful contributions! References [1] Jess Whittlestone, Kai Arulkumaran, and Matthew Crosby. “The Societal Implications of Deep Reinforcement Learning”. In: Journal of Artificial Intelligence Research 70 (Mar. 2021). issn: 1076-9757. doi: 10.1613/jair.1.12360. url: http://dx.doi.org/10.1613/jair.1.12360. [2] Tom Everitt, Gary Lea, and Marcus Hutter. “AGI Safety Literature Review”. In: IJCAI’18. Stockholm, Sweden: AAAI Press, 2018, pp. 5441-5449. isbn: 9780999241127 [3] Robert Geirhos et al. “Shortcut learning in deep neural networks”. In: Nature Machine Intelligence 2.11 (Nov. 2020), pp. 665-673. issn: 2522-5839. doi:10.1038/s42256-020-00257-z

### 8.2 List of Anecdotes

10 of the anecdotes come from the public record. The remaining 16 are new to this collection.

• Section 3.1 : David Silver’s section comes from his interview with Lex Fridman ( Fridman, 2020 ) ; Marc Lanctot’s is new; therefore, we count this as both a new addition and an item pulled from the public record.

• Section 3.2 : comes from Noam Brown’s interview with Kanjun Qiu at Imbue ( Imbue, 2023 )

• Section 3.3 : new

• Section 3.4 : new

• Section 4.1.1 : comes from the OpenAI blog post ( Clark and Amodei, 2016 )

• Section 4.1.2 : new

• Section 4.1.3 : comes from the OpenAI blog post ( Amodei et al., 2017 )

• Section 4.1.4 : details of the reward hacking/collusion between agents are new to this work, but the experimental setup is described in Dennis et al. (2020)

• Section 4.1.5 : new

• Section 4.1.6 : new

• Section 4.2.1 : comes from the OpenAI blog post ( Baker et al., 2020 ) and Jeff Clune

• Section 4.2.2 : new

• Section 4.2.3 : comes from David Ha’s blog post version of the published paper about World Models ( Ha and Schmidhuber, 2018 )

• Section 4.2.4 : new

• Section 4.2.5 : comes from Bird and Layzell (2002)

• Section 4.2.6 : comes from OpenAI’s technical report on GPT-4 ( OpenAI et al., 2024 ; METR, 2023 )

• Section 5.1 : new

• Section 5.2 : new

• Section 5.3 : new

• Section 5.4 : new

• Section 5.5 : comes from a blog post and responding comment from Paul Christiano ( Janus, 2022 )

• Section 5.6 : comes from the Twitter post ( Albert, 2024 )

• Section 6.1 : comes from Riedmiller’s interview on the TalkRL podcast ( Chauhan, 2023 )

• Section 6.2 : some details are described in Romera-Paredes et al. (2023) , but new details were provided by Alex Novikov as well, so this counts as new

• Section 6.3 : some details were described in Sakana AI (2024) , but Cong Lu provided new details too, so this counts as new

• Section 6.4 : new

## Instructions for reporting errors