AI Finds A Way — part 3/13

Cleaned-text code points [31625, 47184). Bibliography omitted here; source.md includes it.

The game is played in two formats relevant here. In the full game of Diplomacy, players negotiate using natural language. In Gunboat Diplomacy, explicit communication is forbidden, so players must instead infer one another’s intentions from board maneuvers. In 2022, Meta AI introduced Diplodocus for Gunboat Diplomacy ( Bakhtin et al., 2022b ) and Cicero for the full game ( Bakhtin et al., 2022a ) .

Markus Zijlstra, an expert Diplomacy player involved in the development of Cicero, observed unexpected strategies emerging from both systems. In Gunboat Diplomacy, Diplodocus repeatedly found strategies that experienced human players would usually avoid. One example was its decision to play Germany, traditionally an army-based power, primarily as a naval power. On this strategy, Zijlstra noted:

When Diplodocus is playing Germany, which is traditionally an army-based power, it did so as a primarily naval power in the early game. This is extremely unintuitive because the fleets cannot really be used defensively, so it’s something of an all-or-nothing strategy focused on attacking Scandinavia and is dependent on France not attacking Germany’s weak western front. In practice, it was extremely effective. Despite France having [a] window where they could attack Germany, most did not, because they thought that a fleet-based Germany was not a threat to them.

Another memorable example showed Diplodocus using non-tactical moves to signal cooperation. In Gunboat Diplomacy, signaling moves show intent to other players rather than serving a purely tactical purpose. In one game where Diplodocus played Italy and Zijlstra played Austria, he recalled:

There was a game in which Diplodocus moved into an Austrian home center early in the game, which in Gunboat [Diplomacy] would often be seen as an act of war. But it then signalled an intention to work with me by building double fleets […] and repeatedly issuing support holds to my units which served no tactical purpose and seemed to be stating “I want to work with you”. That got me back onside enough that I started working with it and left myself a little open to it, at which point it immediately launched a crippling attack on me. More generally, Diplodocus often abandoned areas that human players would consider vital to defend in order to aggressively push into more defensible areas that it did not yet control. There was one particularly stunning example of this where as England, it abandoned its entire homeland (meaning it gave up any possibility of building new units). Later in the game Diplodocus successfully launched an attack back into its homeland again and ended the game with an extremely strong position.

Those ideas also fed back into his own play, though not without hesitation:

Diplodocus really pushed the skill ceiling on Gunboat Diplomacy. With Diplodocus, I now much more aggressively push for certain “safe areas” (primarily Scandinavia and Iberia) in Gunboat Diplomacy when I’m under attack, rather than prioritising my home centers. This was generally quite effective and helped me match the bot’s performance for a long time in one of the tournaments I played against it. Furthermore, I prioritise creating alliances which will let me gun for those safe areas off the bat – e.g. trying to work with France as Germany. This is kind of a scaled-down version of Diplodocus’ alliance behavior. I still don’t play the full opening that Diplodocus used as Germany, because doing so is going all in on that alliance in a way that feels uncomfortably committal to me. It’s an approach humans get trained out of as they get more experienced with the game. […] You learn that actually, it’s better to hang back, figure out who appears to be most friendly toward you, and then go in alongside that person; that way your performance is likely to be much more consistent. Diplodocus flipped that on its head and went all-in in a way that made a strong pitch for its preferred alliance at the same time. […] It got very good results from it, and I suspect that experienced human players (myself included) could improve their average by adopting it – but I hate the idea that I could be throwing away a game I’d otherwise do very well in by doing this, even though it might improve my results in others.

Zijlstra observed a different form of strategic innovation in Cicero, Meta AI’s agent for the full game of Diplomacy ( Bakhtin et al., 2022a ) . Because Cicero had to negotiate using natural language, it learned to exploit features of the communication phase. Although it was trained not to communicate false plans, it sometimes withheld messages or omitted information when an honest response would have been strategically disadvantageous.

Cicero’s messages were set to be aligned with its plans, meaning it generally would not lie about what it was doing. This was for a good strategic reason – lying in Diplomacy is something that has to be done sparingly to be effective since each time you do it, the other player will be less likely to trust you in future. That meant when it did not intend to work with a player – either by stabbing said player in the back, or just by doing something that the player would clearly be annoyed about – it just wouldn’t message that player once it had decided on its plan. The process there would be that it would generate a set of message candidates which were honest about those intentions, and then all message candidates would be rejected by the filter checking for whether it was strategically advantageous to send those messages. This also happened when it proposed a plan that it very much wanted to go with, but its ally rejected said plan and proposed an alternative that was suboptimal for Cicero – it sometimes wouldn’t message them back. Particularly in the case where it was still working with the player but had just made a move it knew they wouldn’t be happy about, Cicero would quite often then begin the Press (aka discussion) maneuver in the next phase with something along the lines of “Sorry, I didn’t see your message in time, otherwise I would have changed my orders”, presumably because in training data this is something often said after a span of not saying anything. This worked surprisingly well (when it did not use it multiple times with the same player, anyway) and the player would often excuse the move as an understandable mistake, and continue working with Cicero, with Cicero in a much better position than it would have been otherwise.

Cicero and Diplodocus each learned to exploit various aspects of Diplomacy. In both cases, though, these reinforcement-learning-trained models learned strategies that, while effective, were high-risk: Diplodocus by committing to a strategy that depends on how other players act, and Cicero by taking actions that, if used too many times, can burn an opponent’s trust and willingness to work with the agent.

Part of what makes these strategies surprising is that both systems were trained with substantial human priors, yet still discovered strategies that human experts regarded as unusually risky, showing how RL favors strategies with high long-term expected reward, whereas human decisions under risk often rely on heuristics that give greater weight to worst-case outcomes ( Homma and Takezawa, 2024 ; Brandstätter et al., 2006 ; Zhang et al., 2014 ) . One consequence of this focus on expected reward is that the learned solutions offer higher expected utility, but leave little margin for adverse outcomes. As Zijlstra’s comments illustrate, human experts may maintain a wide safety margin to achieve more consistent outcomes, whereas the RL models discover that higher expected utility can be found by operating directly on this margin. As seen here (and in Section 3.2 with Libratus using overbets), the result is that agents can learn playstyles that appear uncomfortably risky to human experts.

### 3.4 Opponent Shaping Leads to the Spontaneous Rediscovery of Game-Theoretic Strategies ( Lu et al., 2022 ; Foerster et al., 2018 )

The final anecdote in this section steps back from the large, highly engineered systems discussed previously to comparatively small-scale multi-agent learning experiments at the intersection of game theory and reinforcement learning, asking whether similarly surprising strategic phenomena appear even in a much simpler repeated-game setting. Researchers used RL algorithms to study repeated games ( Aumann, 1959 ) —situations where the same players face each other over and over again. In a single game, it often makes sense to be selfish; however, in a repeated game, players must weigh their immediate gains against the risk of future retaliation by the opponent.

Learning in multi-agent environments ( Lanctot et al., 2017 ) is complex because the presence of other learning agents makes the environment non-stationary ( Amato, 2025 ) : as one agent’s strategy evolves, it changes the optimal strategy for other agents in the environment. Typically, agents are trained to improve their own policies myopically, without considering how their actions will influence the future learning behavior of others. In an attempt to address this shortcoming, researchers explored algorithms designed to enable agents in multi-agent interactions to anticipate and influence their opponents’ learning processes. Specifically, in the paper introducing LOLA (Learning with Opponent-Learning Awareness; Foerster et al., 2018 ), agents explicitly accounted for how their actions affected their opponents’ policy updates by having an explicit model of their opponent. While researchers initially observed that independently trained agents in the Iterated Prisoner’s Dilemma continuously defected, they were surprised when the LOLA agents instead spontaneously (re)discovered the cooperative strategy known as tit-for-tat ( Axelrod and Hamilton, 1981 ) , where the agents first cooperate and only defect if the other player betrays them.

Similarly, in their follow-up work on M-FOS (Model-Free Opponent Shaping; Lu et al., 2022 ), the researchers introduced a framework enabling agents to shape opponents’ learning behavior without an explicit model of the opponent’s behavior (hence, model-free). Remarkably, the M-FOS-trained agents independently discovered sophisticated strategies such as zero-determinant extortion ( Stewart and Plotkin, 2013 ; Hao et al., 2015 ; Press and Dyson, 2012 ) , which had only recently been identified by human theorists within the last fifteen years. The emergence of such complex strategies from model-free RL methods was unexpected and notable to the researchers.

LOLA rediscovered the foundational game-theoretic algorithm of tit-for-tat ( Axelrod and Hamilton, 1981 ) , while M-FOS independently discovered a solution that human game theorists had only recently identified ( Stewart and Plotkin, 2013 ; Hao et al., 2015 ; Press and Dyson, 2012 ) . In a slightly different timeline, the M-FOS ( Lu et al., 2022 ) discovery could easily have been recorded as a machine-first discovery rather than a rediscovery, as the two were separated by at most a decade, and M-FOS had no prior knowledge of the strategy. In essence, the capacity for strategic innovation is not confined to a special set of marquee benchmarks or internet-scale models: even modest RL agents in simple environments can surface ideas that theory is only just beginning to catalog.

### 3.5 Takeaways

The superhuman and optimal-performance stories highlight the upside of pointing powerful optimizers and algorithms at well-specified goals: they can uncover strategies and structures that eluded experts for decades or even centuries. Section 3 ’s anecdotes demonstrate examples of agents achieving superhuman performance on narrowly defined tasks. We see that in both AlphaGo ( Section 3.1 ) and M-FOS ( Section 3.4 ), agents are able to rediscover well-known strategies in their respective games. Reward-maximizing agents are even able to invent entirely new strategies that are surprising and counter-intuitive to expert players, as we see in AlphaGo’s Move 37 ( Section 3.1 ), Libratus’s overbetting strategy ( Section 3.2 ), Diplodocus’s subtle approach to signaling intentions ( Section 3.3 ), and the unusual Diplomacy strategy of completely abandoning one’s home territory ( Section 3.3 ). These examples provide definitive proof that AI is capable of surpassing human performance on constrained problems, and suggest that some forms of creativity, like innovative strategies, can be invented via straightforward optimization of environment rewards. In these histories, the model’s creativity led to surprising and effective strategies that expert players and AI researchers did not anticipate.

Indeed, while this section has highlighted the triumphs of deep reinforcement learning—where agents learned to generate surprising, superhuman insights—the focus on these successes can obscure a critical challenge: aligning an algorithm’s objective with human intent. When the objective captures only a rough proxy for what we actually care about, the same creativity displayed by AlphaGo and Libratus can instead manifest as reward hacking, with agents satisfying the literal objective while violating the spirit of the task ( Skalse et al., 2022 ; Krakovna et al., 2020 ) .

## 4 Reward Hacking

Reward hacking, also known as specification gaming ( Krakovna et al., 2020 ) , occurs when an agent discovers an effective but unintended way to obtain reward ( Skalse et al., 2022 ; Amodei et al., 2016 ) . It often reflects the agent optimizing the objective exactly as specified, while exposing a mismatch between that specification and the behavior its designers intended.

Reward hacking typically falls into two broad, overlapping categories. In the first, the reward signal itself is flawed: the agent maximizes the stated objective through behavior that is valid under the scoring rules but inconsistent with the underlying goal. In the second, the learning environment is insufficiently constrained: the reward logic may be reasonable, but the agent discovers glitches, omissions, or unintended interactions that allow it to bypass the challenge. Addressing the former generally requires revising what the agent is rewarded for, whereas addressing the latter usually requires modifying the learning environment to fix bugs or enforce constraints that were previously left implicit. Reflecting this distinction, Section 4.1 focuses on the exploitation of reward and score functions, while Section 4.2 examines cases in which agents exploited bugs in or escaped the intended learning environment. Because reward definitions and environmental constraints jointly specify a task, the boundary between these categories is necessarily fuzzy.

### 4.1 Exploiting Reward and Score Functions

As shown in Section 3 , reinforcement learning researchers often develop and evaluate their algorithms using games. Games are inexpensive to simulate, can often run faster than real time, and provide practical testbeds for cognitive capabilities such as perception, planning, navigation, and multi-agent reasoning ( Schaul et al., 2011 ) . Moreover, many games are carefully refined over years to guide human skill acquisition through their mechanics, balance, feedback, and scoring systems ( Jaccard et al., 2021 ; DeWitt et al., 2004 ; Plass et al., 2015 ) , making them appealing testbeds precisely because they are designed to elicit and reward complex behaviors in humans that we would also like artificial agents to learn.