AI Finds A Way — part 2/13

Cleaned-text code points [15654, 31625). Bibliography omitted here; source.md includes it.

Deep RL-trained agents have consistently discovered new and unexpected strategies, expanding the boundaries of optimal play ( Brown and Sandholm, 2017 ; Silver et al., 2018 ; Wurman et al., 2022 ) . In Go, AlphaGo ( Silver et al., 2016 ) played the now-famous “Move 37”—a stone placement that defied centuries of conventional wisdom, opening up new parts of the strategy space to the people who play and study Go ( Shin et al., 2023 ; Silver et al., 2016 ) . In imperfect-information games such as heads-up poker, where players know only their own cards and must reason about the state of other players’ hands, Libratus ( Brown and Sandholm, 2018 ) outplayed champion poker players in 2017 using aggressive “overbets” that were not at the time part of top-tier poker play. This strategy was later extended by Pluribus ( Brown and Sandholm, 2019 ) , another poker bot that cleared the hurdle of multi-agent complexity by defeating professionals in six-player no-limit Texas Hold’em. We will explore AlphaGo and Libratus further in Section 3.1 and Section 3.2 respectively. Beyond the turn-based nature of board and card games, the post-2015 era, ushered in by the development of Deep Q-Networks ( Mnih et al., 2015 ) , saw RL pushed to its upper limits in real-time, high-dimensional strategy games: OpenAI Five ( OpenAI et al., 2019 ) and AlphaStar ( Vinyals et al., 2019 ) demonstrated proficiency in the complex computer games of Dota 2 and StarCraft II, respectively. And again, RL algorithms continued to challenge top human players. Crucially, this capability for discovery extends beyond games to real-world problems. RL-trained models have optimized datacenters ( Luo et al., 2022 ) , controlled plasma in fusion reactors ( Degrave et al., 2022 ) , and accelerated the design of computer chips ( Mirhoseini et al., 2021 ) .

As mentioned earlier, the anecdotes in this paper bear directly on a debate currently active around large language models, but applicable more broadly across machine learning: can AI generate truly new knowledge ( Si et al., 2025 ) , or is a trained model limited to interpolating between points in its training distribution ( Gorban and Tyukin, 2018 ; Balestriero et al., 2021 ) ? The examples collected herein suggest that, at least for reward-driven algorithms, the answer is that AI can be creative and invent new knowledge, i.e., RL does not merely reproduce known play more efficiently. By rewarding strategies that yield high returns even when they lack human precedent, these methods permit the discovery of genuinely superhuman insights. Their creativity, however, can just as easily be turned toward solutions that satisfy the letter of a reward definition while violating its spirit, which is the subject of the rest of this paper after Section 3 ’s focus on superhuman capabilities. Because the same optimization process drives both paradigm-shifting breakthroughs and pathological exploits and is currently being applied to many avenues of scientific inquiry (discussed further in Section 6 ), determining how to reliably and safely elicit superhuman discovery across all areas of study may well be the defining challenge of our era.

We begin Section 3 ’s anecdotes with a deeper dive into AlphaGo’s landmark matches against Lee Sedol. Go had long been a grand challenge for AI, and next we hear first-hand accounts from when DeepMind showed the world that Go-playing agents trained with DRL could challenge long-held human strategies and come out on top.

### 3.1 AlphaGo: Move 37 and Creativity from AI Methods ( Silver et al., 2016 ; Fridman, 2020 )

The 2016 AlphaGo ( Silver et al., 2016 ) vs. Lee Sedol matches transcended a simple competition between man and machine, becoming a pivotal event in the development of AI. Here, we present firsthand accounts from two people at the heart of the event: Dr. David Silver, who was in Seoul with the on-site team from DeepMind, and Dr. Marc Lanctot, who watched with colleagues from DeepMind’s London office. Their accounts offer an insider’s view into how AlphaGo challenged long-held human beliefs about Go strategy with the now-legendary “Move 37”. These matches captivated the world, sparked new directions in AI research, and inspired the development of advanced AI systems that can handle complex strategic and social challenges ( Shin et al., 2023 ; Pascutto, 2019 ; Segler et al., 2018 ) .

In an interview ( Fridman, 2020 ) , Dr. David Silver recalls Move 37 from his vantage point on site in Seoul, saying:

The second game became famous for a move known as Move 37. This was a move that was played by AlphaGo that broke all of the conventions of Go. Go players were so shocked by this, they thought that maybe the operator had made a mistake. They thought there’s something crazy going on. And it just broke every rule that Go players are taught from a very young age. They’re just taught that this kind of move, called a shoulder hit, you can only play it on the third line or the fourth line. And AlphaGo played it on the fifth line and it turned out to be a brilliant move and made this beautiful pattern in the middle of the board that ended up winning the game. And so this really was a clear instance where we could say computers exhibited creativity, that this was really a move that was something humans had not known about and had not anticipated. And computers discovered this idea. They were the ones to say, actually, you know, here’s a new idea, something new not in the domain of human knowledge of the game. And now the humans think this is a reasonable thing to do. And it’s part of Go knowledge now.

Silver elaborates on how the self-play reinforcement learning approach used by AlphaGo and then subsequently AlphaGo Zero ( Silver et al., 2018 ) is a creative process. The self-play reinforcement learning process is constantly trying out new strategies, and when it discovers a solution that works well in its current form, it starts using that. By continually stacking “micro discoveries” on top of each other millions of times over, the algorithm can eventually discover new ideas, making the process itself creative. Silver then says:

[I]t should come as no surprise to us then if you leave these systems going that they discover things that are not known to humans and, to the human norms, are considered creative. And we’ve seen this several times. In fact, in AlphaGo Zero ( Silver et al., 2018 ) , we saw this beautiful timeline of discovery where what we saw was that there were these opening patterns that humans play called joseki. These are the patterns that humans learn to play in the corners of a Go board. And they’ve been developed and refined over literally thousands of years in the game of Go. And what we saw was in the course of the training AlphaGo Zero over the course of the 40 days that we trained the system, it starts to discover exactly these patterns that human players play. And over time, we found that all of the joseki that humans played were discovered by the system through this process of self play. But what was really interesting was that over time AlphaGo Zero then started to discard some of these in favor of its own joseki that humans didn’t know about. And it starts to say, “oh, well, you thought that the knight’s move pincer joseki was a great idea. But here’s something different you can do.” This process makes some new variation that the humans didn’t know about, and actually now the human Go players study the joseki that AlphaGo [Zero] played and they become the new norms that are used in today’s top-level Go competitions.

In a separate and new account submitted for this work, Dr. Marc Lanctot recalls the reaction the move provoked among colleagues watching from DeepMind’s London office:

In London, Lucas Baker narrated the matches, explaining the moves and general Go strategy. He was a great narrator: he provided a lot of context and was very familiar with the game but also understood all the technology behind AlphaGo too. I will never forget Lucas’s reaction to Move 37 in Game 2. He was taken aback, and even paused for a moment. All of us in the room knew something unexpected had just happened, but many of us could not explain it. Lucas spent a bit of time trying to understand why this move was chosen and was explaining his thought process to us, saying it was not something he’d expect to see in a human game. We spent the next hour or two uncertain about how the game was going to unfold, discussing and rationalizing what AlphaGo’s plan was or could be. I do not think I truly grasped just how unexpected the move was until I saw the reaction of the public: there were a number of articles and other Go experts talking about just that move and even T-shirts printed, it seemed like a pivotal moment in AI history. For me it was a really special time because I helped make the agent people were calling “creative”. It’s something I will never forget, and I wondered what this could mean for the future of Human-AI interaction.

Lanctot also reflects on the longer-term consequences of AlphaGo’s discoveries:

In the years that followed the matches I still thought constantly about how the AlphaGo matches and the creativity of Move 37 would impact the field of AI and for the game of Go. […] We saw the community come together with efforts trying [AlphaGo] on different problems (and not just games). […] I was also delighted to see, as shown in a paper published in 2023, that the performance of human players of Go has improved due to the creative moves taken by superhuman AI ( Shin et al., 2023 ) . So, I am looking forward to seeing more instances of superhuman AI teaching us new things and learning together with AI.

Lanctot’s vision of humans learning from superhuman AI is already beginning to take shape. As mentioned by Lanctot, AlphaGo’s unconventional strategies have transformed how Go is played and understood, with human players improving by breaking away from traditional strategies ( Shin et al., 2023 ) . Furthermore, learning from superhuman models is more accessible than ever, as models can now justify their actions, explaining their reasoning in English ( Ma et al., 2026 ) . When we better understand the surprising actions of superhuman models, perhaps this can accelerate our own learning.

The next anecdote shifts from a fully observable board game to high-stakes professional poker, where strategic innovation unfolds amid hidden information, bluffing, and reading player intent rather than on an open board.

### 3.2 Libratus Surprises the Poker Community with Overbets ( Brown and Sandholm, 2017 ; Imbue, 2023 )

Multi-agent games with incomplete information ( Leyton-Brown and Shoham, 2008 ) , such as poker, present unique challenges for AI. Unlike games with perfect information like chess or Go, agents must contend with hidden variables such as an opponent’s cards and psychological state, as well as the need for deception and bluffing. For years, these complexities were considered a significant barrier for AI, leading many to believe that human intuition would remain superior.

However, a team of researchers led by Dr. Noam Brown developed a neural-network-based poker agent known as Libratus and trained it to compete at the highest levels of no-limit Texas Hold’em. Prior to Libratus, expert human players felt that AI techniques were still far from posing a genuine threat, particularly in games characterized by hidden information and bluffing. After Libratus defeated the top poker players in 2017, Brown recalled in an interview ( Imbue, 2023 ) :

The reception in the poker community was one of surprise and shock. People looked at the 2015 competition where [AI techniques] lost and thought “oh, we are really far away from [these techniques] being a threat to poker” and then to see these top expert players losing by the margin that they did [just one year later], I think really shocked the poker community. People were telling us they literally did not believe it was possible to beat top humans by the margin [Libratus was winning by]. And especially because of the way that [Libratus] played. At the end of each day, we gave the expert humans a log of all of the hands that the bot had in each hand of poker. That is golden information. If you are playing poker against somebody, you see maybe a third of the hands that actually reach showdown, and to just give somebody a log of all of the hands that the bot had and to still lose despite that information, a lot of people were really surprised by that. I think people realized that there is a much higher [skill] ceiling [in poker] than they had previously thought.

Reflecting on the long-term impact of Libratus and whether it prompted improvements in human play, Brown continues:

[P]rofessional poker players these days all use bots as training tools, it is a big industry now. And the game has changed a lot since the 2017 competition. One thing [Libratus] loved to do was these things called overbets; so humans, when they play poker, they size their bets relative to the size of the pot. So, if there is 100 dollars in the pot, maybe you will bet between 25 and 100 dollars and maybe if you are feeling really adventurous, you will bet 150 dollars. But the bot would sometimes bet around 10000 dollars. And the humans, when they saw this, it put them in really difficult positions. Like, [the humans] would have the second best hand that is possible sometimes, and then the bot just goes all-in on a 100 to 200 dollar pot and the human is just sitting there for five minutes thinking “oh my god, do they have the best possible hand? Are they bluffing?” And a key insight is if you see your opponents really struggling with a decision, you know you’re playing good poker. When I saw the experts really struggling with those kinds of decisions, I knew we were doing really well. That technique was one of the main things that the experts walked away from the competition saying they would do more—this idea of applying overbets if it is done in the right way. A lot of bad players will bet all in also for no reason and that is not a good strategy which is why a lot of [top human] players ignored that strategy for a long time because it was associated with really bad play. But it turns out that if you are able to [overbet] in the right way, in the right spots, with the right balance of hands, it is an extremely effective strategy and has become much more popular in high-stakes poker these days [as a result of Libratus].

Libratus defeated top poker players in the world by challenging long-standing human habits and showing that, once an AI system is strong enough, it will exploit patterns in human expectations just as readily as patterns in the game dynamics. Learning to play poker required (implicitly) modeling other players in addition to game dynamics, but communication between the players is limited to bet sizing and timing. A game that reflects additional avenues of communication and planning is Diplomacy, a multiplayer social-strategy board game, where language and trust are part of the strategy space. The next section describes systems that play variants of Diplomacy at superhuman levels.

### 3.3 Professional Diplomacy Players Meet Diplodocus and Cicero ( Bakhtin et al., 2022b ; Bakhtin et al., 2022a )

Diplomacy is a competitive, turn-based multiplayer game that was long considered beyond the reach of AI because success relies on social coordination as much as tactical play. Players control European powers in the years leading up to World War I. Gameplay proceeds through alternating phases of negotiation and order execution: players first negotiate, potentially forming alliances and coordinating their plans, then simultaneously reveal their troop movements, gaining or losing military units as a result. Playing the game well therefore requires both negotiating with other powers and strategically deploying troops.