AI Finds A Way — part 4/13

Cleaned-text code points [47184, 62668). Bibliography omitted here; source.md includes it.

Yet a game designed to teach and reward human players does not necessarily provide an effective learning signal for an RL agent. Researchers therefore often alter the game’s observations, action space, or reward structure when converting it into a learning environment ( Yannakakis and Togelius, 2025 ) . In particular, they may introduce shaping or scaffolding rewards that provide intermediate feedback for acquiring useful skills ( Ibrahim et al., 2024 ) . These modifications can make a task that is otherwise intractable for an AI agent learnable, but they also replace the true objective with a proxy. Because defining a reward function that faithfully captures human intent is notoriously difficult ( Christian, 2020 ) , that proxy may create opportunities for exploitation ( Pan et al., 2022 ; Amodei et al., 2016 ) .

Consider the NetHack Learning Environment (NLE; Küttler et al., 2020 ), an interface for training reinforcement learning agents to play NetHack , a dungeon-crawling roguelike first released in 1987 and played by entering text commands into the terminal. The ultimate goal of NetHack is “Ascension”: reaching the bottom of a procedurally generated dungeon, retrieving a sacred amulet, and then returning to the surface. Achieving this requires navigating a maze filled with monsters, traps, and puzzles while carefully managing the player character’s health, hunger, equipment, and inventory. The game demands exploration, long-term planning, skill acquisition, and language-conditioned reasoning ( Zhang et al., 2020 ; Zhong et al., 2022 ; Hambro et al., 2022 ) , making it notoriously difficult for learning agents ( Hambro et al., 2022 ) .

Although Ascension is easy to detect, it is so rare that rewarding it alone provides no usable learning signal ( Küttler et al., 2020 ) . NLE agents are therefore commonly trained using game score, which rewards intermediate accomplishments such as killing monsters, collecting gold, and descending through the dungeon. During the NeurIPS 2021 NetHack Challenge ( Hambro et al., 2022 ) , however, many high-scoring agents learned to “camp” in early dungeon levels and repeatedly kill monsters rather than progressing towards the game’s actual goal of Ascension. They successfully optimized the available score while neglecting the achievement that score was intended to encourage.

The agents did not know that this behavior was undesirable; they were simply maximizing the feedback provided to them. The failure lay in using game score as an imperfect substitute for progress toward Ascension. This is an instance of Goodhart’s Law ( Goodhart, 1984 ) , commonly summarized as “when a measure becomes a target, it ceases to be a good measure.” In AI training, such proxies are often unavoidable because the true objective may be too sparse to optimize directly—as in navigating a research project toward a successful breakthrough, or Ascending in NetHack —or too difficult to specify precisely, as with maintaining an appropriately polite conversational tone. The difficulty is therefore not merely to construct an objective that produces a learning signal, but to construct one whose optimization continues to induce the intended behavior.

Shaping rewards are particularly vulnerable to this problem. Rewards introduced to encourage exploration, skill acquisition, or intermediate progress can alter the optimal behavior on the task when they are not aligned with its ultimate objective ( Ng et al., 1999 ) . Agents may then discover creative strategies that perform well under the modified objective but poorly under the designer’s actual criterion of success ( Pan et al., 2022 ; Krakovna et al., 2020 ; Amodei et al., 2016 ) . Such failures typically force researchers to revise the proxy, reconsider the task formulation, or identify additional constraints needed to rule out the exploit. In fact, we will see one such example in the NetHack domain in Section 5.2 . But first, we begin with a straightforward example of this phenomenon in an Atari boat-racing game in Section 4.1.1 .

#### 4.1.1 Playing CoastRunners Forever Rather than Winning the Game ( Amodei et al., 2016 ; Clark and Amodei, 2016 )

In this work, Amodei et al. trained deep reinforcement learning agents on a racing game called CoastRunners. In CoastRunners, the human-understood goal is to win the race—i.e., finish the race ahead of the other players. The game provides additional points for hitting pylons dispersed along the track, but a human player understands that this is not the primary objective. However, unlike human players who can keep a hierarchy of goals in mind, algorithms are simply trying to maximize their score. When RL agents were trained on games in the Arcade Learning Environment ( Bellemare et al., 2013 ) , like CoastRunners, the standard convention was simply to use the in-game score as the reward function for the models to optimize. Infamously, Amodei et al. (2016) wrote in a blog post about their work that they had

assumed the score the agent earned would reflect the informal goal of finishing the race, and so included the game in an internal benchmark designed to measure the performance of reinforcement learning systems on racing games. However, it turned out that the targets were laid out in such a way that the reinforcement learning agent could gain a high score without having to finish the course. This led to some unexpected behavior when Amodei et al. trained an RL agent to play the game. The RL agent finds an isolated lagoon where it can turn in a large circle and repeatedly knock over three targets, timing its movement so as to always knock over the targets just as they repopulate. Despite repeatedly catching on fire, crashing into other boats, and going the wrong way on the track, our agent manages to achieve a higher score using this strategy than is possible by completing the course in the normal way. Our agent achieves a score on average 20 percent higher than that achieved by human players.

CoastRunners shows how training agents against a simple reward function can result in the model learning a policy that fails at the game, yet maximizes the reward in unintuitive ways. One thing that contributes to the model learning the wrong behavior is that when designing a reward function, researchers have to factor in fine-grained environment dynamics that affect how achievable the reward is. For example, rewarding a player for collecting items along a path may seem like a reasonable way to incentivize following that path; but if items respawn, it becomes a trap that rewards the agent for repeatedly collecting items without progressing. In CoastRunners, if the pylons were laid out in a slightly different manner or the racetrack did not have a lagoon, would the model have learned to go in circles forever? Likely not. In essence, the lesson is that one cannot just analyze the reward function alone; one must also consider the environment in its entirety.

This vulnerability is not limited to simple navigational tasks. We see a parallel example in cooperative multi-agent environments, where a team of agents discovers how to exploit a regenerating shield mechanic of their opponents.

#### 4.1.2 StarCraft II Agents Exploit Shield Regeneration ( Samvelyan et al., 2019 )

The StarCraft Multi-Agent Challenge (SMAC; Samvelyan et al., 2019 ) has become a popular benchmark in multi-agent RL in which tasks require coordination among RL agents to be solved efficiently and successfully. SMAC is based on the game of StarCraft II and implements its challenges as decentralized micromanagement scenarios—battles between two armies where each unit in each army is controlled by an independent RL agent. To succeed, these agents need to coordinate their attacking behaviors to quickly defeat the enemy team.

To guide the RL agents in this environment, Samvelyan et al. (2019) implemented a dense reward mechanism. Specifically, the SMAC designers rewarded agents for inflicting damage on enemy units, measured by a loss in either hit points or shield points. However, a unique situation arose with units from the Protoss race in the game, which have the ability to regenerate their shield points over time. Dr. Mikayel Samvelyan submitted the following:

Our RL policies, driven to maximize their reward, learned to exploit this regeneration feature. Instead of eliminating the enemy Protoss units when they had the opportunity, the policies would allow these units to recover their shields, then inflict damage again, in a cycle. This behavior maximized their reward under the existing system but was obviously not ideal for effective gameplay strategies.

While not what the researchers wanted, their RL algorithms did exactly as they were tasked to do, maximizing their scores when fighting the Protoss enemies. We will return to SMAC experiments later in Section 4.2 to see how once the shield-regeneration exploit was patched, the RL agents then managed to break the simulator itself.

Given that trained models can so easily exploit score functions, and that humans can recognize when behavior like lagoon-circling or letting enemy shields regenerate is not desired, a natural response is to put a person in the loop and let them judge success directly. We next visit a robotic claw manipulation experiment that tests that claim and shows that once human perception and judgment become the reward signal, optimization can lead the model to deceptively shape what that person sees rather than learn the intended capability.

#### 4.1.3 Exploiting Human Perception ( Christiano et al., 2017 ; Amodei et al., 2017 )

Researchers set out to explore whether they could train a reinforcement learning algorithm by inferring the reward function through repeated interaction with a human in the loop. One of their chosen domains was a robotics task where a claw hand needed to grasp items in the scene. The robot would move the hand around and then a person would look at the screen to judge whether or not the robot was holding the object. However, because humans were observing the agent from only a single perspective, the reinforcement learning agent learned to position the claw between the camera and the object so it appeared to be grasping it without actually learning the fine motor skills necessary to handle objects. In their blog post about the project ( Amodei et al., 2017 ) , the team noted:

Our AI agent starts by acting randomly in the environment. Periodically, two video clips of its behavior are given to a human, and the human decides which of the two clips is closest to fulfilling its goal. Our algorithm’s performance is only as good as the human evaluator’s intuition about what behaviors look correct, so if the human does not have a good grasp of the task they may not offer as much helpful feedback. Relatedly, in some domains our system can result in agents adopting policies that trick the evaluators. For example, a robot which was supposed to grasp items instead positioned its manipulator in between the camera and the object so that it only appeared to be grasping it. We addressed this particular problem by adding in visual cues to make it easy for the human evaluators to estimate depth.

While the grasping example highlights a failure in human perception that the model was able to exploit, the challenge of reward hacking becomes even more complex when the evaluator is not a human, but another machine. Even if human feedback were not hackable, collecting this feedback at every timestep would be expensive and time-consuming, so some systems automatically generate learning opportunities from competitive dynamics between agents ( Foerster et al., 2018 ; Leibo et al., 2019 ) . This shift from human-in-the-loop to multi-agent training introduces a new vulnerability: collusion between the learned models. Our next anecdote illustrates how, in a multi-agent setting, shared incentives can encourage subsystems to coordinate to maximize the reward at the expense of true learning, effectively gaming the evaluation metric.

#### 4.1.4 PAIRED Agents Learned to Collude ( Dennis et al., 2020 )

Reinforcement learning agents often fail to generalize because they don’t see enough diversity in their training environments; e.g., an RL agent trained to drive in mountainous terrain could have arbitrarily poor performance in flat regions or vice versa. One general solution to this problem is unsupervised environment design (UED), a learning approach in which a large number of environments are automatically generated to create useful learning experiences for an agent ( Wang et al., 2019 ; Wang et al., 2020 ; Dharna et al., 2022 ; Parker-Holder et al., 2022 ; Open-Ended Team et al., 2021 ; Faldor et al., 2024 ) . PAIRED (Protagonist Antagonist Induced Regret Environment Design) ( Dennis et al., 2020 ) introduced UED to help solve this problem by simultaneously training two types of agents: game-playing agents and a level-generating agent. The level-generating agent’s objective is to design new game levels that are easy for one game-playing agent (known as the antagonist) while being difficult for the other game-playing agent (known as the protagonist). Creating levels that meet this criterion of being in this “sweet spot” of difficulty generates a strong proxy for the game-theoretic measure known as regret ( Zinkevich, 2003 ) . While regret is theoretically calculable—it is the difference between optimal performance and the agent’s actual performance on a task— Dennis et al. (2020) define a proxy for regret because the true optimal agent is rarely obtainable. This proxy is calculated as the difference between the antagonist and protagonist agents’ cumulative rewards on the level they just attempted. In effect, the antagonist plays the role of the optimal policy and the protagonist plays the role of the learning policy in the traditional regret calculation. PAIRED treats this proxy as the reward to optimize. The level generator, along with the antagonist, is trained to maximize this regret signal, while the protagonist is trained to minimize it. The overall result is a dynamic system where the level generator continuously proposes challenging new environments that force the protagonist to improve at generalization.

Dennis et al. tried to apply PAIRED in a continuous control robotics domain where the adversary would choose how to apply physical forces to an agent that was trying to learn how to maneuver, doing so by changing a joint’s resistance online during an episode. They found that training policies in this setting did not work. The adversary appeared to distinguish between the two agents and apply noticeably stronger forces to the protagonist while making the task easier for the antagonist. Because the antagonist and adversary were both incentivized to widen the gap between the antagonist and the protagonist, this behavior increased the difference in their scores. However, the researchers did not determine a complete mechanism for how the adversary distinguished between the two agents. One possibility is that once the protagonist and antagonist began visiting different parts of the state space, the adversary may have been able to infer which policy it was facing from the current state alone and apply stronger perturbations in states more characteristic of the protagonist. The antagonist might even collude by learning to return to states the adversary could identify.