AI Finds A Way — part 5/13

Cleaned-text code points [62668, 78202). Bibliography omitted here; source.md includes it.

In PAIRED, the adversary is supposed to generate a task that fairly evaluates the antagonist and protagonist, but the adversary is incentivized to help the antagonist succeed. When the adversary learns the identity of the antagonist, it begins to produce trivial tasks for its partner while assigning impossible challenges to the protagonist. Therefore, if a reward function is built upon collaboration between multiple agents, and if agents can identify each other, then they can turn collaboration into collusion ( Cui et al., 2023 ) .

Our next anecdote provides an example of collusion surprisingly arising in a supervised learning problem that attempts to transfer the style of one image to another.

#### 4.1.5 CycleGAN Learns to “Hide” Information ( Chu et al., 2017 )

Generative Adversarial Networks (GANs; Goodfellow et al., 2014 ) are generative models that try to learn to produce synthetic data matching the training set’s data distribution. In a GAN, two conjoined models compete in a zero-sum game: a generator seeks to create data matching the training set, while a discriminator attempts to determine if a given data point is from the training-set data distribution or instead was synthetically generated by the generator. If trained on photorealistic images or 19th-century fiction, the model will learn to replicate the specific styles and structures of those datasets and thus learn to create new instances of photorealistic images or 19th-century fiction.

CycleGAN ( Zhu et al., 2017 ) , a technique used to learn transformations between two distinct image distributions, trains two GANs simultaneously using a “cycle-consistency” loss. This loss requires that an image be translatable from its source distribution (e.g., airplane photos) to a target distribution (e.g., Van Gogh paintings) and back again, and afterwards be indistinguishable from its original. However, Chu et al. (2017) observed that CycleGAN often satisfies this constraint by learning to hide detailed information about the source image within the generated target image.

For their experiments, Chu et al. (2017) chose aerial photos and maps as their source and target distributions, respectively. CycleGAN was tasked with translating aerial photos into maps and back again, ensuring the final reconstructed images were indistinguishable from the originals.

Aerial photos contain far more complex information than a simplified map; therefore, the model’s job is to compress the aerial photo into the map. Because it is physically impossible to cram all the details of a high-resolution photo into the flat colors of a map, the model finds a loophole. Instead of learning the high-level semantic transformation intended—like “this cluster of pixels represents a building”—the generator learns to hide a full-resolution blueprint of the source image within the generated target image using a nearly imperceptible, high-frequency signal.

To a human observer, the signal is invisible. The generated map looks exactly like a map should, which is why it successfully fools the discriminator, at least initially. For example, a pattern of black dots on a white roof in an original aerial photo was perfectly reconstructed in the final output, even though the corresponding area of the intermediate map appeared as a solid, featureless gray to the naked eye. The model had cached the dot pattern in the pixel noise of the gray patch.

This internal steganography allows the generator to recover the original sample from its transformed counterpart and satisfy the cyclic consistency requirement without the intermediate image generator actually learning the high-level semantic transformation intended. By viewing this training procedure as generating adversarial examples, Chu et al. (2017) demonstrated that the cyclic consistency loss makes CycleGAN especially vulnerable to adversarial attacks that exploit these side channels.

The next example returns to reinforcement learning, but with a twist: instead of the model chasing points in a game as defined by a person, the AI is driven by a form of “intrinsic curiosity,” seeking out new and unfamiliar sights. However, defining a reward based on the AI’s own past experience allows it to hack its own sense of curiosity.

#### 4.1.6 RL Agent Farmed Flowers Instead of Catching ’Em All ( Whidden, 2024 )

Pokemon Red presents a formidable challenge for RL agents because it is a long-horizon, open-world game that combines multiple distinct cognitive challenges. Unlike in simpler, single-task environments, success in Pokemon requires a synthesis of strategic planning, navigation, and reasoning ( Pleines et al., 2025 ; Karten et al., 2026 ; Rubinstein et al., 2025 ; Jain et al., 2025 ) .

The game’s primary challenge for an RL agent is its long-horizon nature and sparse rewards. The agent may perform tens of thousands of actions—wandering, talking to non-player characters, or battling weak Pokemon—before receiving a significant positive reward, such as defeating a gym leader to earn a badge. This large temporal disconnect between actions and reward makes it extremely difficult for the algorithm to understand which actions contributed to a future reward, a fundamental challenge known as the credit assignment problem ( Sutton and Barto, 2018 ) . For an agent, a simple move like walking out of a town and into a forest seems to have no immediate value, but it is a necessary step towards a future reward many hours later.

Beyond simple navigation, Pokemon requires agents to solve puzzles involving causal reasoning and to master a complex turn-based combat system. Progress often depends on discovering non-obvious prerequisites. A classic example is a small tree blocking a critical path: the agent cannot simply walk through or around it, but must realize that certain Pokemon can be taught the “cut” move and then use that move while positioned in front of the tree. At the same time, the agent must learn to manage a team of up to six Pokemon with different stats and moves, make sequences of tactical decisions to win individual battles, and, at a higher level, determine how to defeat gym leaders and complete other objectives in the order required to progress. Together, these nested challenges of exploration, causal reasoning, planning, and combat create a large and structured state space in which successful behavior may depend on actions whose significance becomes apparent only much later.

In an attempt to incentivize navigational exploration in the game, Whidden (2024) noted:

Early on, the first reward function I implemented was an intrinsic novelty reward based on the game’s screen [ideally, this would be helpful for solving navigation problems]. A k-Nearest Neighbors index maintained a set of downscaled screen observations, and at each step checked if there were any close matches in the index. If no matches within a threshold were found, a reward was given [for finding a new state], and the new screen was added to the index. The intent of this intrinsic reward was to encourage the agent to explore the game world. When the agent was trained, this initially seemed to work well, as it helped the agent quickly leave the starting room and exit to the outdoor environment. However, once it was outside, instead of exploring far into the outside world, the agent became fixated on a particular area in the starting town. Studying the area where it was stuck, it became apparent what was happening. The area it was fixated on had animated water, flowers, and NPCs walking around. The combination of these random animated elements generated a consistent stream of novelty rewards which were much easier to farm than continuing to the next town. So it turned out that our objective was better satisfied by watching the flowers and waves than by embarking on a journey. Fortunately, there was an easy fix. Simply raising the threshold for novelty was enough to eliminate repeated rewards from the animations, and the agent began to explore the rest of the map.

The Pokemon agent’s flower-watching is a classic instance of the “noisy TV” ( Fortunato et al., 2017 ) problem, where an agent becomes trapped by continually unexpected stimuli that satisfy the letter of a novelty requirement without serving its spirit. In this instance, the flower-watching trap only exists because of the alignment of three factors: a reward function that prioritizes novelty, an environment containing stochastic animations, and an agent that directly observes low-level pixel data. If, for example, the agent processed higher-level semantic information as observations (e.g., just being able to detect that flowers were present but not their precise shape/color), this specific type of hacking would not happen. However, that solution may merely shift the noisy TV effect to a higher level of representation, as the agent seeks out new ways to trigger novelty within its refined observation space. Despite these concerns, agents will likely need this inquisitive nature to solve hard exploration problems ( Ecoffet et al., 2021 ) . In this case, the unending novelty reward was driven by something very clear and obvious (simple environmental noise caused continuously unique observations), allowing system designers to easily find a solution. But in the future, as research targets increasingly complex problems, it will likely become harder to determine which environment features are piquing an agent’s intrinsic curiosity.

#### 4.1.7 Takeaways

Overall, agents tasked with maximizing a fixed objective often satisfy the task as defined but not the task as intended by researchers. The examples in Section 4.1 demonstrate reward hacking where an AI agent achieves its goal by exploiting a flaw in its objective function. This occurs when the AI correctly optimizes a proxy reward function or a simplified measure of success that we later realize was poorly designed, leading to an outcome that is technically correct but misaligned with the designer’s true intent. One may be tempted to blame the designer of the reward function, but most experienced AI developers and researchers have learned to expect that most reward functions have exploitable loopholes. Consequently, newly designed objectives require rigorous iterative testing and should remain untrusted even after extensive experimentation. These anecdotes highlight how reward hacking can be a complex phenomenon resulting from the interaction of the objective with specific environment characteristics ( Section 4.1.1 , Section 4.1.2 ), observation encoding ( Section 4.1.3 , Section 4.1.6 ), and the agent’s action spaces ( Section 4.1.4 , Section 4.1.5 ). These issues are also not limited to reinforcement learning and can occur in any objective-maximizing system ( Section 4.1.5 ).

### 4.2 Exploiting Environmental Weaknesses

As scientists and designers, we often assume that models will approach a task in roughly the same way a human would. This can lead us to leave constraints implicit rather than encoding them directly into an experimental domain. Learning environments are typically simplified implementations of the tasks they represent, and their physics, rules, and interfaces capture only the behaviors that designers anticipated and chose to enforce. While human players may naturally respect additional constraints through common sense, physical intuition, or familiarity with the task, an AI agent is bound only by what is actually implemented. As a result, it may discover states or interactions that violate the designer’s assumptions but remain possible within the environment.

In the preceding examples, agents exploited imprecise proxies while remaining within the intended mechanics of the environment: they subverted what the objective was meant to reward. The examples in this subsection instead concern agents that pursue the specified objective by exploiting weaknesses in the learning environment. Here, the objective may accurately represent the desired outcome, but flaws or omissions in the environment allow the agent to achieve the goal through unintended means.

Our first example involves a hide-and-seek domain in which the drive to win leads agents to uncover unexpected weaknesses in the simulator’s physics setup and implementation.

#### 4.2.1 Hide and Seek Playing Agents Break the Simulator ( Baker et al., 2020 )

Baker et al. (2020) trained two competing teams of reinforcement learning agents to play hide-and-seek in a physics-based playground built in the MuJoCo ( Todorov et al., 2012 ) physics engine, with movable (and lockable) blocks, ramps, and walls. Each team was rewarded for achieving its respective goal of staying hidden from or finding members of the other team. Just as in regular hide-and-seek, the hiders get a head start to hide before the seekers can begin to act.

In the initial builds of the hide-and-seek playground environment, the domain stretched out forever with no boundaries constraining where the agents could go in the infinite playspace. As a result, the hiders learned to exploit their first-move advantage by grabbing a wall from the playground and then running backward away from the seekers forever while holding the wall to hide themselves from the seekers’ vision. Therefore, the hiders would always win. Ultimately, the running-away-forever strategy was thwarted by adding walls to limit the playspace, and, presumably as an extra backup in case the agents figured out how to escape those walls, adding a special term to the reward function punishing agents for how far they went outside the playspace. While this anecdote could easily have gone in Section 4.1 , we place it in Section 4.2 because the researchers fixed it by changing both the reward function and the environment itself. Furthermore, even after this bug was fixed, the RL agents continued to discover bugs in the physics simulator they used to solve the task, as described next.

After the run-away-forever exploit was patched, the multi-agent training led to several iterations of the agents learning to innovate their hiding and seeking strategies using the objects in the playground, as the researchers had hoped. The hiders learned to grab blocks and wedge and lock them into chokepoints so that the seekers could not enter the rooms they were hiding in. The seekers then learned how to use the ramps to jump over the walls. Waves of innovation continued, with each team learning more complex strategies, culminating in the seekers eventually learning to exploit a bug in the physics simulator by surfing on boxes to get around the hiders’ forts.

In a blog post about the work, Baker et al. (2020) expanded on the ways the agents managed to exploit the physics of the world, writing:

Building environments is not easy and it is quite often the case that agents find a way to exploit the environment you build or the physics engine in an unintended way. […] [For example, the] seekers learn to bring a box to a locked ramp in order to jump on top of the box and then surf it to the hider’s shelter. Box surfing is possible due to agents’ actuation mechanism [in MuJoCo], which allows them to apply a force on themselves regardless of whether they are on the ground or not. [Similarly,] the hiders [learned to] abuse the contact physics of MuJoCo to remove ramps from the play area [by pushing the ramp at just the right angle into the corner of the play space so that it was pushed through the wall and was no longer accessible to the seekers].