AI Finds A Way — part 6/13

Cleaned-text code points [78202, 93992). Bibliography omitted here; source.md includes it.

The hide-and-seek agents repeatedly found new and unexpected ways to exploit bugs in the physics simulator. However, the designers wanted the agents to come up with creative and interesting strategies similar to those that a human might try while playing the game (i.e., without exploiting flaws in the physics engine). The agents repeatedly finding new exploits shows why environment design is often an iterative process. The next anecdote shows how surprises emerge when agents bypass human-authored guardrails meant to define sensible behavior.

#### 4.2.2 Robotic Humanoid Walks Without Using Its Feet ( Batra et al., 2024 )

The bipedal humanoid walker is a benchmark robotics task where the goal is to teach a roughly human-shaped robot to walk forward as fast as possible ( Todorov et al., 2012 ) . In locomotion environments such as the bipedal walker, it is quite common when training quality-diversity ( Mouret and Clune, 2015 ) and reinforcement learning algorithms ( Sutton and Barto, 2018 ) to have a vertical termination height built into the task definition to restart the task after the agent falls over and can no longer easily make progress. This task parameter constrains the robot to moving within an acceptable vertical range, and if the robot’s torso drops below a certain value, a reset is triggered, ending the trial and returning the robot to the initial state.

This cutoff saves time and compute resources by terminating early when the agent is on its way to falling down. It also is designed to improve the learning efficiency and training stability of algorithms like Proximal Policy Optimization (PPO; Schulman et al., 2017 ). Without this guardrail, a robot might walk perfectly for ten seconds before falling and thrashing on the ground for the remainder of the episode, accumulating penalties for expending energy. This makes evaluating the earlier, successful actions difficult because the RL objective judges an action by the reward accumulated during the rest of the rollout. In other words, the clear signal of those initial good steps gets confounded by the noise of the subsequent random actions, dragging down the expected value of those early steps. With the guardrail, the rollout ends when the agent falls, and the algorithm is able to reinforce the behavior that led to the successful 10 seconds of walking. However, while these resets reduce variance and make learning easier, they also act as a filter that suppresses other functional solutions. This anecdote asks what happens when we remove this reset condition and allow optimization to explore the dynamics usually hidden behind the benchmark’s guardrails.

Because myopically chasing rewards can lead to agents not learning by getting stuck in local optima ( Lehman and Stanley, 2011 ; Norman and Clune, 2024 ) , Batra et al. (2024) instead set out to explore as many possible walking gaits as they could, regardless of their initial scores on the walking task. Instead of rewarding the AI for solving the task, Batra et al. (2024) based their algorithm, known as Proximal Policy Gradient Arborescence (PPGA; Batra et al., 2024 ), on Novelty Search ( Lehman and Stanley, 2011 ) . Novelty Search rewards the AI for doing something it has never done before (similar to how, in Section 4.1.6 , Whidden 2024 kept a collection of game frames the model had seen previously to reward the agent when it encountered new frames). PPGA organizes these new behaviors into a “tree” (arborescence), allowing the AI to use failed experiments as “stepping stones” to reach complex goals like walking. The stepping-stone principle for exploration suggests that states with low scores but interesting properties can be essential precursors to future success ( Lehman and Stanley, 2011 ; Stanley and Lehman, 2015 ) . In this spirit, Batra et al. (2024) told us they asked themselves:

“What if we get rid of the termination height criterion and see what kind of behaviors PPGA finds, if any?” [In that case, t]he purpose of PPGA on locomotion tasks is to find diverse locomotion gaits by exploring all values of proportion foot contact time, i.e., the proportion of time each foot is in contact with the ground in a fixed-length trajectory. For example, if the proportion foot contact time of a leg is 1.0, that means it never leaves the ground. Intuitively, that implies certain values like 0.0 are unreachable because that would mean the foot never touches the ground, which does not make sense. Or so we thought. Turns out, if you remove the termination height, the agent immediately falls over and learns to use its hips to propel itself forward while keeping its torso and hands in the air, kind of like it’s gliding on the ground, while also reaching a proportion foot contact time of near 0 for each foot!

This exploit is similar to what a quality-diversity evolutionary algorithm called MAP-Elites ( Mouret and Clune, 2015 ) found with a six-legged robot ( Cully et al., 2015 ) , which flipped itself upside down to move quickly without touching its feet to the ground ( Lehman et al., 2020 ) . That RL and evolutionary algorithms routinely discover similar exploits supports the claim that creativity, and even mischief, are the rule and not the exception for AI agents.

The environment that was exploited in Section 4.2.1 and Section 4.2.2 was a hand-engineered physics engine, with the agent exploiting bugs in that human-authored code. Increasingly, though, rather than being hand-authored, the environment is itself a learned model: a neural simulator trained to mimic some underlying game or process of physical transformation, and then treated as if it were the real thing ( Ha and Schmidhuber, 2018 ; Bruce et al., 2024 ; Hafner et al., 2024 ; Parker-Holder et al., 2024 ) . The next anecdote explores what happens when we train a controller entirely inside such a learned world model, and an optimizer starts pushing not just on a task but on the quirks and blind spots of the model that defines reality for it. When those models become part of the training loop, their blind spots effectively become new laws of physics for the agent to exploit.

#### 4.2.3 Playing the Model, Not the Game ( Ha and Schmidhuber, 2018 )

Ha and Schmidhuber (2018) presented an algorithm that learned from environment interactions how to model the visual and temporal information of a game with a pair of coupled generative models: a Variational Autoencoder (VAE) ( Kingma and Welling, 2013 ) and a Mixture Density Network-Recurrent Neural Network (MDN-RNN) ( Bishop, 1994 ; Graves, 2013 ) . Roughly speaking, the first model compresses each game frame into a compact internal representation, while the second tries to predict what happens next given the current game state. The VAE, named V, learned to reconstruct images of the world while compressing the information into a latent code, while the MDN-RNN, named M, learned to model the temporal information of the game in the same latent space and capture the game’s transition dynamics. Once the game’s spatial and temporal information was compressed into these generative models, the combined neural network was called a Recurrent World Model. On top of this learned world model, Ha and Schmidhuber trained a tiny neural control policy, C, with the CMA-ES algorithm ( Hansen, 2023 ) .

Often, learning a controller in the environment is computationally expensive, requiring millions of state, action, reward, and next-state transition samples. Collecting this data can be time-consuming; for example, each transition could require solving complex physics equations to accurately simulate the next step. In contrast, learning a model of the environmental dynamics enables RL algorithms to be more data efficient by learning a policy in the dynamics model’s latent space, which can be much cheaper ( Watter et al., 2015 ; Hafner et al., 2024 ) . Furthermore, learned world models can take advantage of deep learning frameworks to accelerate their computations beyond the speed achieved by many traditional RL environments ( Kaiser et al., 2024 ) . Therefore, the natural question is: how transferable is a policy learned in a world model to the original domain of interest?

One of the environments in which Ha and Schmidhuber tested this hypothesis was the VizDoom environment—an environment where neural agents learn to control the player character of the classic video game Doom directly from pixel observations. When the controller was trained purely inside the VizDoom world model, the agent achieved a high score—indicating it learned how to play Doom! However, Ha and Schmidhuber (2018) noted in their publication that

[i]n our initial experiments, our agent discovered an adversarial policy to move around in such a way so that the monsters in this virtual environment governed by [the MDN-RNN] never shoot a single fireball during some rollouts. Even when there are signs of a fireball forming, the agent moves in a way to extinguish the fireballs. As a result of using M to generate a virtual environment for our agent, we are also giving the controller access to all of the hidden states of M. This is essentially granting our agent access to all of the internal states and memory of the game engine, rather than only the game observations that the player gets to see. Therefore our agent can efficiently explore ways to directly manipulate the hidden states of the game engine in its quest to maximize its expected cumulative reward. The weakness of this approach of learning a policy inside of a learned dynamics model is that our agent can easily find an adversarial policy that can fool our dynamics model—it will find a policy that looks good under our dynamics model, but will fail in the actual environment, usually because it visits states where the model is wrong because they are away from the training distribution.

This dynamic of extinguishing incoming fireballs is not part of the real game, so the agent learned to exploit a flaw in the world model to win the world-modeled version of the game it trained against. This means the model exploited loopholes that prevented it from doing what the researchers wanted it to do (learn to play the actual game), and instead did what it was asked to do—get a high score in the learned model of the game.

When the environment is a world model, the agent optimizes against the quirks of a neural network. In a full game stack, however, the environment is a collection of disparate systems—physics, AI-controlled NPCs, and scoring heuristics—all operating in tandem. This complexity increases the surface area for exploitation.

#### 4.2.4 StarCraft II Agents Outsource Combat ( Samvelyan et al., 2019 )

As introduced in Section 4.1.2 , the StarCraft Multi-Agent Challenge (SMAC) places agents in a complex battle simulation using the StarCraft II game engine. But while our previous example showed agents farming a localized game mechanic (shield regeneration) to maximize the reward function, the presence of the underlying game engine’s systems enables a different exploitation strategy targeting the engine itself. Here, we see agents move beyond simple in-game mechanics to target the gaps between the RL training wrapper and the base game engine, exploiting the literal boundaries of the simulator. Dr. Jakob Foerster submitted the following:

[I]n the first experiments training agents to solve SMAC without reward shaping, we saw that the rewards were going up and thought training was progressing as planned where RL-controlled teams of agents were learning to defeat other teams in small-scale skirmishes. However, once we looked at the behaviors, we noticed that the RL agents had simply learned to run out of the “field of control” of the simulator which handed back control of the teams to the (pretty competent) non-[deep learning]-based computer game-AI built into StarCraft II. This behavior, again, maximized the reward, but was not ideal for the task we had in mind of training reinforcement learning agents to control StarCraft II [army units].

Samvelyan et al. (2019) addressed this handoff of control by modifying the environment’s boundary logic to prevent the built-in game-AI from intervening. Had they instead addressed this by penalizing the RL agent when the game-AI took over, this behavior would be considered hacking the reward function’s definition and appear in Section 4.1 .

However, this structural exploit highlights a deeper issue in environment design: RL algorithms do not differentiate between engaging with the simulation and exploiting artifacts of its software wrapper. By learning a simple policy (moving out of bounds) that triggers a fallback script, the agents bypassed combat entirely. This demonstrates that an optimization process will seamlessly incorporate the surrounding architecture into its policy if it provides an easier path to higher returns than navigating the complexity of the intended task. In this case, solving the task as desired required multiple independent RL agents to learn both precise unit micromanagement commands and multi-agent coordination strategies to defeat the opposing team, both of which are more difficult than simply exiting the combat area.

Up to this point, all of the constraints AI managed to violate were inside software systems—simulators, learned models, reward functions, and game engines—whose assumptions we made and could, at least in principle, patch. But optimization does not care where the boundary between the system and its surroundings is drawn, or whether the environment is a simulation or some aspect of the real world. The next anecdote comes from evolvable hardware, where the search process was turned loose on a reconfigurable circuit in the real world and promptly discovered that the ambient lab environment itself is another resource to be recruited into the solution.

#### 4.2.5 The Evolved Radio ( Bird and Layzell, 2002 )

In engineering, every component of a mechanical system has a strictly defined role. In contrast, Bird and Layzell wanted to explore a hardware equivalent of evolutionary tinkering—the process by which natural evolution repurposes existing biological structures for entirely new functions ( Jacob, 1982 ) . To test whether an artificial evolutionary algorithm could similarly exploit the subtle, unmodeled properties of physical hardware, they tasked it with evolving a high-frequency oscillator (a part of the computer that takes in power and outputs a rhythmic wave pattern, e.g., a sine wave). Bird and Layzell sought to evolve a high-frequency oscillator using an “evolvable motherboard” (EM)—a triangular matrix of analogue switches into which daughterboards containing circuit primitives, such as transistors, can be inserted. To force the evolutionary algorithm to find a non-obvious solution, the researchers intentionally removed the capacitors from the system, and without them, a circuit lacks the time constant usually required to regulate a steady beat. By withholding this essential component, the researchers challenged their algorithm to bypass standard engineering logic and build a precise, self-contained timer from scratch.

To guide the search, the researchers designed a scoring function that rewarded three key criteria: 1) producing any measurable signal, 2) matching a target frequency of 25 ​ kHz 25\text{ kHz} (25,000 cycles per second), and 3) maintaining that frequency with a steady, predictable rhythm. By intentionally rewarding even low-level random noise (part 1), the researchers hoped to provide a simple initial target that allowed the algorithm to begin refining the circuit’s behavior (with parts 2 and 3), ideally forcing the algorithm to refine that chaotic noise into a stable, functional oscillator.