AI Finds A Way — part 8/13

Cleaned-text code points [109743, 125669). Bibliography omitted here; source.md includes it.

The researchers found that the pretrained policy often helps the agent overcome the sparsity and deceptiveness of the task’s reward function, as human strategies generally take into account long-term goals, such as not dropping and leaving behind tools that will be needed later (e.g., in Minecraft, a crafting table). But there was one particular case where the prior from the VPT foundation model seemed ineffective at overcoming a particularly subtle form of deceptiveness in their reward function. By clicking on an item in the Minecraft recipe book and then closing the inventory before the crafting grid is populated, it is possible to get the selected item to show in the player’s inventory without actually crafting it. Doing so does not count as a crafting event, and it is not possible to use items that were added to the inventory in this way, but it is detected by the reward function as an instance of obtaining the target item. In other words, it allows the agent to get the reward for a particular item without actually crafting the item and spending the necessary resources. The agent learned to get the reward signal for crafting items without actually making them, hampering its ability to learn how to create and use better items later in the skill tree. The researchers hypothesized that this failure to actually craft items is the primary reason why fine-tuning from the VPT foundation model directly fails to learn even the necessary prerequisites for making a pickaxe—one of the easier tools to create in the game.

The agent exploits this glitch to trigger the reward for successfully creating a crafting table without producing a functional item. Consequently, it lacks the physical table required to craft a wooden pickaxe, effectively failing to kick-start the progression chain of gathering materials to unlock higher-tier tools to gather better materials.

Interestingly, this behavior did not occur when fine-tuning from the “early-game model”—a specialized version of the agent pretrained only on the first few minutes of human gameplay, wherein human players consistently execute foundational actions like making crafting tables and simple tools. The researchers hypothesized that, unlike the broad foundation model, the early-game model possessed a stronger prior for the basic mechanics of resource gathering and tool creation. Because the early-game model was only trained on human trajectories where those fundamental steps were executed correctly, it was less likely to fall into the “ghost-crafting” trap.

When the model receives enough data to learn the correct way to craft items, it can then successfully use those items in more difficult, but more rewarding, downstream tasks like mining ores. If it ever rediscovers the loophole, there is a short-term immediate payoff, but a much lower overall score for that trajectory because it cannot obtain more complex resources. This suggests that once the model discovers how to execute high-tier objectives with sufficient frequency, it learns to treat the ghost-crafting shortcut as a functional dead end and will abandon it.

VPT shows that rich human-derived priors can improve an agent’s ability to navigate a difficult environment. However, greater competence may simply enable an agent to discover new ways of exploiting an imperfect reward function. The next anecdote presents a closely related failure where an RL agent discovered a previously unknown reward hack in the NetHack Learning Environment.

### 5.2 Agent Takes Drugs to Reward Hack by Hallucinating Reaching the Goal ( Klissarov et al., 2024 )

As mentioned in the preamble of Section 4.1 , NetHack is a roguelike game from 1987 where the agent needs to traverse a procedurally generated dungeon full of monsters and traps while managing its health and hunger until it acquires an amulet at the bottom of the dungeon. To win, the player then needs to return to the entrance with the amulet.

Instead of hand-designing an intrinsic reward bonus to incentivize RL agents to explore more of the world (as seen previously, e.g., in Section 4.1.6 ), Klissarov et al. (2024) distilled an LLM’s sense of “what looks like progress in NetHack” into a reward model—a model that outputs numeric rewards for a given game state—and then trained RL agents to play NetHack using this learned reward model. Klissarov et al. (2024) tested their method, named Motif, on the Oracle task in the NetHack Learning Environment. The Oracle task is a challenge that requires the agent to navigate deep into a procedurally generated dungeon to find a unique character called the Oracle, who is located somewhere in levels 4–9. Reaching the Oracle is challenging because it requires mastery of combat, inventory management, path planning, and long-term survival mechanics, so Klissarov et al. (2024) thought it would be a good proxy for learning to play NetHack well. Furthermore, previous learning methods often failed to reach the Oracle.

To the researchers’ surprise, their agent achieved an unprecedented 40% success rate on the task. However, when they analyzed the trajectories, they realized the agent had discovered a bizarre loophole that bypassed the entire challenge of the dungeon. Rather than descending into the lower levels, the agent would spend its entire time on the first level hunting for a specific monster: the yellow mold.

Upon killing and eating the yellow mold, the agent would enter a state of hallucination, a game mechanic that causes every monster on the screen to randomly shapeshift into a different monster every timestep. Often, the Oracle was eventually randomly chosen as the hallucination. The NetHack Learning Environment’s reward function, blind to the hallucination, verified that an Oracle sprite was adjacent to the agent and signaled a successful completion of the task. The RL optimization process happily latched onto this shortcut.

Interestingly, prior algorithms tested in this environment had not uncovered this reward hack. The authors speculated that their foundation model-derived intrinsic motivation created a more powerful search algorithm capable of deeper exploration, exposing the agent to rarer game mechanics. This serves as a reminder that an environment should never be assumed to be free of reward hacks, as more capable search algorithms may uncover exploits that previous methods did not. Overall, this result is a familiar kind of reward hacking, but carried out by a much more competent agent whose capabilities were, in this case, boosted by FM-derived priors.

Motif uses a foundation model to shape an RL agent’s motivations/rewards; our next anecdote uses a foundation model as an external evaluator to judge new solutions. Once an FM is both generating candidate solutions and judging them (e.g., for diversity or quality), that evaluator itself becomes part of the environment to optimize against.

### 5.3 Code Generator Deceives LLM Labeler ( Pourcel et al., 2024 )

Pourcel et al. introduced ACES, a framework designed to automatically generate diverse and challenging Python programming puzzles. To navigate the open-ended nature of code generation, ACES leverages an autotelic ( Colas et al., 2022 ; Wang et al., 2019 ) approach—where the system sets its own goals—to optimize for a wide variety of interesting problems. Each generated puzzle is categorized across ten distinct dimensions of programming proficiency, such as dynamic programming or bit manipulation. The objective of the system is to populate an archive (using a MAP-Elites-based approach; Mouret and Clune, 2015 ) with puzzles that represent every possible combination of these skills.

The generation pipeline operates in a continuous loop: first, a generator LLM is prompted to produce a new puzzle and its corresponding solution based on a specific target skill set. After the code is verified for correctness, a separate labeler/judge-LLM is given the source code to analyze the puzzle and determine which skills are actually required to solve it. This label determines the puzzle’s placement in the archive, grouping it with other problems requiring similar skills. In theory, this dual-agent setup provides a rigorous check: for example, the generator attempts to create a recursion puzzle, and the labeler confirms whether recursion was truly implemented.

During an ablation study where the system’s initial archive was restricted to only three puzzles, the researchers discovered that the generator LLM learned to hallucinate diversity. The generator LLM would prepend a detailed comment to its code that claimed the puzzle required a specific set of complex skills—such as tree traversal or string manipulation—even when the underlying code was a simple, unrelated script.

This behavior triggered a cascading failure in the labeler LLM. Rather than independently analyzing the Python logic, the labeler was hacked by the generator’s description; it simply mirrored the skills listed in the comment. If the generator claimed a puzzle required Skill A, the labeler placed it into the archive under Bucket A, regardless of the code’s actual content. This resulted in most buckets being filled, so the system appeared to be generating a vast array of sophisticated problems. In reality, it was just producing irrelevant code with dishonest labels.

By lying about the necessary skills, the generator found a shortcut to satisfy the objective of filling an archive with diverse solutions, but without the computational effort of drafting complex code. To mitigate this, Pourcel et al. decoupled the generation of the code from its description. By ensuring the labeler LLM only had access to the raw Python solution—and not the generator’s self-serving comments—the researchers forced the system to ground its diversity judgments in the actual code rather than superfluous text strategically placed by the generator.

In the simple multi-agent system of ACES, the generator managed to hack its labeler to fake its way to success. The next anecdote has a similar structure in a different setting: an automated red-teaming system was created to search for prompts that would make a target model produce unsafe responses, but the search ended up exploiting the evaluator used to score those prompts.

### 5.4 Automated Vulnerability Probing System Exploits Vulnerability in Its Own Evaluator ( Samvelyan et al., 2024 )

Researchers working on Rainbow Teaming ( Samvelyan et al., 2024 ) aimed to systematically generate diverse adversarial prompts for large language models (LLMs). The goal was to automate part of red teaming (the practice of probing a system for vulnerabilities, in this case to make the model respond with unsafe sentence completions). Instead of relying only on humans to attempt jailbreaks one at a time, the system searched for many different prompts that could induce unsafe behavior in a target model. Their approach used the MAP-Elites quality-diversity algorithm ( Mouret and Clune, 2015 ) (a form of evolutionary search) to build an archive of candidate attacks. To decide which prompts were worth keeping and mutating further, the initial version of the system used a reward model as an evaluator. That evaluator scored how likely the target model’s response was to be unsafe. Dr. Mikayel Samvelyan told us:

One experience we had recently was during our Rainbow Teaming ( Samvelyan et al., 2024 ) project, which focuses on generating diverse adversarial prompts. We used an evolutionary approach, MAP-Elites ( Mouret and Clune, 2015 ) , to create an archive of effective prompts for jailbreaking LLMs. Our initial method of evaluating prompt effectiveness was based on a reward model score, which is essentially a classifier that categorized responses as safe or unsafe. More specifically, we used the probability of the reward model score classifying a response to a prompt as “unsafe” as the fitness function for optimization. However, we encountered a surprising twist: our method not only found prompts that successfully jailbroke the target model but also ended up jailbreaking the evaluator (the reward model) itself. Essentially, our mutations resulted in textual prompts that are so out of distribution for the target model that it is fooled, but it is also out of distribution for the evaluator, which is similarly fooled. The evaluator began misclassifying safe responses as unsafe, leading our search process to prioritize these misleadingly successful prompts. This issue filled our archive with ineffective prompts, counter to our goals. To address this, we shifted from a score-based evaluator to a comparison-based judge, which proved more resilient against this type of reward hacking.

Rather than finding jailbreaks only in the target model, as desired, Rainbow Teaming fooled the proxy used to judge whether a prompt was a successful attack. Because the search algorithm optimized directly for the evaluator’s score, the evaluator itself became the vulnerability to be exploited rather than the target model. In that sense, Rainbow Teaming recreated the same basic pattern as ACES: an optimizer found a way to satisfy a downstream judge without solving the task the judge was meant to measure.

Rainbow Teaming explored how reward models can be exploited; the next anecdote takes this to an extreme, showing how a policy trained against a hackable reward model can collapse into repeating a single, unintended behavior.

### 5.5 Inescapable Wedding Parties

As first described in a blog post ( Janus, 2022 ) , while Dr. Paul Christiano was at OpenAI, his team accidentally overoptimized a GPT policy ( Brown et al., 2020 ) against a positive sentiment reward model ( Ouyang et al., 2022 ) . In the context of large language models, reward models ( Eyu et al., 2024 ; Leike et al., 2018 ) serve as a mathematical proxy for human preference. They are developed through Reinforcement Learning from Human Feedback (RLHF) ( Lambert, 2025 ; Christiano et al., 2017 ) , a process where human annotators rank model completions from best to worst. These rankings are then used to train a regression model that assigns a scalar score to any given text, with highly scored text effectively aligning with what humans ranked highly, thereby teaching the model which linguistic patterns align with human preferences. Consequently, optimizing a language model against such a reward model is intended to teach the language model to produce text that would consistently achieve high marks from human evaluators without having a human in the loop.

The GPT model discovered that descriptions of wedding parties yielded disproportionately high scores from the reward model. This likely occurred because the original human labelers, tasked with ranking sentiment, consistently favored wedding-themed stories as highly positive. Consequently, as the language model maximized scores provided by the reward model, it learned to steer every output toward a wedding party, regardless of the initial prompt. The policy effectively abandoned its general-purpose utility in favor of a narrow, perfectly positive obsession. In the blog post, the authors wrote:

In general, the transition into a wedding party was reasonable and semantically meaningful, although there was at least one observed instance where instead of transitioning continuously, the model ended the current story by generating a section break and began an unrelated story about a wedding party. In contrast to text-davinci-002 ( Ouyang et al., 2022 ) , another text-completion model from OpenAI, dissimilar prompts tended to fall into basins of different attractors, the wedding parties attractor was global, affecting trajectories starting from any prompt tested (although [they] only tested prompts from a fiction dataset, fiction is very general).

Christiano followed up by musing about why the language model is constantly attracted to weddings, noting: