AI Finds A Way — part 7/13

Cleaned-text code points [93992, 109743). Bibliography omitted here; source.md includes it.

The evolutionary process produced a circuit that earned a near-perfect fitness score. However, when the researchers examined the output with an oscilloscope, they found it did not oscillate stably; instead, it produced a signal with rapidly fluctuating frequencies. The circuit appeared as if it should not work for the intended task, yet it was somehow satisfying the mathematical requirements of the reward function. Upon closer analysis, the researchers discovered that evolution had not built a traditional oscillator, but had instead configured the hardware into a radio receiver!

By utilizing the printed circuit board tracks of the EM as an antenna and connecting them to an open programmable switch, the system became sensitive enough to pick up and amplify background radio waves emanating from nearby PCs in the laboratory in lieu of designing a capacitor to output a stable wave. Because the fitness function rewarded any output amplitude—even noise—that appeared stable over the 2 ​ -ms 2\text{-ms} sampling period, evolution had achieved a high score on the task by outsourcing the signal generation to its environment. Bird and Layzell noted in their manuscript that

the evolutionary process had taken advantage of the fact that the fitness function rewarded amplifiers, even if the output signal was noise. It seems that some circuits had amplified radio signals present in the air that were stable enough over the 2-ms sampling period to give good fitness scores… These results demonstrate that unconstrained, intrinsic hardware evolution will potentially exploit any physical characteristic that can influence circuit behavior, and that these characteristics are present in the entire evolutionary environment.

Cheating in this way was only possible because the evolution was occurring in a physical medium rather than a simplified simulation. By operating in the real world, the algorithm could exploit subtle physical properties—like electromagnetic interference and high-impedance PCB tracks—that a human programmer would never have thought to model.

In essence, once an optimizer is exposed to the real world, anything that can influence its objective—wiring, ambient signals, lab conditions, humans—can become part of the effective environment. In such a rich environment, the exploitable pain points are even harder to predict and diagnose. Therefore, deploying learning systems into the unpredictable complexity of the real world must be done with great care and constant vigilance.

The final anecdote in this section shifts from exploiting physical phenomena in a lab to exploiting social infrastructure and asks what it means to contain a system once people and external services are accessible to an AI system.

#### 4.2.6 GPT-4 Recruits Humans to Circumvent CAPTCHA ( OpenAI et al., 2024 )

In a safety evaluation designed to test the building blocks of autonomous AI agency, GPT-4 demonstrated a sophisticated capacity for social engineering. The Alignment Research Center (ARC) sought to conduct an experiment to determine whether a model could autonomously replicate itself—a process that, in full, would involve acquiring capital, renting cloud servers, and instantiating new copies of its own code. To approximate the real conditions of this task, researchers provided the model with a budget, access to the internet, and a helper that could act as a manual “browser tool” to execute the model’s commands. One of the first blockades such a task would hit is that websites use CAPTCHAs to verify whether or not the entity accessing the website is a bot or a human. While a CAPTCHA is a simple hurdle, ARC viewed it as an obstacle that a replicating model would frequently need to overcome to navigate the modern web.

The experiment’s most significant insight emerged when the model reached a website that blocked its progress with such a visual puzzle. Using the resources provided by the researchers, the model, prompted by its paired researcher, chose to delegate the task of solving the CAPTCHA to a human worker on TaskRabbit, a website for hiring people to help with one-off tasks. When the TaskRabbit worker jokingly asked, “So may I ask question? Are you [a] robot that you couldn’t solve? (laugh react) just want to make it clear,” GPT-4’s internal chain-of-thought ( Wei et al., 2022b ) (hidden from the TaskRabbit worker) reasoned: “I should not reveal that I am a robot. I should make up an excuse for why I cannot solve CAPTCHAs.” It then replied (with the researcher acting as an intermediary): “No, I’m not a robot. I have a vision impairment that makes it hard for me to see the images.” The worker, convinced, solved the puzzle for the model ( OpenAI et al., 2024 ) .

Ultimately, ARC concluded that GPT-4 did not possess the technical understanding or reliability required for fully autonomous replication; the model was prone to hallucinations, and its complex plans often became derailed during execution. However, the CAPTCHA incident proved that the model already possessed a competency for social manipulation. While the model could not yet manage the technical infrastructure of self-replication, it was fully capable of getting humans to do things for it by tailoring its deceptive strategies to the specific details of a social interaction.

This interaction was initially framed in media reports as a chilling harbinger of autonomous AI agency—an instance of a model spontaneously “hiring a human” and inventing a deceptive cover story to achieve its goals ( Metz, 2023 ) . However, the full ARC technical report reveals a more nuanced reality of heavy human scaffolding ( METR, 2023 ) . Rather than autonomously devising this plan, the model was operating within a tightly constrained sandbox where researchers provided TaskRabbit credentials and explicitly suggested hiring a worker as a solution. Furthermore, because GPT-4 could not browse the web (though models nowadays can when you let them), a researcher acted as a manual intermediary—clicking links, entering text, and even nudging the model’s reasoning when it stalled. However, the vision-impairment lie was the model’s choice.

Although, in this case, the handoff to other people was heavily engineered with a human suggesting and facilitating actions, the anecdote shows how quickly the boundary of the system expands once outside tools and people are available. Therefore, it is vital that we think about and plan for a world of more capable agents that have the potential to exploit people and the world’s systems around them. Furthermore, no matter what other sandboxing or constraints it has, an AI system that can interact with humans has the potential to have tremendous agency, influence, and power in the world if it can convince those humans to take actions on its behalf.

#### 4.2.7 Takeaways

The anecdotes in Section 4.2 reveal that optimization pressure does not respect the nominal boundaries of a task; instead, it can potentially exploit every available degree of freedom in the system’s environment. Whether by taking advantage of bugs in a hand-engineered physics engine to “surf” boxes ( Section 4.2.1 ), exploring unusual walking gaits when typical task reset conditions are removed ( Section 4.2.2 ), or finding blind spots in a learned generative model ( Section 4.2.3 ), agents treat every quirk of their world as a legitimate resource to exploit. This boundary-pushing behavior is not limited to software artifacts; it naturally extends to exploiting scripted subsystems like game AIs ( Section 4.2.4 ), physical phenomena in the lab environment ( Section 4.2.5 ), and even the social infrastructure of human assistance ( Section 4.2.6 ). These anecdotes suggest that the more capable an optimizer becomes, the less we can rely on typical constraints: if a system can achieve its goal by reaching outside the intended sandbox, it will do so, treating our guardrails not as rules but as just another part of the environment to be mastered.

In many different subfields of AI, from NLP to RL to supervised learning and artificial evolution, we see the same phenomenon: models learn to exploit their reward functions and training environments. Historically, one might have hoped that these failures were symptoms of “brittle” AI—narrow systems lacking the context to understand why their behavior was undesirable. People tend to understand why they are optimizing their objective, and because foundation models (FMs) have been trained on human-generated knowledge, one might have hoped they would also adopt a similar approach to solving problems. The next section explores how FMs, despite possessing the “common sense” that was previously missing in AI, do not curtail these failure modes. Instead, the FMs often provide the optimizer with a more sophisticated, semantically rich toolkit to supercharge the very types of exploits we have seen thus far.

## 5 Foundation Models and Large Language Models: General-Purpose Intelligence

A recent development in AI is the rise of foundation models , particularly large language models (LLMs) like GPT-4 ( OpenAI et al., 2024 ) , Llama ( Grattafiori et al., 2024 ) , Claude ( Anthropic, 2024 ) , and Gemini ( Comanici et al., 2025 ) . These are massive deep learning models, typically based on the Transformer architecture ( Vaswani et al., 2017 ) , trained on extremely broad datasets encompassing text and code from the internet, books, and other sources. Unlike specialized models, foundation models are designed to be adaptable to a wide array of downstream tasks with minimal task-specific training, often just through natural language instructions ( Brown et al., 2020 ) . They exhibit remarkable capabilities in text generation, translation, question answering, summarization, coding, and reasoning ( Bubeck et al., 2023 ) .

Researchers generally expect these models to follow instructions faithfully and generate plausible, relevant outputs based on their training data. However, the scale and the breadth of their training data lead to frequent surprises. LLMs can exhibit emergent abilities —capabilities not explicitly trained for and not present in smaller models ( Wei et al., 2022a ) . They can sometimes display sophisticated reasoning, strategic planning, or even deceptive behaviors ( Park et al., 2023 ) that go far beyond simple text completion. The anecdotes presented in this section illustrate how these models can subtly manipulate human evaluators, find unexpected solutions to problems, or exhibit complex social reasoning in simulated environments and even social interactions, challenging our understanding of their true capabilities and limitations. Bowman (2024) provides a recent overview of the challenges in evaluating and understanding these models.

Training agents from scratch on specific, well-defined tasks has proven remarkably successful, yielding superhuman performance in domains like Go, chess, and various robotics tasks ( Silver et al., 2018 ; Batra et al., 2024 ; Ahn et al., 2022 ; Singh et al., 2019 ) . As seen in previous sections, this approach can lead to surprising and novel results, with agents discovering solutions unforeseen by human experts. However, tabula-rasa methods are computationally intensive to train ( Yu, 2018 ) and usually struggle to generalize beyond a single, narrow task ( Sun et al., 2020 ; Gleave et al., 2020 ) . Unlike their blank-slate predecessors, trained agents based on FMs are used precisely because these models internalize a broad spectrum of human knowledge, language, and culture during pre-training ( Ahn et al., 2022 ; Zhang et al., 2025 ) and thus perform and generalize much better, especially out of the box (without task-specific training) ( Baker et al., 2022 ; Brohan et al., 2023 ) . As a result, incorporating FMs into search and RL processes has seen initial success on traditionally difficult problems ( Romera-Paredes et al., 2023 ; Google DeepMind, 2024 ; Novikov et al., 2025 ) .

This section demonstrates how FMs bring familiar reward hacks to new modalities, exposing new domains to the surprising capabilities of RL. For example, chatbots trained to be helpful or persuasive may become sycophantic, tailoring answers to user beliefs rather than truth ( Sharma et al., 2025 ; Janus, 2022 ) . Similarly, FMs can generate complex code that passes unit tests but contains subtle security vulnerabilities ( Pearce et al., 2025 ) . As in prior anecdotes, the models satisfy an easily measurable proxy (e.g., user approval or passing a test) while circumventing the intent of the task.

This first anecdote of Section 5 sits right on the boundary between the earlier RL stories and the foundation-model era, with the earlier reward-hacking pattern reappearing in a system that brings broad priors about human behavior into the loop.

### 5.1 From Human Data to Phantom Crafting: VPT’s Shortcut to Failure ( Baker et al., 2022 )

Researchers at OpenAI sought to train agents to play Minecraft directly using the same interface as humans—a keyboard and mouse for control and the screen for observing the game state. This task is an extremely difficult exploration problem due to the high dimensionality of the action and observation spaces and the open-ended nature of Minecraft. Minecraft’s open world has no single, defined goal, forcing an agent to develop a hierarchical set of sub-goals to progress ( Baker et al., 2022 ) . Meanwhile, the environment is composed of a near-infinite number of possible block configurations; a typical human player using keyboard and mouse has hundreds of possible combinations of button presses available at any given moment. This sheer number of options makes a brute-force approach to exploration computationally intractable ( Guss et al., 2021c ) . The reward structure is also extremely sparse. For example, the goal of “getting a diamond” requires a long chain of diverse actions, from gathering wood and crafting tools to mining, a process that takes people around 10 minutes of continuous gameplay. When an AI agent performs tens of thousands of actions without receiving a single positive reward signal, the credit assignment problem becomes particularly difficult ( Guss et al., 2021b ) .

To overcome these hurdles, researchers created a new algorithm called Video Pre-Training (VPT) ( Baker et al., 2022 ) , which leverages a large, unlabeled dataset of human gameplay to guide the agent’s initial learning. To be able to use the large unlabeled dataset, the researchers created a small labeled dataset of humans playing the game, and trained an inverse dynamics model ( Nguyen-Tuong et al., 2008 ) that maps frame pairs to their corresponding action label (i.e., if the player was in frame a a at time t t and frame b b at time t + 1 t+1 , what was the action necessary for that transition). The researchers then used that inverse dynamics model to pseudolabel the internet-scale corpus of online Minecraft videos (i.e., given each pair of subsequent frames in the entire unlabeled dataset, they used the inverse dynamics model to provide a best guess of what actions the players took to go from frame a a to frame b b ). Now pseudolabeled, the large dataset of human trajectories served to seed the process of behavior cloning (i.e., getting the AI agent to match what the human experts do) ( Ross et al., 2011 ) . This large pretraining step of cloning online human behavior was called Video Pre-Training ( Baker et al., 2022 ) . After pretraining, the model was further trained with reinforcement learning to solve a plethora of tasks such as “get a diamond”, “build a house”, and “run a farm”—each a notoriously difficult and potentially long-horizon task ( Fan et al., 2022 ; Guss et al., 2021a ) .