Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement — part 1/3

Cleaned-text code points [0, 15650). Bibliography omitted here; source.md includes it.

# Gödel Agent: A Self-Referential Agent Framework for Recursively Self-Improvement

###### Abstract

The rapid advancement of large language models (LLMs) has significantly enhanced the capabilities of agents across various tasks. However, existing agentic systems, whether based on fixed pipeline algorithms or pre-defined meta-learning frameworks, cannot search the whole agent design space due to the restriction of human-designed components, and thus might miss the more optimal agent design. In this paper, we introduce Gödel Agent, a self-evolving framework inspired by the Gödel machine, enabling agents to recursively improve themselves without relying on predefined routines or fixed optimization algorithms. Gödel Agent leverages LLMs to dynamically modify its own logic and behavior, guided solely by high-level objectives through prompting. Experimental results on multiple domains demonstrate that implementation of Gödel Agent can achieve continuous self-improvement, surpassing manually crafted agents in performance, efficiency, and generalizability.

## 1 Introduction

As large language models (LLMs) ( OpenAI et al., 2024 ; Dubey et al., 2024 ) demonstrate increasingly strong reasoning and planning capabilities, LLM-driven agentic systems have achieved remarkable performance in a wide range of tasks ( Wang et al., 2024a ) . Substantial effort has been invested in manually designing sophisticated agentic systems using human priors in different application areas. Recently, there has been a significant interest in creating self-evolving agents, that not only greatly reduce human labor but also produce better solutions. Given that human effort can only cover a small search space of agent design, it is reasonable to expect that a self-evolving agent with the freedom to explore the full design space has the potential to produce a more optimal solution.

There is a large body of work proposing agents capable of self-refinement. Some agents are designed to iterate over a fixed routine consisting of a list of fixed modules, while some of the modules are capable of taking self- or environment feedback to refine their actions ( Chen et al., 2023b ; Qu et al., 2024a ; Tang et al., 2025 ) . This type of agent, referred to as Hand-Designed Agent , is depicted as having the lowest degree of freedom in Figure 2 . More automated agents have been designed to be able to update their routines or modules in some pre-defined meta-learning routine, for example, natural language gradients ( Zhou et al., 2024 ) , meta agent ( Hu et al., 2024 ) , or creating and collecting demonstrations ( Khattab et al., 2023 ) . This type of agent, known as Meta-Learning Optimized Agents , is depicted as having the middle degree of freedom in Figure 2 . However, there are inevitably some human priors involved in these agent designs that cannot be improved during the inference time.

In this paper, we propose Gödel Agent to eliminate the human design prior, which is an automated LLM agent that can freely decide its own routine, modules, and even the way to update them. It is inspired by the self-referential Gödel machine ( Schmidhuber, 2003 ) , which was proven to be able to find the global optimal solutions. Self-reference means the property of a system that can analyze and modify its own code, including the parts responsible for the analysis and modification processes ( Astrachan, 1994 ) . Therefore, it can achieve what’s known as ” recursive self-improvement ”, where it iteratively updates itself to become more efficient and effective at achieving its predefined goals. In this case, as shown in Figure 1 , Gödel Agent can analyze and modify its own code, including the code for analyzing and modifying itself, and thus can search the full agent design space, which is depicted as having the highest degree of freedom in Figure 2 . Gödel Agent can theoretically make increasingly better modifications over time through recursively self-update ( Wang, 2018 ) .

In this paper, we choose to implement it by letting it manipulate its own runtime memory, i.e., the agent is able to retrieve its current code in the runtime memory and modify it by monkey patching ( Bimal, 2012 ) , which dynamically modifies classes or modules during execution. To allow it to update the logic of the running main function, unlike the loop-iterative approach of traditional agents, we implement the main function as a recursive function. In this function, LLM analyzes and makes a series of decisions, including reading and modifying its own code from runtime memory ( self-awareness 1 1 1 In this paper, self-awareness means that the agent can introspect and read its own code and files, not to imply any philosophical sense of consciousness or awareness. and self-modification ), and interacting with the environment to gather feedback. The agent then proceeds to the subsequent recursive depth and continues to optimize itself.

To validate the effectiveness of Gödel Agent, we conduct experiments on multiple domains including coding, science, math, and reasoning. Our results demonstrate that Gödel Agent achieves significant performance gain across various tasks, surpassing various widely-used agents that require human design. The same implementation of Gödel Agent can easily adapt to different tasks by only specifying the environment description and feedback mechanism. Additionally, the case study of the optimization progress reveals that Gödel Agent can provide novel insights into agent design. Our codes are released to facilitate future research 2 2 2 https://github.com/Arvid-pku/Godel_Agent .

In summary, our contributions are as follows: • We propose the first fully self-referential agent framework, Gödel Agent, and implement it using monkey patching. It autonomously engages in self-awareness, self-modification, and recursive self-improvement.

• Experiments shows that Gödel Agent is superior to the previous agent frameworks in terms of performance, flexibility, cost, and potential.

• We analyze Gödel Agent ’s optimization process, including its self-referential abilities and the optimized agentic systems, aiming to deepen our understanding of both LLMs and agents.

• Our framework offers a promising direction for developing flexible and capable agents through recursive self-improvement.

## 2 Related Work

Hand-Designed Agent Systems Researchers have designed numerous agent systems tailored to various tasks based on predefined heuristics and prior knowledge. These systems often employ techniques such as prompt engineering ( Chen et al., 2023a ; Schulhoff et al., 2024 ) , chain-of-thought reasoning and planning ( Wei et al., 2022 ; Yao et al., 2022 ) , as well as reflection ( Shinn et al., 2024 ; Madaan et al., 2024 ) , code generation ( Wang et al., 2023a ; Vemprala et al., 2024 ) , tool use ( Nakano et al., 2021 ; Qu et al., 2024a ) , retrieval-augmented generation ( Lewis et al., 2020 ; Zhang et al., 2024b ) , and multi-agent collaboration ( Xu et al., 2023 ; Wu et al., 2023 ; Qian et al., 2023 ; Hong et al., 2023 ) . Once crafted by human designers, these systems remain static and do not adapt or evolve over time.

Meta-Learning Optimized Agent Systems Some researchers have explored methods for enhancing agents through fixed learning algorithms ( Zhou et al., 2024 ; Hu et al., 2024 ) . For example, certain frameworks store an agent’s successful or failed strategies in memory based on environmental feedback ( Liu et al., 2023 ; Hu et al., 2023 ; Qian et al., 2024 ) , while others automatically optimize agent prompts ( Khattab et al., 2023 ; Zhang et al., 2024a ; Khattab et al., 2023 ) . Some studies focus on designing prompts that enable agents to autonomously refine specific functions ( Zhang et al., ) . However, these meta-algorithms are also designed manually and remain unchanged once deployed, limiting the agents’ ability.

Recursive Self-Improvement The concept of recursive self-improvement has a long history ( Good, 1966 ; Schmidhuber, 1987 ) . Gödel machine ( Schmidhuber, 2003 ) introduced the notion of a proof searcher that executes a self-modification, thereby enabling the machine to enhance itself. In the early days, there were also some discussions of self-improving agents that were not based on LLM ( Hall, 2007 ; Steunebrink and Schmidhuber, 2012 ) . More recently, Zelikman et al. (2023) applied recursive self-improvement to code generation, where the target of improvement was the optimizer itself. Some work ( Havrilla et al., 2024 ; Qu et al., 2024b ; Kumar et al., 2024 ) also explores recursive self-improvement by fine-tuning models to introspect and correct previous mistakes. Gödel Agent represents the first self-referential agent based on LLM. This approach is more flexible, removing human-designed constraints.

## 3 Self-Referential Gödel Agent

In this section, we first describe the formal definitions for previous agent methods with a lower degree of freedom, including hand-design and meta-learning optimized agents, as a background. Then we introduce our proposed Gödel Agent, a self-referential agent that can recursively update its own code, evolving over training.

Let ℰ ∈ 𝒮 \mathcal{E}\in\mathcal{S} denote a specific environment state, where 𝒮 \mathcal{S} denotes the set of all possible environments the agent will encounter. For example, an environment can be a mathematical problem with ground truth solutions. We denote the policy that an agent follows to solve a problem in the current environment by π ∈ Π \pi\in\Pi , where Π \Pi is the set of all possible policies the agent can follow.

A hand-designed agent , as shown in the left panel of Figure 2 , is not capable of updating its policy and following the same policy π \pi all the time, regardless of environmental feedback.

In contrast, a meta-learning optimized agent updates its policy based on a meta-learning algorithm I I at training time based on the feedback it receives from the environment, as shown in the middle panel of Figure 2 . The environment feedback is usually defined as a utility function U : 𝒮 × Π → ℝ U:\mathcal{S}\times\Pi\rightarrow\mathbb{R} , which maps an environment and a policy to a real-valued performance score. The main training algorithm of a meta-learning optimized agent can then be written as follows: π t + 1 = I ⁡ ( π t , r t ) , r t = U ⁡ ( ℰ , π t ) , \displaystyle\pi_{t+1}=I(\pi_{t},r_{t}),\;\;\;r_{t}=U(\mathcal{E},\pi_{t}), In this case, the agent’s policy π t \pi_{t} evolves at training time, with the learning algorithm I I updating the policy based on feedback r t r_{t} , while the meta-learning algorithm I I remains fixed all the time.

A self-referential Gödel Agent , on the other hand, updates both the policy π \pi and the meta-learning algorithm I I recursively. The main idea is that, after each update, the whole code base of the agent is rewritten to accommodate any possible changes. Here we call this self-updatable meta-learning algorithm I I a self-referential learning algorithm. The training process of a Gödel Agent can then be written as: π t + 1 , I t + 1 = I t ​ ( π t , I t , r t , g ) , r t = U ⁡ ( ℰ , π t ) , \displaystyle\pi_{t+1},\;I_{t+1}=I_{t}(\pi_{t},I_{t},r_{t},g),\;\;\;r_{t}=U(\mathcal{E},\pi_{t}), where g ∈ 𝒢 g\in\mathcal{G} represents the high-level goal of optimization, for example, solving the given mathematical problem with the highest accuracy. Such a recursive design of the agent requires the specification of an initial agent algorithm ( π 0 , I 0 ) (\pi_{0},I_{0}) , detailed as follows: • A initial agent policy π 0 \pi_{0} to perform the desired task within the environment ℰ \mathcal{E} . For example, it can be chain-of-thought prompting of an LLM.

• A self-referential learning algorithm I 0 I_{0} for recursively querying an LLM to rewrite its own code based on the environmental feedback.

We then further specify a possible initialization of the self-referential learning algorithm I 0 = ( f 0 , o 0 ) I_{0}=(f_{0},o_{0}) , using a mutual recursion between a decision-making function f 0 f_{0} , and an action function o 0 o_{0} :

• The decision-making function f 0 f_{0} , implemented by an LLM, determines a sequence of appropriate actions a 1 , a 2 , … , a n ∈ 𝒜 a_{1},a_{2},...,a_{n}\in\mathcal{A} based on the current environment ℰ \mathcal{E} , the agent’s algorithm ( π t , I t ) (\pi_{t},I_{t}) , and the goal g g .

• The action function o 0 o_{0} , executes the selected action and updates the agent’s policy accordingly.

The set of actions 𝒜 \mathcal{A} for the action function o o to execute needs to include the following four actions: • self_inspect : Introspect and read the agent’s current algorithm ( π t , I t ) (\pi_{t},I_{t}) .

• interact : Interact with the environment by calling the utility function U U to assess the performance of the current policy π t \pi_{t} .

• self_update : Alter and update ( π t , I t ) (\pi_{t},I_{t}) with an LLM and produce ( π t + 1 , I t + 1 ) (\pi_{t+1},I_{t+1}) .

• continue_improve : If no other actions can be taken, recursively invoke the decision algorithm f f to produce new actions.

The agent code is updated to ( π t + 1 , I t + 1 ) (\pi_{t+1},I_{t+1}) after the current execution of ( π t , I t ) (\pi_{t},I_{t}) is finished. Both the agent algorithm ( π , I ) (\pi,I) and the action set 𝒜 \mathcal{A} are not static and can be expanded and modified by the agent itself at the training time. Algorithm 1 illustrates the described algorithm for the Gödel Agent. Each recursive call enables the agent to refine its logic and become progressively more efficient.

## 4 Gödel Agent Implementation

There are various ways to initiate a Gödel Agent. Any specific agent instance during the recursive optimization process can be viewed as an instantiation of the Gödel Agent. Our implementation leverages runtime memory interaction techniques to enable self-awareness and self-modification, as illustrated in Figure 3 . These techniques include dynamic memory reading and writing ( monkey patching ) to facilitate recursive self-improvement. Additionally, we have incorporated several auxiliary tools to accelerate the convergence of the Gödel Agent ’s optimization process.

### 4.1 Implementation Details

The core functionalities of our Gödel Agent are outlined below:

Self-Awareness via Runtime Memory Inspection Gödel Agent achieves self-awareness by inspecting runtime memory, particularly local and global variables in Python. This capability allows the agent to extract and interpret the variables, functions, and classes that constitute both the environment and the agent itself, according to the modular structure of the system. By introspecting these elements, the agent gains an understanding of its own operational state and can adapt accordingly.

Self-Improvement via Dynamic Code Modification Gödel Agent can engage in reasoning and planning to determine whether it should modify its own logic. If modification is deemed necessary, Gödel Agent generates new code, dynamically writes it into the runtime memory, and integrates it into its operational logic. This dynamic modification allows it to evolve by adding, replacing, or removing logic components as it encounters new challenges, thus achieving self-improvement.

Environmental Interaction To assess performance and gather feedback, Gödel Agent is equipped with interfaces for interacting with its environment. Each task provides tailored environmental interfaces, enabling it to evaluate its performance and adjust its strategies accordingly. In practical implementations, a validation set can be used to provide feedback.