Language Agents as Optimizable Graphs — part 1/4

Cleaned-text code points [0, 14777). Bibliography omitted here; source.md includes it.

# GPTSwarm: Language Agents as Optimizable Graphs

###### Abstract

Various human-designed prompt engineering techniques have been proposed to improve problem solvers based on Large Language Models (LLMs), yielding many disparate code bases. We unify these approaches by describing LLM-based agents as computational graphs. The nodes implement functions to process multimodal data or query LLMs, and the edges describe the information flow between operations. Graphs can be recursively combined into larger composite graphs representing hierarchies of inter-agent collaboration (where edges connect operations of different agents). Our novel automatic graph optimizers (1) refine node-level LLM prompts (node optimization) and (2) improve agent orchestration by changing graph connectivity (edge optimization). Experiments demonstrate that our framework can be used to efficiently develop, integrate, and automatically improve various LLM agents. The code can be found here .

###### Keywords:

https://gptswarm.org

## 1 Introduction

Interest in LLM-powered autonomous problem solvers or agents and their varied applications is continually rising ( Wang et al., 2023 ; Xi et al., 2023 ) . However, much work remains to be done to effectively incorporate these agents into a cohesive society and improve their structure automatically.

Early approaches zero-shot-prompted LLMs or prompted them with few-shot examples ( Kojima et al., 2022 ; Brown et al., 2020 ) . Recent methods prompt LLMs in a structured way, such as chain of thought (COT) ( Wei et al., 2022 ) , ReAct ( Yao et al., 2022 ) , tree of thought (TOT) ( Yao et al., 2023 ) , Reflexion ( Shinn et al., 2023 ) , and Graph of Thought (GOT) ( Besta et al., 2023 ) , to improve text-based reasoning. Single agent applications such as AutoGPT ( Torantulino et al., 2023 ) , BabyAGI ( Nakajima, 2023 ) , LangChain ( Chase, 2022 ) , and Llama-index ( Liu, 2022 ) utilize LLMs for various functionalities, including tool usage, function calling, and embodied actions. In multi-agent frameworks ( Zeng et al., 2022 ; Zhuge et al., 2023 ) several LLMs take on different roles ( Li et al., 2023 ; Park et al., 2023 ; Qian et al., 2023 ; Wu et al., 2023 ) to communicate in natural language and collectively solve a given task. This approach often outperforms single agents, exploiting the specialization ( Hong et al., 2023 ) of various LLM agents. Unfortunately, it also leads to increasingly different and disparate code bases that require a lot of human engineering to define prompting schemes and the workflow of agents.

In a “society of mind” (SOM) ( Minsky, 1988 ; Zhuge et al., 2023 ) , higher-level intelligence emerges from the combination of simpler and modular cognitive components. Inspired by SOMs, we describe language agent systems through graph representations. Language agents querying LLMs and utilizing external tools are modeled as computational graphs where each node is dedicated to a specific function, while the edges define a topology of how inputs are processed across nodes, mirroring the prompting schemes in prior studies. A swarm is defined as a composite graph, where each subgraph represents a collaborative agent. This creates a deeper hierarchy of intelligence. Agent graphs combine basic LLM operations ( Kennedy, 2006 ; Nepusz & Vicsek, 2013 ) , and swarm graphs contain subgraphs representing agents. Approaches such as COT ( Wei et al., 2022 ) , TOT ( Yao et al., 2023 ) , and Self-Consistency ( Wang et al., 2022 ) can be represented by our graphs.

Our graph representation lends itself to optimization via prompting and evolutionary or reinforcement-learning techniques, so that agents can improve their communication (or orchestration) patterns. The graph connectivity (adjacency matrices) between agents can self-improve online as a task is being solved or its solution is transferred to another task.

As a proof-of-concept, we demonstrate how suboptimal agent organization can be overcome and how existing prompting techniques, such as Tree of Thought and Reflexion, can be automatically recombined by optimizing edges in a composite graph. Apart from edge optimization, our framework allows each node in the graph to self-improve by adapting its prompts based on previous input and task feedback.

Our contributions can be summarized as follows:

(1) We unify language agent systems by describing them as optimizable computational graphs.

(2) We introduce an open-source framework that allows for constructing arbitrary agent systems by recombining fundamental operations. We describe these engineering-level contributions in Appendix A .

(3) We develop optimization methods for nodes and edges, enabling automatic improvements of agent prompts and inter-agent orchestration.

(4) We validate our framework on various benchmarks including MMLU, Mini CrossWords, HumanEval, and GAIA, with an emphasis on the benefits of automatic graph optimization.

## 2 GPTSwarm

### 2.1 Language Agents as Graphs

Taking inspiration from the society of mind (SOM) ( Minsky, 1988 ; Zhuge et al., 2023 ) , we propose to organize intelligence within a modular and hierarchical framework. This framework consists of nodes, graphs, and composite graphs, with each component playing a specific role. A node represents a fundamental operation that includes, but is not limited to, LLM inference, tool use, function calls, and various embodied actions. An agent, conceptualized as a graph, consists of multiple nodes that form a coherent functional entity. A swarm, or composite graph, represents a complex system of agents where the collective capabilities of this system may exceed those of individual agents. Finally, the edges within an agent define its execution topology, while the edges between agents establish collaboration and communication among them.

### 2.2 Graph Definition

##### Single language agent as a graph.

We model a language agent as a directed computational graph G G , defined by a tuple ( N , E , F , o ) (N,E,F,o) , where N N is a set of computational nodes, E ⊂ N × N E\subset N\times N is a set of directed edges, F = { f n } n ∈ N F=\{f_{n}\}_{n\in N} is a set of computational routines and o ∈ N o\in N is an output node. The set of predecessors of node n n is denoted by pre ⁡ ( n ) \pre(n) . In this paper, we focus on directed acyclic graphs (DAGs). Given an input x x , a graph G G iteratively executes its nodes according to their topological order. Each node n ∈ N n\in N receives as input x x and the output z n z_{n} from its predecessor nodes. In this work, inputs and outputs are strings in natural language, but may take on other data types more generally. Node n n applies the computational routine f n ​ ( z n , x ) f_{n}(z_{n},x) and sends the output to its successor nodes. The graph output, denoted y ^ = G ⁡ ( x ) \hat{y}=G(x) , is the output f o ​ ( z o , x ) f_{o}(z_{o},x) from the output node o o . Note that in a DAG, some nodes will not have predecessors. For such nodes, the context z z will be empty. This graph execution procedure is summarized in Algorithm 1 .

In the context of language agents, for example, the input x x may correspond to a question in natural language. Each node processes the input x x and context information z z from its predecessor nodes by applying a computational routine f f . Examples of routines include LLM queries with input data from other agents, instructions to generate prompts for web searches that gather task-related information, or tool usage. Although our formalization specifies that the input x x is given to each node, in practice, many routines might be designed to ignore the input and operate solely in the context provided by the predecessor nodes. Finally, the output provided by the output node corresponds to the answer to the input question or, more generally, to the solution of the input task.

##### Swarm of language agents as a composite graph.

Given a set of K K language agents, each represented by a computational graph { G k = ( N k , E k , F k , o k ) } k = 1 K \{G_{k}=(N_{k},E_{k},F_{k},o_{k})\}_{k=1}^{K} , one can compose these agents to achieve high performance in specific tasks. Let N ′ = ∪ k N k N^{\prime}=\cup_{k}N_{k} represent the union of the nodes of the agents, E ′ = ∪ k E k E^{\prime}=\cup_{k}E_{k} be the union of the edges of the agents, F ′ = ∪ k F k F^{\prime}=\cup_{k}F_{k} be the union of the computational routines of the agents, and o ′ ∈ ∪ k { o k } o^{\prime}\in\cup_{k}\{o_{k}\} be the output node for the composite graph. Consider a selection of edges ℰ ⊂ ∪ i ≠ j N i × N j \mathcal{E}\subset\cup_{i\neq j}N_{i}\times N_{j} that describe a set of connections between nodes from different agents. We define the composite graph representing the swarm of agents as G ℰ = ( N ′ , E ℰ , F ′ , o ′ ) G_{\mathcal{E}}=(N^{\prime},E_{\mathcal{E}},F^{\prime},o^{\prime}) , where E ℰ = E ′ ∪ ℰ E_{\mathcal{E}}=E^{\prime}\cup\mathcal{E} is the union of the edges of the agents and the new edges connecting them. Composite graphs are restricted to DAGs. The composite graph G ℰ G_{\mathcal{E}} can be executed as described in Algorithm 1 . In a swarm of language agents, the newly specified edges represent communication channels between agents. In the following sections, we explore how to optimize such a computational graph.

### 2.3 Edge Optimization

Given a task τ \tau and its associated utility function u τ u_{\tau} that maps the candidate graphs to real numbers, we formulate an optimization problem about the choice of additional edges. The goal is to identify the edges that connect various language agents in a swarm, maximizing the utility. This process involves determining the most effective patterns of communication and information exchange among agents for the task at hand. We consider a set of potential edges { e i } i = 1 d = ℰ \{e_{i}\}_{i=1}^{d}=\mathcal{E} , which leads to 2 d 2^{d} possible edge configurations, symbolized as ℰ ∈ { 0 , 1 } d \mathcal{E}\in\{0,1\}^{d} . We further restrict the search space to only consider composite graphs that are DAGs. Formally, optimization of the composite graph of language agents is achieved by solving the problem max ℰ ⁡ u τ ​ ( G ℰ ) \max_{\mathcal{E}}u_{\tau}(G_{\mathcal{E}}) .

#### 2.3.1 Problem Reformulation

DAG optimization through pruning of nodes and edges was already present in the first work on “deep learning” with deep feedforward networks ( Ivakhnenko et al., 1965 ; Ivakhnenko, 1968 ) . Due to the combinatorial complexity induced by DAGs, recent studies have increasingly focused on the continuous optimization approach ( Vowels et al., 2022 ) . This is particularly relevant in scenarios where most node executions require one or more queries to LLMs for moderate-scale applications. Moreover, the utility function is typically non-differentiable due to the tokenization of LLMs, and this remains true even when a differentiable DAG sampling technique is employed. Therefore, we reformulate our edge optimization as a continuous optimization problem. Instead of optimizing in a discrete space, our approach is to optimize over a continuum of probabilistic distributions, each representing a distribution over the feasible DAGs. Formally, rather than solving the maximum utility function arg ​ max ℰ ⁡ u τ ​ ( G ℰ ) \argmax_{\mathcal{E}}u_{\tau}(G_{\mathcal{E}}) , we propose solving arg ​ max θ ∈ Θ ⁡ 𝔼 G ′ ∼ D θ ​ [ u τ ​ ( G ′ ) ] , \displaystyle\argmax_{\theta\in\Theta}\E_{G^{\prime}\sim D_{\theta}}[u_{\tau}(G^{\prime})], (1) where D θ D_{\theta} is a parameterized distribution and Θ \Theta represents a feasible set of real-valued parameters.

#### 2.3.2 Solution Parameterization

A straightforward way to define a parameterized probabilistic distribution over DAGs with fixed nodes N N and required edges E E is to assign a real-valued parameter θ i ∈ ℝ \theta_{i}\in\R to each potential edge e i e_{i} . Let θ = [ θ 1 ; θ 2 ; … ; θ d ] ∈ [ 0 , 1 ] d \theta=[\theta_{1};\,\theta_{2};\,\dots;\,\theta_{d}]\in[0,1]^{d} . The probability of G ′ = G ℰ G^{\prime}=G_{\mathcal{E}} for G ′ ∼ D θ G^{\prime}\sim D_{\theta} is ∏ i = 1 d { θ i if ​ ( N , E ∪ ( { e j } j = 1 i − 1 ∩ ℰ ) ∪ { e i } ) ​ is a DAG, 0 otherwise. \displaystyle\prod_{i=1}^{d}\begin{cases}\theta_{i}\quad&\text{if}\,(N,E\cup(\{e_{j}\}_{j=1}^{i-1}\cap\mathcal{E})\cup\{e_{i}\})\,\text{is a DAG,}\\ 0&\text{otherwise.}\end{cases} A sampling method that realizes this distribution is first to initialize a graph G ′ ← ( N , E ) G^{\prime}\leftarrow(N,E) . Then, iteratively sample whether to include edge e i e_{i} in G ′ G^{\prime} for all i i ’s. If including e i e_{i} causes a cycle in current G ′ G^{\prime} , then the edge would not be included. Otherwise, add the edge to G ′ G^{\prime} with probability θ i \theta_{i} .

#### 2.3.3 Optimization Algorithm

To optimize the objective function ( Equation 1 ), we apply the REINFORCE algorithm ( Williams, 1992 ) by applying a gradient ascent variant (e.g., Adam ( Kingma & Ba, 2014 ) ) with an unbiased gradient estimation: ∇ θ 𝔼 G ℰ ∼ D θ ​ [ u τ ​ ( G ℰ ) ] ≈ 1 M ​ ∑ i = 1 M u ^ τ ​ ( G i ) ​ ∇ θ ​ log ⁡ ( p θ ​ ( G i ) ) , \displaystyle\nabla_{\theta}\E_{G_{\mathcal{E}}\sim D_{\theta}\ }[u_{\tau}(G_{\mathcal{E}})]\approx\frac{1}{M}\sum_{i=1}^{M}\hat{u}_{\tau}(G_{i})\nabla_{\theta}\log(p_{\theta}(G_{i})), (2) where G 1 , G 2 , … , G N ∼ D θ G_{1},G_{2},\dots,G_{N}\sim D_{\theta} are mutually independent and u ^ τ ​ ( G i ) \hat{u}_{\tau}(G_{i}) is an independent unbiased estimate of u τ ​ ( G i ) u_{\tau}(G_{i}) for all i i and some M ∈ ℕ M\in\mathbb{N} . Algorithm 2 describes the optimization algorithm with vanilla gradient ascent.

### 2.4 Node Optimization

In our framework, each node implements a fundamental operation, such as querying an LLM, using a tool, calling an API, etc. In a language agent, most of these operations involve prompting an LLM once or several times. Optimizing the prompts of these nodes is crucial for improving the system’s overall performance.

Unlike many other works on prompt optimization, which optimize a single global prompt ( Yang et al., 2023 ; Pryzant et al., 2023 ; Deng et al., 2022 , e.g.,) , our node optimization problem naturally involves several operations where each of them consists of a node-level prompt. In our case, the optimization problem is more complex due to prompts affecting how other prompts operate on connected nodes. At the same time, our graph representation leads to a separation of concerns where each node has a specific purpose with its own associated prompt. Due to this separation of concerns, we hypothesize that, for every optimization step, it is sufficient to update each node-level prompt individually, assuming that all other prompts are fixed.