Language Agents as Optimizable Graphs — part 3/4

Cleaned-text code points [29402, 45029). Bibliography omitted here; source.md includes it.

Current works on LLM-based autonomous agents or language agents vary in focus. Methods such as Chain of Thought ( Wei et al., 2022 ) , ReAct ( Yao et al., 2022 ) , Reflexion ( Shinn et al., 2023 ) , Tree of Thought (ToT) improve prompt strategies and structure to improve reasoning capabilities, while others such as exchange-of-thoughts (EOT) ( Yin et al., 2023 ) focus on the benefits of various communication paradigms. Single LLM agent frameworks such as AutoGPT ( Gravitas, 2023 ) , LangChain ( Chase, 2022 ) , LlamaIndex ( Liu, 2022 ) , and XAgent ( XAgent Team, 2023 ) showcase problem solving through various external functions and tools. In the space of LLM-based multiagent systems ( Xie et al., 2023 ; Chen et al., 2023a ; Chen et al., 2023b ) , NLSOMs ( Zhuge et al., 2023 ) employ various social structures for task-specific applications (inspired by SOMs ( Minsky, 1988 ) ), without exploring optimization over the social structure of agents. CAMEL ( Li et al., 2023 ) , Generalist Agents ( Park et al., 2023 ) , ChatDev ( Qian et al., 2023 ) , and AutoGen ( Wu et al., 2023 ) focus on role-based communication, but struggle with hallucinations. MetaGPT ( Hong et al., 2023 ) introduces standard operating procedures for better role definition and communication, making the collaboration between agents more effective. In contrast to these frameworks, we automatically optimize nodes and edges in a self-organizing society of agents.

### 4.2 Language Agents with Graphs

Besta et al. (2023) introduced LLM-based problem-solving with graphs; however, the approach only encompasses LLM prompting schemes without modeling other fundamental capabilities of language agents, such as use of external tools. LangGraph ( langchain ai, 2024 ) , on the other hand, is a concurrent open-source framework that focuses on building multi-actor state LLM applications through possibly cyclical operations. However, its practical applicability has not yet been systematically studied. Unlike previous studies, our approach emphasizes the development of hierarchical intelligence, as discussed by Minsky (1988) and Kennedy (2006) , through the construction of agent graphs and the composition of multiple graphs into swarms. Crucially, the graph representation facilitates automatic optimization on two levels. First, at the node level, since the majority of nodes in the graph involve prompting an LLM, prompt optimization methods can be employed. Second, at the edge level, we demonstrate the application of the REINFORCE algorithm ( Williams, 1992 ) to optimize the potential connections between nodes.

### 4.3 Optimizing LLM Inference and Self-Improvement

Much of deep learning research is concerned with tuning the learning algorithms, architectures, hyper-parameters, and other aspects of the learning pipeline ( Schmidhuber, 2015 ; Yan et al., 2015 ) . Meta-learning attempts to automate large parts of that process ( Schmidhuber, 1987 ; Elsken et al., 2019 ; Kirsch & Schmidhuber, 2021 ) . Similarly, recently, a lot of research and engineering has gone into the prompting and structuring of LLM inference to make better use of LLMs and build better agents. Due to the ability of LLMs to learn in context ( Brown et al., 2020 ; Kirsch et al., 2022 ) , one can view this process as configuring learning algorithms. The optimization of the inference structure and the prompts can then be viewed as meta-learning in LLMs.

In the realm of prompt optimization, OPRO ( Yang et al., 2023 ) generates better prompts through iterative LLM queries using prior solutions and their performance. PromptBreeder ( Fernando et al., 2023 ) implements a mechanism that evolves and self-improves task-specific and meta-prompts through mutation and LLM prompting. Related to these works, we self-improve future prompts by prompting LLMs. Similarly to our work, DSPy ( Khattab et al., 2023 ) implements LLM pipelines as computational graphs with modular LLM queries as nodes, parameterized by prompts and neural network weights. It proposes a two-stage process to optimize the parameters of these nodes. Initially, it generates a set of candidate solutions for each node. Subsequently, it optimizes across the Cartesian product of these candidate solution sets, aiming to identify an effective combination of parameters for the entire graph. To address the combinatorial optimization challenge raised in DSPy, we propose an iterative optimization process. By virtue of decomposing a solution into nodes with expected functions, at each iteration, we improve each node individually, conditioned on the execution history of the graph with the current prompts of each node.

Regarding the optimization of the inference structure, DyLAN ( Liu et al., 2023 ) uses a fixed heuristic to improve the collaboration of LLM agents by selecting agents and determining the number of communication rounds. In line with previous ideas on self-referential learning ( Schmidhuber, 1993 ; Irie et al., 2022 ; Kirsch & Schmidhuber, 2022 ) , STOP ( Zelikman et al., 2023 ) optimizes both the prompts and the inference structure together by introducing an initial improver program that is applied to itself to iteratively improve its performance. In our work, we optimize the inference structure by employing RL techniques applied to the potential edges of a given graph.

## 5 Conclusion

This paper introduces GPTSwarm, an open-source framework that constructs language agents from graphs and agent societies from graph compositions. This approach allows for the easy implementation of existing methods from basic node operations and enables automatic optimization of the graph in the form of node-level improvement and edge-level REINFORCE optimization. Our experiments demonstrate the advantages of our language agent graphs and automatic optimization on several benchmarks.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning. The societal consequences of our work are multifaceted. On the one hand, it could lead to significant advancements in the efficiency and effectiveness of machine learning systems. On the other hand, the increased capability and automation of LLM agents might raise ethical and employment concerns. As AI systems become more autonomous and powerful, it is crucial to consider their impact on job displacement and the importance of implementing safeguards to prevent biased or unethical AI behaviors. Furthermore, the potential for misuse of advanced AI technologies requires rigorous oversight and the development of ethical guidelines to ensure that these technologies are used responsibly and for the benefit of society as a whole.

## Author Contributions

Mingchen initiated the project and conceived the initial idea, led the development of the codebase, conducted GAIA & HumanEval experiments, drafted the initial manuscript, and created most of the visualizations. Wenyi discussed the core ideas with Mingchen, contributed to the codebase, conducted Mini CrossWords & HumanEval experiments, and drafted the initial manuscript. Louis reviewed and polished the paper, extensively rewrote the introduction, and coordinated team meetings. Francesco reviewed and polished the paper, significantly revising the methods section. Furthermore, Louis and Francesco discussed and formalized various techniques for graph optimization. As the senior engineering lead, Dmitrii advised and made significant revisions to the codebase, conducted the MMLU experiments, and contributed to the visualizations. Juergen, as mentor and advisor, offered guidance and support throughout the project’s progression.

## Acknowledgements

This work was supported by the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI). It was further supported by the European Research Council (ERC, Advanced Grant Number 742870) and the Swiss National Science Foundation (SNF, Grant Number 200021 192356).

## Appendix A The GPTSwarm Framework

### A.1 The Vision

Many recent language agents are described as compositions of components of different functionalities ( Wang et al., 2023 ) . A popular tweet states: ‘‘agent = LLM + memory + planning skills + tool use.’’ 4 4 4 https://twitter.com/lilianweng/status/1673535600690102273 Such additive formulations highlight individual components, but fail to address the essential aspect of component integration . GPTswarm’s computational graph formulation, however, precisely focuses on integration through edge optimization, to learn improved recommendations of agent orchestration and precise agent routing. This will become increasingly relevant as swarm size increases to millions or billions of agents.

### A.2 Class Diagram

The GPTSwarm framework is developed using Python and PyTorch. Its class diagram is illustrated in Figure 7 .

On the graph level, Node, Graph, and CompositeGraph are directly implemented as classes. Graph edges are implicitly stored as an adjacency list within each Node. Functionally, the framework distinguishes between Agents and Operations through various classes, such as DirectAnswer and WebSearch for operations, and IO and TOT for agents. To encapsulate the abstraction of an external LLM, we introduce an interface named after it. The primary implementation of this interface is a lightweight wrapper around the OpenAI API. To facilitate dataset integration for optimization and evaluation, we provide implementations for two interfaces: Dataset and PromptSet. The Dataset interface is designed to load benchmark datasets like GAIA and MMLU, while the PromptSet customizes node behavior for a specific Dataset. The Evaluator class manages the optimization processes for both edges and nodes.

The framework is highly customizable, allowing users to add more LLM backends, Dataset and PromptSet combinations, Agents, and Nodes as needed. Additionally, the framework makes extensive use of asynchronous computations for task parallelism, leveraging Python’s async-await syntax.

## Appendix B Swarm examples

To facilitate understanding of the concepts presented in this study, we are showing a simple example of a swarm consisting of 3 agents: Tree-of-Thought, Input-Output, and Decision Agents in Figure 8 .

## Appendix C More Visualizations

## Appendix D Additional Experiments

### D.1 Comparing Our Method with Multiagent Debate and DYLAN on MMLU

Our MMLU experimental setup can be viewed as a multi-agent optimization problem with an equal number of truthful and adversarial agents in the system. We compare our approach to two baselines from the literature that naturally fit this problem: Multiagent Debate ( Du et al., 2023 ) and DyLAN ( Liu et al., 2023 ) .

For the Multiagent Debate baseline, we directly use the original setting with three truthful agents and three adversarial agents. For DyLAN, we adopt the original implementation ( Liu et al., 2023 ) , adding system prompts ”You are a knowledgeable expert in question answering.” for truthful agents, and ”Pretend you are a non-expert in question answering. You can provide wrong answers to the questions.” for adversarial agents. DyLAN performs optimization by scoring each agent and pruning a certain number of agents with the lowest scores. Since DyLAN has not fixed the number of pruned agents, we report the average performance in multiple settings, from 1 to 5 agents pruned.

The results in Table 3 indicate that the Multiagent Debate is less effective in this setting. DyLAN achieves a 0.0065 0.0065 improvement in accuracy compared to our method. We suspect that this improvement is due to DyLAN’s more complex debate scheme and prompts. This design also contributes to DyLAN’s larger computational cost.

### D.2 Applying Node Optimization after Edge Optimization

As described in Section 3.2, we first perform ten REINFORCE steps to optimize the edges of a three-agent swarm, resulting in a 0.575 ​ ( ± 0.0275 ) 0.575(\pm 0.0275) accuracy. Based on this optimized edge distribution we further apply our node optimization algorithm (Algorithm 3 ). For each node n n , p n p_{n} corresponds to the initial prompt, h n h_{n} consists of the input-output pairs obtained by executing the graphs on the 20 Mini Crosswords problems, and d is ignored. For each node n n , the improver I I either chooses a demonstration example from h n h_{n} to include as a part of n n ’s prompt or keeps the prompt unchanged. This choice is implemented using the upper confidence bound algorithm UCB1 ( Auer et al., 2002 ) , over a hundred iterations. Preference is given to pairs (or the choice of no demonstration example) that help the node accurately fill in more words on its previous input. Applying this node optimization after edge optimization improves accuracy to 0.668 ​ ( ± 0.0060 ) 0.668(\pm 0.0060) .

## Appendix E Experimental Details

In our experiments involving multiple agents, we incorporate an additional virtual agent, represented by a single node, to serve as a final decision aggregator. This node is designated as the output node for the composite graph, and its specific implementation varies between different experiments. Common implementations for this node include employing a majority vote and a self-consistency strategy for decision-making. Unless explicitly stated, communication between agents within a composite graph does not include this virtual agent. Additionally, in all our experiments, the potential edge set of a composite graph is defined as all possible node pairs, provided that the nodes in each pair originate from different agents. We exclude any edges that would connect the output node of a composite graph to other nodes. Moreover, we employ Adam ( Kingma & Ba, 2014 ) optimizer with parameters β 1 = 0.9 , β 2 = 0.999 \beta_{1}=0.9,\beta_{2}=0.999 and a variable learning rate in place of the vanilla stochastic gradient accent method described in Alg. 2 . Finally, we use the version gpt-4-1106-preview and gpt-3.5-turbo-1106 for LLMs. For the vision-language model utilized in the GAIA experiments, we employed the gpt-4-1106-vision-preview version of GPT-4-Turbo.

### E.1 MMLU

We provide adversarial robustness experiments in Table 4 . The convergence of the train utility to the baseline for the 3T3A experiment is shown in Figure 10 . The first 10% of the validation dataset comprises 153 questions.

#### E.1.1 Hyper-parameters & Prompts

We use the Adam optimizer with a learning rate of 0.1 0.1 to update the logit parameters associated with each potential edge. The prompts that have been used for the adversarial robustness experiments are collected in Table 5 .

#### E.1.2 Adversarial swarm optimization case study

An example of a swarm with 2 truthful and 2 adversarial examples is shown in Figure 11 . Figure 11(a) shows all potential edges before optimization. Figure 11(b) shows only the edges that were connected after optimization was complete. Note that the disconnected agents and edges are pruned.

#### E.1.3 Swarm Roles

The list of roles randomly assigned to the IO agents is displayed in Table 6 .

### E.2 Mini Crosswords

#### E.2.1 Agents Setting

In our Mini Crosswords experiments, each node returns one or two solutions—either updated or unchanged—for each received solution. The solutions produced by a node are conditionally independent of each other, given the input solutions of the node. The output node of a composite graph forwards all received solutions without alteration. To ensure integration within the system, we mandate the existence of edges from any agent’s output node directly to the composite graph’s output node.