Language Agents as Optimizable Graphs — part 2/4
Cleaned-text code points [14777, 29402). Bibliography omitted here; source.md includes it.
Consider a parameterized computational graph G P = ( N , E , F P , o ) G^{P}=(N,E,F^{P},o) , where F P = { f n p n } F^{P}=\{f_{n}^{p_{n}}\} are computational routines, each parameterized by a prompt p n p_{n} to be optimized for all n ∈ N n\in N . To enable effective node optimization, we also require a natural language description of the intended function for each routine f n p n ∈ F f_{n}^{p_{n}}\in F denoted by d n d_{n} . For example, a suitable description for a node designed to write Python programs would be “a Python code generator”. Here, existing prompt optimization methods, such as OPRO ( Yang et al., 2023 ) , can be described as a function I I that iteratively maps a prompt, a function description, and a set of node input-output pairs (which may include annotations such as a quality measure for each pair) to an improved prompt. For example, I I could take a prompt such as “generate Python code”, a description “a Python code generator”, and an input-output pair “Input: evaluate two divided by one as an integer. Output: 2 / 1”, where the output yields 1.0 1.0 as the result of execution. A prompt optimization method would return an improved prompt “generate Python code and pay attention to data types”.
Formally, our method begins by initializing an empty history set, denoted h n h_{n} , one for each node n ∈ N n\in N . The process then proceeds iteratively: first, the graph G P ( x ) G^{P}(x) is executed using a randomly sampled input x x following Algorithm 1 . Subsequently, for each node, a tuple consisting of the input to the node ( z n , x ) (z_{n},x) , where z n z_{n} is the context vector that includes the outputs of the predecessor nodes, and the node’s own output f n p n ( z n , x ) f_{n}^{p_{n}}(z_{n},x) , is added to the node’s history h n h_{n} . The final step involves updating the node prompts. This is done by applying I I to the node’s updated history, its current prompt, and its function description, resulting in an improved prompt I ( h n , p n , d n ) I(h_{n},p_{n},d_{n}) . This iterative process, described in Algorithm 3 , continuously improves the operations of the nodes in the entire graph.
### 2.5 General Applicability
Frameworks such as AutoGPT ( Torantulino et al., 2023 ) and LangChain ( Chase, 2022 ) have set a standard for flexibility and reusability in various language-based tasks. Our framework, GPTSwarm, introduces a graph-based design of agents and swarms. This design further simplifies the reuse of modular components (nodes & agents) and the integration of such modules. For instance, GPTSwarm supports 41 types of file analysis, web search (e.g., Google Search), and index-based memory. By offering a wide range of modules, our framework makes it easier to implement various language agent systems. See Section 3.4 for further details.
## 3 Experiments
### 3.1 MMLU
#### 3.1.1 Adversarial setting
Motivation: In our first experiments, we demonstrate that edge optimization effectively filters adversarial agents from a swarm, mirroring a scenario in multi-agent systems where some agents are detrimental rather than beneficial. Ideally, optimization would automatically eliminate harmful agents. We conducted this experiment using the 4-choice MMLU general knowledge question answering dataset, as detailed by Hendrycks et al. (2021b) ; Hendrycks et al. (2021a) . Our setup involves initializing a swarm consisting of k k Input-Output (IO) agents and k k adversarial agents, following the terminology by Besta et al. (2023) . The IO agents query an LLM and relay the LLM’s responses directly. In contrast, adversarial agents are deliberately programmed to manipulate the LLM to provide incorrect answers. The collective decision on the final answer is made through majority voting, bypassing any additional LLM query that could introduce corrective intelligence against adversarial influence. We benchmark the performance of a single IO agent as our baseline. An effective optimization is expected to elevate the swarm’s performance on the MMLU dataset to match this baseline level.
Analysis: In Figure 2 , we present the comparative performance scores of different swarm configurations: the baseline, the graph formed by sequentially including edges that do not create loops (denoted as the ‘full graph’), a randomly connected swarm sampled from the initial distribution D θ D_{\theta} with θ = 0.5 \theta=0.5 , and the optimized swarm. These scores are derived from evaluating the initial 10% of the MMLU validation set. The edge optimization process applies REINFORCE ( Alg. 2 ) for 200 iterations. Each iteration assesses four graph samples, each on a specific problem sourced from the MMLU dev set. In all experiments, we used GPT-4-Turbo with the token sampling temperature 0.2 0.2 . Figure 10 demonstrates how the optimized swarm score aligns asymptotically with that of the baseline. Table 4 compiles the key statistics and findings of these experiments. The findings indicate that our approach successfully safeguards a swarm against harmful adversaries.
#### 3.1.2 Collaborative setting
In the adversarial setting above, we observe no score improvement over the single agent baseline. We attribute this to the fact that all IO agents are prompted identically. To elicit score improvement, we run a set of 7 different IO agents instructed to behave according to various roles. See Sec. E.1.3 for the list of roles. The optimized swarm improves over the baseline from Sec. 3.1.1 by 2.1% ± \pm 1.1% averaged over 5 training seeds.
### 3.2 Mini Crosswords
Motivation: This section investigates to what extent edge optimization can improve the performance of standard agents from the literature. We conduct our evaluation on the Mini Crosswords dataset 1 1 1 https://www.goobix.com/crosswords . A subset of 20 problems is used to optimize and evaluate our methods, in agreement with previous studies ( Yao et al., 2023 ; Sel et al., 2023 ) . The choice of Mini Crosswords for this analysis is strategic, as it highlights how the algorithmic structure of the solvers, such as the tree search employed by TOT, significantly influences their performance ( Yao et al., 2023 ) . Our hypothesis is that edge connections can meaningfully determine the algorithmic structure. Through edge optimization, we anticipate the automatic discovery and implementation of high-performance algorithms.
Analysis: In our experiments, we explore the performance of swarms of three distinct agents. The first agent, which implements the TOT approach, iteratively branches over candidate solutions provided by an LLM, processing one word at each step. The second agent is based on the Reflexion method ( Shinn et al., 2023 ) . This agent first proposes a solution through a greedy approach and then creates an alternative solution informed by feedback from a critic, which is based on an LLM analysis of the initial solution. The third agent we examine is a Chain of Thought (COT) agent consisting of three nodes. Each node within the COT performs an internal brute-force search to select the optimal subset of candidates generated by the LLM for the current state, scored by the LLM. The agent or swarm then returns all the solutions generated by their output node.
For the utility function, we choose the best of all the graph-returned solutions according to the number of words correctly filled (i.e., best state word accuracy) as done by Yao et al. (2023) . During the evaluation, we average over 20 graph samples from the graph distribution, each evaluated on a unique question randomly sampled from the dataset.
We optimize our composite graph of agents using the REINFORCE (Alg. 2 ), setting the initial edge probability to θ = 10 % \theta=10\% and the learning rate to α = 0.4 \alpha=0.4 . For each iteration, the gradient is estimated according to equation ( 2 ) by sampling M = 20 M=20 graphs, each evaluated on a crossword problem. For cost-effectiveness, we optimize and evaluate graphs with the GPT-3.5-Turbo language model, where the temperature is set to zero. Figure 3 visualizes the evolution of probability parameters in the form of adjacency-like matrices over ten iterations. We observe that the parameters first change chaotically. However, after iteration 6, the parameters change almost monotonically.
We follow Alg. 2 to optimize the objective in Equation 1 , achieving an average accuracy of 0.575 ( ± 0.0275 ) 0.575(\pm 0.0275) after ten iterations (we report the average over 3 3 runs and the standard error). This surpasses the initial distribution’s score of 0.465 ( ± 0.0509 ) 0.465(\pm 0.0509) . Furthermore, we evaluate the best-of-three performance by aggregating the top results from each problem across the three agents, which yields an accuracy of 0.320 ( ± 0.0415 ) 0.320(\pm 0.0415) .
Note that denser graphs are likely to require more computational resources. To verify that the improvements of our method are not solely due to an increase in the number of edges and therefore a larger computational budget, we compare it with a distribution with all parameters set to θ = 12.5 % \theta=12.5\% . This value reflects the average number of edges in the learned distribution, determined by sampling 1000 graphs from each run’s resulting distribution. The expected number of edges for both the learned distribution and the 0.125 0.125 parameter-valued distribution are approximately 32.76 ( ± 1.93 ) 32.76(\pm 1.93) and 32.80 ( ± 0.11 ) 32.80(\pm 0.11) , respectively. Despite the similarity in the edge count, the 0.125 0.125 parameter-valued distribution achieves an accuracy of 0.510 ( ± 0.0552 ) 0.510(\pm 0.0552) , allowing us to attribute the improvements to factors beyond the mere edge density.
Furthermore, we evaluate one of our final optimized distributions (randomly selected) with the GPT-4-Turbo language model 2 2 2 We are limiting our evaluation to a single graph distribution due to the high cost associated with API calls for this type of evaluation. (Figure 4 ). It achieves an accuracy of 0.800 ( ± 0.0616 ) 0.800(\pm 0.0616) , significantly outperforming the previous state-of-the-art method, TOT evaluated with GPT-4, with an accuracy of 0.675 0.675 ( Yao et al., 2023 ) . For a fair comparison, we also evaluate the TOT implementation by Yao et al. (2023) with GPT-4-Turbo. It yields an accuracy of 0.668 0.668 .
### 3.3 HumanEval
Motivation: In the previous experiments on MMLU (math problems) and Mini Crosswords (open-ended puzzles) we have validated the utility of optimizing graph edges. In this section, we test the HumanEval dataset ( Chen et al., 2021 ) , which is known to be sensitive to prompt design. Previous research involved manually crafting prompts ( Shinn et al., 2023 ; Hong et al., 2023 ) and achieved impressive performance. In contrast, here, we explore how node-based optimization can simplify this process.
Analysis: In this section, we optimize the prompts of a ReAct-style ( Yao et al., 2022 ) agent. The agent first generates a Python program in response to a given question. If the generated program passes all test cases included in the problem statement, then the program is returned. Otherwise, the agent regenerates a program based on the execution feedback. We optimize the prompts by adding input-output pairs selected by assessing their effectiveness as demonstration examples, particularly in improving the node operation applied to the node’s ten most recent inputs. To evaluate a node operation, we determine if the generated program successfully solves the unit tests provided in the input problem statement. For more details on the node optimizer, please refer to Appendix E.3.1 .
Figure 5 shows the optimization results of the first eight iterations. After each iteration, the optimized solution is evaluated on the entire dataset. The mean and standard errors are presented for three repeated runs. We observe that the accuracy increases during the first five iterations. We also test our method in an online learning setting, continuously optimizing and evaluating without restarting. An improvement is observed from 0.76 (no optimization) to 0.88 ( ± 0.007 ) 0.88(\pm 0.007) (mean and standard error).
### 3.4 GAIA
Motivation: GAIA is a benchmark specifically designed for testing the generality of AI assistants focusing on real-world questions ( Mialon et al., 2023 ) . Abilities required to answer GAIA questions include reasoning, multi-modality processing, web browsing, and other tool use. Although conceptually straightforward for humans, these questions present significant challenges for current AI systems.
Using this benchmark, we evaluate the general applicability of our framework. We construct swarms with multiple agents of the same type and employ self-consistency (a prompt-based majority vote) for the final decision ( Wang et al., 2022 ) . We also experimented with adding different types of agents to the swarm and using prompt-based best answer selection. The results indicate that prompt-based self-consistency yields the best performance. Note that these experiments are meant to demonstrate the generic capabilities of our modular framework and include neither edge-based nor node-level optimization, which is left for future work.
Analysis: Table 1 shows the results of our swarm with seven TOT agents and the self-consistency strategy for the final decision. We compare the performance of the GPT-Series ( Achiam et al., 2023 ) with plugins and AutoGPT ( Torantulino et al., 2023 ) performance as reported by Mialon et al. (2023) . Our methods significantly outperform these baselines.
Table 2 presents a more comprehensive set of results. We experiment with varying numbers of agents and different node operations, such as different tool uses. Our observations indicate that the time requirement of a swarm grows approximately linearly with the number of agents. Despite the increased computational time, incorporating more agents notably improves the overall performance of the system. We also found that a greater variety of node operations leads to better performance. As illustrated in Figure 6 , web browsing is required for 43.9% of the tasks. Our current implementation accesses the Internet by only downloading materials directly from the URLs provided in the problem statement or querying a Google search 3 3 3 We use SearchApi (https://www.searchapi.io/) in the experiments without further website navigation. Therefore, we believe that enhancing web capabilities would further increase performance significantly.
## 4 Related Work
### 4.1 LLM-based Autonomous Agents