Automated Design of Agentic Systems — part 2/4

Cleaned-text code points [15893, 30870). Bibliography omitted here; source.md includes it.

We conduct extensive experiments on: (1) the ARC challenge ( Chollet, 2019 ) ( Figure 3 ), (2) four popular benchmarks assessing the agent’s abilities on reading comprehension, math, science questions, and multi-task problem solving ( Section 4.2 ), and (3) the transferability of discovered agents on math to held-out math tasks and non-math tasks ( Section 4.3 ). We use an identical implementation of the algorithm across different tasks, with the only variation being task-specific descriptive text included in the prompt (details are available in Appendix B ). Across all experiments, we find that the discovered agents substantially outperform baseline state-of-the-art hand-designed agents and maintain superior performance even when transferred across domains and models.

### 4.1 Case Study: ARC Challenge

We first demonstrate how Meta Agent Search discovers novel agentic systems and outperforms existing state-of-the-art hand-designed agents in the Abstraction and Reasoning Corpus (ARC) challenge ( Chollet, 2019 ) . This challenge aims to evaluate the general intelligence of AI systems through their ability to acquire new skills. Questions in ARC include (1) showing multiple examples of visual input-output grid patterns, (2) the AI system learning the transformation rule of grid patterns from examples, and (3) predicting the output grid pattern given a test input grid pattern. Since each question in ARC has a unique transformation rule, it requires the AI system to learn efficiently with few-shot examples, leveraging capabilities in number counting, geometry, and topology.

Setup. Following common practice ( Greenblatt, 2024 ) , we require the agent to write code for the transformation rule instead of answering directly. We provide tool functions in the framework (described in Section 3 ) that evaluate the generated transformation code. Given the significant challenge that ARC poses to current AI systems, we sample our data from questions with grid dimensions ≤ 5 × 5 \leq 5\times 5 in the “Public Training Set (Easy)”. We sample a validation set and a test set with 20 and 60 questions, respectively, for searching and testing. We calculate the validation and test accuracy of an agent by assessing it over the validation and test sets five times to reduce the variance from the stochastic sampling of FMs. We evaluate all discovered agents on the held-out test set and report the test accuracy in Figure 3 . Meta Agent Search runs for 25 iterations and the meta agent uses GPT-4 ( OpenAI, 2024 ) , while discovered agents and baselines are evaluated using GPT-3.5 ( OpenAI, 2022 ) to reduce compute cost. More algorithmic details and examples of ARC questions can be found in Figure 4 .

Baselines. We compared against five state-of-the-art hand-designed agents: (1) Chain-of-Thought (COT, Wei et al. (2022) ), which instructs the agent to output the reasoning before answering to improve complex problem-solving through intermediate steps; (2) Self-Consistency with Chain-of-Thought (COT-SC, Wang et al. (2023b) ), which ensembles multiple parallel answers from COT to produce a more accurate answer; (3) Self-Refine ( Madaan et al., 2024 ; Shinn et al., 2023 ) , which allows iterative self-reflection to correct mistakes made in previous attempts; (4) LLM-Debate ( Du et al., 2023 ) , which enables different LLMs to debate with each other, leveraging diverse perspectives to find better answers; (5) Quality-Diversity, a simplified version of Intelligent Go-Explore ( Lu et al., 2024c ) , which produces and ensembles diverse answers to better explore potential solutions. The selected baselines represent widely adopted agent designs in the agent literature, embodying key design patterns and approaches frequently utilized across various applications. By “state-of-the-art,” we refer to these baseline designs as exemplifying important advancements and practices within the field. We also use all baselines as initial seeds in the archive for Meta Agent Search, with additional results for empty initialization provided in Appendix I . To ensure fair comparisons, all baseline implementations were developed using the same framework as the Meta Agent, providing a consistent and equitable evaluation environment. More details about baselines can be found in Appendix F .

Results and Analysis. As shown in Figure 3(a) , Meta Agent Search effectively and progressively discovers agents that perform better than state-of-the-art hand-designed baselines. Important breakthroughs are highlighted in the text boxes. As is critical in prior works on open-endedness and AI-GAs ( Zhang et al., 2024a ; Faldor et al., 2024 ; Wang et al., 2019 ; Wang et al., 2020 ; Lehman & Stanley, 2011 ) , Meta Agent Search innovates based on a growing archive of previous stepping stones. For example, an important design pattern emerged in iteration 3 where it uses multiple COTs to generate possible answers, refines them, and finally ensembles the best answers. This became a crucial stepping stone that subsequent designs tended to utilize. Additionally, the best-discovered agent is shown in Figure 3(b) , where a complex feedback mechanism is adopted to refine answers more effectively. Careful observation of the search progress reveals that this sophisticated feedback mechanism did not appear suddenly. Instead, the ideas of incorporating diverse feedback, evaluating for various specific traits (via experts) such as efficiency and simplicity, and simulating human-like feedback emerged in iterations 5, 11, and 12, respectively. The final mechanism is an innovation based on these three stepping stones. This illustrates that even though these stepping stones did not achieve high performance immediately upon emergence, later discoveries benefited from these innovations by combining different stepping stones, resembling crossover in evolution via LLMs ( Meyerson et al., 2023 ) . Overall, the results showcase the potential of ADAS and the effectiveness of Meta Agent Search to progressively discover agents that outperform state-of-the-art hand-designed baselines and invent novel design patterns through the innovation and combination of stepping stones.

### 4.2 Reasoning and Problem-Solving Domains

Setup. Next, we investigate the potential of our algorithm to improve the capabilities of agents across math, reading, and reasoning domains. We test Meta Agent Search on four popular benchmarks: (1) DROP ( Dua et al., 2019 ) for evaluating Reading Comprehension ; (2) MGSM ( Shi et al., 2023 ) for evaluating Math capability under a multi-lingual setting; (3) MMLU ( Hendrycks et al., 2021 ) for evaluating Multi-task Problem Solving; and (4) GPQA ( Rein et al., 2023 ) for evaluating the capability of solving hard (graduate-level) questions in Science . The search is conducted independently within each domain. Meta Agent Search runs for 30 iterations. The meta agent uses GPT-4 ( OpenAI, 2024 ) , while the discovered agents and baselines are evaluated using GPT-3.5 ( OpenAI, 2022 ) . More details about datasets and experiment settings can be found in Appendix E .

Baselines. We adopt all baselines introduced in Figure 3 . Additionally, since the above domains require strong reasoning skills, we include two additional baselines that specifically focus on enhancing the reasoning capabilities of agents for a more thorough comparison: (1) Step-back Abstraction ( Zheng et al., 2023 ) , which instructs agents to first consider the principles involved in solving the task for better reasoning; (2) Role Assignment ( Xu et al., 2023 ) , which assigns different roles to FMs to obtain better answers. Furthermore, we compare our approach with the state-of-the-art prompt optimization baseline OPRO ( Yang et al., 2024 ) to highlight the advantages of learning all possible components of agents rather than focusing solely on prompts. More details about the baselines can be found in Appendix F .

Results and Analysis. The results across multiple domains demonstrate that Meta Agent Search can discover agents that outperform state-of-the-art hand-designed agents ( Table 1 ). We want to highlight the substantial gap between the learned agents and hand-designed agents in the Reading Comprehension and Math domains, with improvements in F1 scores by 13.6 /100 and accuracy rates by 14.4% , respectively. While Meta Agent Search also outperforms baselines in the Multi-task and Science domains, the gap is smaller. We hypothesize that for challenging questions in the Science and Multi-task domains, the knowledge in FMs is not sufficient to solve the questions, limiting the improvement through optimizing agentic systems, which is a problem that will diminish as FMs improve. In contrast, in the Reading Comprehension and Math domains, FMs possess adequate knowledge to solve the questions, and errors could mainly be hallucinations or calculation mistakes, which can be mitigated through well-designed agentic systems, like the ones discovered by Meta Agent Search. Additionally, when compared to prompt optimization methods, the results demonstrate that our proposed Meta Agent Search consistently outperforms them across all domains. This comparison further strengthens our argument that defining agents in code and enabling the learning of all components offer significant advantages. Overall, the results across various domains showcase the effectiveness of Meta Agent Search in searching for agents tailored to specific domains. This could be increasingly useful for saving human efforts and developing better task-specific agents as we continue to create agents for a diverse set of applications ( Wang et al., 2024 ) .

### 4.3 Generalization and transferability

In the previous sections, we illustrated that Meta Agent Search can find effective agents for individual tasks. In this section, we further demonstrate the transferability and generalizability of the discovered agents. To demonstrate the generalizability of the invented building blocks and design patterns, we transfer discovered agents from the MGSM (Math) domain to both math and non-math domains to test their ability to generalize across different tasks. We evaluate the top 3 agents from MGSM by transferring them to (1) popular math domains: GSM8K ( Cobbe et al., 2021 ) , GSM-Hard ( Gao et al., 2023 ) , and (2) non-math domains: MMLU (Multi-task) and DROP (Reading Comprehension), as detailed in Section 4.2 . As shown in Table 2 , Meta Agent Search consistently outperforms the baselines. Notably, our agents improve accuracy by 25.9% on GSM8K and 13.2% on GSM-Hard compared to the baselines when transferring within math domains. More surprisingly, we find that agents discovered in the math domain can also be transferred to non-math domains. While their performance does not fully match agents specifically designed for the target domains, they still outperform state-of-the-art hand-designed baselines. More results of transfers across domains are shown in Appendix A .

We also observe similar superiority when transferring agents across different FMs on ARC. We test the top 3 agents with the best test accuracy evaluated with GPT-3.5 on ARC and then transfer them to Claude-Haiku ( Anthropic, 2024a ) , GPT-4 ( OpenAI, 2024 ) , and Claude-Sonnet ( Anthropic, 2024b ) . As shown in Table 3 , we observe that the searched agents consistently outperform the hand-designed agents, with a substantial gap. Notably, we found that Claude-Sonnet, the most powerful model from Anthropic, performs the best among all tested models, enabling our best agent to achieve nearly 50% accuracy on ARC. These results on transferring across domains and models highlight Meta Agent Search ’s ability to discover generalizable design patterns and agentic systems.

## 5 Related Work

Agentic Systems. Researchers develop various building blocks and design patterns for different applications. Important building blocks for agentic systems include: prompting techniques ( Chen et al., 2023a ; Schulhoff et al., 2024 ) , chain-of-thought-based planning and reasoning methods ( Wei et al., 2022 ; Yao et al., 2023 ; Hu & Clune, 2024 ) , reflection ( Madaan et al., 2024 ; Shinn et al., 2023 ) , developing new skills for embodied agents in code ( Wang et al., 2023a ; Vemprala et al., 2023 ) , external memory and RAG ( Zhang et al., 2024c ; Lewis et al., 2020 ) , tool use ( Qu et al., 2024 ; Schick et al., 2023 ; Nakano et al., 2021 ) , assigning FM modules in the agentic system with different roles and enabling them to collaborate ( Hong et al., 2023 ; Wu et al., 2023 ; Qian et al., 2023 ; Xu et al., 2023 ; Qian et al., 2024 ) , and enabling the agent to instruct itself for the next action ( Richards, 2023 ) , etc. While the community has invested substantial effort in developing all the above important techniques, this is only a partial list of the discovered building blocks, and many more remain to be uncovered. Therefore, in this paper, we describe a newly forming research area, ADAS, which aims to invent novel building blocks and design powerful agentic systems in an automated manner.

AI-Generating Algorithms and AutoML. Research in AI-Generating Algorithms (AI-GAs, Clune (2019) ) and AutoML ( Hutter et al., 2019 ) aims to replace handcrafted components in AI systems by learning them. This field has three key pillars: (1) meta-learning architectures, (2) meta-learning learning algorithms, and (3) generating learning environments and training data ( Clune, 2019 ) . Neural Architecture Search ( Elsken et al., 2019 ; Lu et al., 2019 ; Hu et al., 2021 ) exemplifies the first pillar by automating neural network design, while works like MAML ( Finn et al., 2017 ) and Meta-RL ( Wang et al., 2016 ; Duan et al., 2017 ; Norman & Clune, 2023 ; Zintgraf et al., 2021a ; Zintgraf et al., 2021b ) exemplify the second pillar, focusing on “learning to learn” for improved sample efficiency and generalizability. The third pillar includes works like POET ( Wang et al., 2019 ; Dharna et al., 2022 ; Wang et al., 2020 ) and OMNI-EPIC ( Faldor et al., 2024 ) , which generate learning environments in an open-ended manner. We position Automated Design of Agentic Systems in both the first and second pillars: meta-learning agentic architectures and leveraging in-context learning to “learn to learn,” as shown in the ARC challenge ( Figure 3 ). Furthermore, recent AI-GA and AutoML advances have also integrated Foundation Models (FMs) to write code, as seen in FunSearch ( Romera-Paredes et al., 2024 ) and EoH ( Liu et al., 2024 ) , where FMs discover optimization algorithms. In DiscoPOP ( Lu et al., 2024a ) , FMs program loss functions for preference learning, and Eureka ( Ma et al., 2023 ) and language-to-reward ( Yu et al., 2023 ) enable FMs to write reward functions for reinforcement learning. OMNI-EPIC ( Faldor et al., 2024 ) allows FMs to create robotics learning environments. Similarly, we enable FMs to program new agents in code.