Automated Design of Agentic Systems — part 4/4

Cleaned-text code points [46533, 54385). Bibliography omitted here; source.md includes it.

An example task from the ARC challenge is shown in Figure 4 . In the ARC challenge experiments ( Figure 3 ), we represent the grids as strings of 2-D arrays, where each color is represented by an integer. We instruct the meta agent to design agents that generate code as solutions rather than directly outputting answers. Additionally, we provide two tool functions within the framework: (1) to test whether the generated code can solve the example grids and (2) to obtain the task’s answer by applying the generated code to the test grid. The accuracy rate is calculated by the Exact Match between the reference solution and the predicted answer. The meta agent uses “gpt-4o-2024-05-13” ( OpenAI, 2024 ) , while discovered agents and baselines are evaluated using “gpt-3.5-turbo-0125” ( OpenAI, 2022 ) to reduce compute cost.

The domain description of ARC for the meta agent is shown below:

Here we present the best agent on ARC discovered by Meta Agent Search.

## Appendix E Experiment Details for Reasoning and Problem-Solving Domains

To reduce costs during search and evaluation, we sample subsets of data from each domain. For GPQA (Science), we use GPQA_diamond and the validation set consists of 32 questions, while the remaining 166 questions form the test set. For the other domains, the validation and test sets are sampled with 128 and 800 questions, respectively. We evaluate agents five times for GPQA and once for the other domains to maintain a consistent total number of evaluations. Each domain uses zero-shot style questions, except DROP (Reading Comprehension), which uses one-shot style questions following the practice in ( OpenAI, 2023 ) . The meta agent uses “gpt-4o-2024-05-13” ( OpenAI, 2024 ) , while discovered agents and baselines are evaluated using “gpt-3.5-turbo-0125” ( OpenAI, 2022 ) to reduce compute cost.

We present the description of each domain we provide to the meta agent.

## Appendix F Baselines

In this paper, we implement five state-of-the-art hand-designed agent baselines for experiments on ARC ( Figure 3 ): (1) Chain-of-Thought (COT) ( Wei et al., 2022 ) , (2) Self-Consistency with Chain-of-Thought (COT-SC) ( Wang et al., 2023b ) , (3) Self-Refine ( Madaan et al., 2024 ; Shinn et al., 2023 ) , (4) LLM-Debate ( Du et al., 2023 ) , and (5) Quality-Diversity, a simplified version of Intelligent Go-Explore ( Lu et al., 2024c ) .

In addition to these baselines, we implement two more for experiments on Reasoning and Problem-Solving domains ( Section 4.2 ): (6) Step-back Abstraction ( Zheng et al., 2023 ) and (7) Role Assignment ( Xu et al., 2023 ) . An example implementation of Self-Refine with our simple framework is shown in Appendix C .

In COT, we prompt the FM to think step by step before answering the question. In COT-SC, we sample N = 5 N=5 answers and then perform an ensemble using either majority voting or an FM query. In Self-Refine, we allow up to five refinement iterations, with an early stop if the critic deems the answer correct. In LLM-Debate, each debate module is assigned a unique role, such as Physics Expert or Chemistry Expert, and the debate lasts for two rounds. In Quality-Diversity, we conduct three iterations to collect diverse answers based on previously proposed ones. In Role Assignment, we use an FM query to first choose a role from a predefined set, and then use another FM query to answer the question by acting within the chosen role.

## Appendix G Example Agents

In this section, we present the detailed implementation of three example discovered agents by Meta Agent Search shown in Figure 1 . The “Multi-Step Peer Review Agent” and “Divide and Conquer Agent” were discovered during the search in the Reading Comprehension domain (GPQA) ( Rein et al., 2023 ) , while the “Verified Multimodal Agent” was discovered during the search in the Math domain (MGSM) ( Shi et al., 2023 ) .

## Appendix H Pseudocode of the Meta Agent Search

In this section, we provide the pseudocode for the Meta Agent Search algorithm to clarify its implementation and workflow. The pseudocode outlines the iterative process of designing, evaluating, and refining agents using a meta agent, as described in the main text.

## Appendix I Impact of Initialization

One of the key claims of our work is that the code space representation allows for better utilization of existing human efforts ( Figure 2 ), enabling a more efficient search process than starting entirely from scratch. To further investigate the effects of initialization, we conducted experiments where the Meta Agent Search algorithm was run without any initial agent designs, contrasting with our standard approach that incorporates human-designed solutions into the search process.

The results, presented in Table 6 , demonstrate that even without initial agent designs, Meta Agent Search discovers agents that outperform all hand-crafted baselines across all evaluated domains. This finding underscores the robustness of our method, as it effectively leverages the inherent structure of the code space to explore and optimize agent designs.

Interestingly, while the inclusion of good initial solutions generally leads to improved performance, the math domain exhibited a unique outcome: starting from scratch resulted in superior performance. We hypothesize that the absence of predefined design patterns in this case encouraged a broader and more diverse exploration of reasoning strategies within the limited number of iterations. Such diversity appears particularly beneficial for math tasks, which demand flexible and varied approaches to reasoning.

This observation opens up an intriguing avenue for future research: exploring how the choice and quality of initialization impact search effectiveness across different domains. For instance, it would be valuable to identify conditions under which starting without initial solutions may yield performance gains, or to design strategies that combine the advantages of both initialization and broad exploration.

## Appendix J Cost of Experiments

A single run of search and evaluation on ARC ( Figure 3 ) costs approximately $500 USD in OpenAI API costs, while a run within the reasoning and problem-solving domains ( Section 4.2 ) costs about $300 USD.

The primary expense comes from querying the “gpt-3.5-turbo-0125” model during the evaluation of discovered agents. Notably, the latest GPT-4 model, “gpt-4o-mini,” is less than one-third the price of “gpt-3.5-turbo-0125” and offers better performance, suggesting that we could achieve improved results with Meta Agent Search at just one-third of the cost. Additionally, as discussed in Section 6 , the current naive evaluation function is both expensive and overlooks valuable information. We anticipate that future work adopting more sophisticated evaluation functions could significantly reduce the cost of ADAS algorithms.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .