# GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning

###### Abstract

Large language models (LLMs) are increasingly adapted to downstream tasks via reinforcement learning (RL) methods like Group Relative Policy Optimization (GRPO), which often require thousands of rollouts to learn new tasks. We argue that the interpretable nature of language often provides a much richer learning medium for LLMs, compared to policy gradients derived from sparse, scalar rewards. To test this, we introduce GEPA ( Ge netic- Pa reto), a prompt optimizer that thoroughly incorporates natural language reflection to learn high-level rules from trial and error. Given any AI system containing one or more LLM prompts, GEPA samples trajectories (e.g., reasoning, tool calls, and tool outputs) and reflects on them in natural language to diagnose problems, propose and test prompt updates, and combine complementary lessons from the Pareto frontier of its own attempts. As a result of GEPA’s design, it can often turn even just a few rollouts into a large quality gain. Across six tasks, GEPA outperforms GRPO by 6% on average and by up to 20%, while using up to 35x fewer rollouts. GEPA also outperforms the leading prompt optimizer, MIPROv2, by over 10% (e.g., +12% accuracy on AIME-2025), and demonstrates promising results as an inference-time search strategy for code optimization. We release our code at https://github.com/gepa-ai/gepa .

## 1 Introduction

Large language models (LLMs) have enabled development of agents and systems that combine fuzzy natural-language specifications with tools like retrieval and code execution. This raises the question of how LLMs should be optimized for downstream performance. One popular approach is Reinforcement Learning with Verifiable Rewards (RLVR), e.g. with Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) , which treats success metrics as end-of-rollout scalar rewards used to estimate policy gradients ( Lambert, 2025 ) . While these RL approaches are effective, they typically require tens of thousands of rollouts in practice to fit new tasks. For example, recent works leveraging GRPO typically use up to hundreds of thousands of rollouts for training ( Chen et al., 2025c ; Wu et al., 2025b ; Zhang et al., 2025 ; Jin et al., 2025 ; Si et al., 2025 ; Wang et al., 2025a ; Chen et al., 2025a ; Sha et al., 2025 ; Lin et al., 2025a ; Peng et al., 2025 ; Song et al., 2025 ) . This sample inefficiency can quickly become a serious bottleneck: many downstream LLM applications invoke expensive tool calls, have limited inference budget for sampling from the LLM itself, or simply cannot finetune the weights of the largest or best-performing LLMs.

We observe that rollouts sampled from even highly sophisticated LLM systems can be serialized into traces of natural language: they contain nothing but the instructions of each LLM module, the resulting LLM reasoning chains, tool calls, and potentially the internal workings of the reward function (e.g., compiler error messages, before they are collapsed into scalar rewards). Because such serialized trajectories are readily understood by modern LLMs, we argue that algorithms that learn deliberately in natural language by reflecting on these trajectories can make more effective use of the strong language priors that LLMs have, compared with standard RL approaches.

We introduce GEPA (Genetic-Pareto), a reflective prompt optimizer for compound AI systems that merges textual reflection with multi-objective evolutionary search. GEPA iteratively mutates prompts using natural language feedback drawn from new rollouts. In each mutation, the candidate prompt is derived from an ancestor, accumulating high-level lessons derived from observations and LLM feedback. To avoid local optima that afflict greedy prompt optimization, GEPA maintains a Pareto front: instead of evolving only the global best prompt, it stochastically explores the top-performing prompts for each problem instance. This diversification enables robust generalization and mitigates getting stuck in local minima.

We evaluate GEPA across multi-hop reasoning (HotpotQA; Yang et al. 2018 ), Math (AIME, LiveBench-Math; Balunović et al. (2025) ; White et al. (2025) ), instruction following (IFBench; Pyatkin et al. 2025a ), privacy-aware delegation (PUPA; Li et al. 2025a ), and retrieval-augmented verification (HoVer; Jiang et al. 2020 ), using both open (Qwen3 8B; Yang et al. 2025 ; Team 2025 ) and proprietary (GPT-4.1 Mini; OpenAI 2025 ) models. We find that GEPA generalizes well and is highly sample-efficient: on Qwen3 8B, it outperforms GRPO (24k rollouts) by up to 20% while using up to 35 × \times fewer rollouts, with an average gain of +6% across six tasks. GEPA also surpasses the prior state-of-the-art, MIPROv2 ( Opsahl-Ong et al., 2024 ) , on all benchmarks and models, achieving +13% aggregate gains, over double MIPROv2’s +5.6%.

Qualitatively, GEPA-learned prompts are quite rich. Figure 2 shows excerpts from a prompt crafted for the query-creation module of a multi-hop question answering system used in HotpotQA, and Figure 5 shows that even a single reflective update often yields large gains. These results highlight that reflective prompt evolution with language feedback enables improved sample efficiency and robust generalization, offering a practical approach to optimizing complex AI workflows in data- or budget-constrained environments. We also demonstrate GEPA as an inference-time search strategy for code optimization on NPUEval ( Kalade and Schelle, 2025 ) & KernelBench ( Ouyang et al., 2025 ) in Sec 5.1 , and for adversarial prompt search in Sec 5.2 .

## 2 Problem Statement

We follow related work in defining a compound AI system as any modular system composed of one or more language model (LLM) invocations, potentially interleaved with external tool calls, orchestrated through arbitrary control flow. This definition subsumes a broad class of real-world LLM-based AI systems, including agents , multi-agent systems , and general-purpose scaffolding techniques like ReAct ( Yao et al., 2023 ) , Archon ( Saad-Falcon et al., 2025 ) , etc. Following Soylu et al. (2024) ; Khattab et al. (2024) ; Opsahl-Ong et al. (2024) ; Tan et al. (2025) , we formalize such a system as Φ = ( M , C , 𝒳 , 𝒴 ) \Phi=(M,C,\mathcal{X},\mathcal{Y}) , where M = ⟨ M 1 , … , M | M | ⟩ M=\langle M_{1},\ldots,M_{|M|}\rangle denotes language modules, C C specifies control flow logic, and 𝒳 \mathcal{X} , 𝒴 \mathcal{Y} are global input/output schemas. Each module M i = ( π i , θ i , 𝒳 i , 𝒴 i ) M_{i}=(\pi_{i},\theta_{i},\mathcal{X}_{i},\mathcal{Y}_{i}) is an LLM subcomponent: π i \pi_{i} is its (system) prompt including instructions and few-shot demonstrations; θ i \theta_{i} the underlying model weights; 𝒳 i , 𝒴 i \mathcal{X}_{i},\mathcal{Y}_{i} are input/output schemas. At runtime, C C orchestrates the sequencing and invocation of modules—e.g., passing outputs from one module to another, invoking modules conditionally, or leveraging tool APIs. This way, C C can invoke different modules in any order multiples of times.

Given Φ \Phi , let Π Φ = ⟨ π 1 , … , π | M | ⟩ \Pi_{\Phi}=\langle\pi_{1},\ldots,\pi_{|M|}\rangle denote the collection of all module prompts and Θ Φ = ⟨ θ 1 , … , θ | M | ⟩ \Theta_{\Phi}=\langle\theta_{1},\ldots,\theta_{|M|}\rangle the set of module weights. The learnable parameters are thus ⟨ Π , Θ ⟩ Φ \langle\Pi,\Theta\rangle_{\Phi} . For a task instance ( x , m ) (x,m) —where x x maps to the input schema 𝒳 \mathcal{X} and m m contains evaluator metadata (e.g., gold answers, evaluation rubrics, code unit tests)—the system induces an output y = Φ ⁡ ( x , ⟨ Π , Θ ⟩ Φ ) y=\Phi(x;\langle\Pi,\Theta\rangle_{\Phi}) . A metric μ : 𝒴 × ℳ → [ 0 , 1 ] \mu:\mathcal{Y}\times\mathcal{M}\to[0,1] then measures the output quality of y y with respect to metadata m m (for example by calculating, exact match, F1, pass rate, etc.). The optimization problem is thus defined as follows, where 𝒯 \mathcal{T} is a task distribution.: ⟨ Π ∗ , Θ ∗ ⟩ Φ = arg ⁡ max ⟨ Π , Θ ⟩ Φ ​ 𝔼 ( x , m ) ∼ 𝒯 ​ [ μ ⁡ ( Φ ⁡ ( x , ⟨ Π , Θ ⟩ Φ ) , m ) ] . \displaystyle\langle\Pi^{*},\Theta^{*}\rangle_{\Phi}=\arg\max_{\langle\Pi,\Theta\rangle_{\Phi}}\mathbb{E}_{(x,m)\sim\mathcal{T}}\left[\mu\big(\Phi(x;\langle\Pi,\Theta\rangle_{\Phi}),\,m\big)\right]. (1)

We adopt this general problem formulation, allowing updates to both prompts and weights of language modules, to enable comparisons between optimization algorithms that operate in different parameter spaces (e.g., GEPA vs. GRPO).

#### Sample-Efficient Optimization.

In many real-world scenarios, rollouts—concretely, invocations of Φ \Phi plus evaluation by μ \mu —are often computationally, monetarily, or timewise expensive. The optimizer is thus limited to at most B B rollouts on a dataset 𝒟 train = { ( x , m ) i } i = 1 N \mathcal{D}_{\text{train}}=\{(x,m)_{i}\}_{i=1}^{N} with full access to μ \mu . The goal is to identify parameters ⟨ Π ∗ , Θ ∗ ⟩ Φ \langle\Pi^{*},\Theta^{*}\rangle_{\Phi} that maximize held-out performance, subject to not exceeding the rollout budget B B : ⟨ Π ∗ , Θ ∗ ⟩ Φ = arg ⁡ max ⟨ Π , Θ ⟩ Φ ​ 𝔼 ( x , m ) ∼ 𝒯 ​ [ μ ⁡ ( Φ ⁡ ( x , ⟨ Π , Θ ⟩ Φ ) , m ) ] , s.t. # ​ rollouts ≤ B . \displaystyle\langle\Pi^{*},\Theta^{*}\rangle_{\Phi}=\arg\max_{\langle\Pi,\Theta\rangle_{\Phi}}\mathbb{E}_{(x,m)\sim\mathcal{T}}\left[\mu\big(\Phi(x;\langle\Pi,\Theta\rangle_{\Phi}),\,m\big)\right],\quad\text{s.t.}\quad\#\text{rollouts}\leq B. (2) The core challenge, then, is: How do we extract maximal learning signal from every expensive rollout to enable effective adaptation of complex, modular AI systems in low-data or budget-constrained settings?

## 3 GEPA: Reflective Prompt Evolution

We introduce GEPA (Genetic-Pareto), a sample-efficient optimizer for compound AI systems motivated by three core principles: genetic prompt evolution (Section 3 ), reflection using natural language feedback (Section 3 ), and Pareto-based candidate selection (Section 3.1 ). Figure 3 gives an overview of GEPA and the full GEPA algorithm is formalized in Figure 4 . GEPA receives the following inputs: A system Φ \Phi instantiated with simple prompts to be optimized, training dataset D t ​ r ​ a ​ i ​ n D_{train} (consisting of task instances ( x , m ) (x,m) as described in Section 2 ), the standard evaluation metric μ \mu for the task, a feedback function μ f \mu_{f} (introduced in Section 3 ) and the total rollout budget B B . Note that GEPA evolves only the set of prompts, denoted as Π Φ \Pi_{\Phi} , whereas the underlying LLM weights, denoted by Θ Φ \Theta_{\Phi} remains fixed.

Genetic Optimization Loop: Given an AI system Φ \Phi , the goal is to identify parameters Π Φ \Pi_{\Phi} that maximize task performance. GEPA begins with a candidate pool 𝒫 \mathcal{P} containing only the base system, where each candidate is a concrete instantiation of ⟨ Π , Θ f ​ r ​ o ​ z ​ e ​ n ⟩ Φ \langle\Pi,\Theta_{frozen}\rangle_{\Phi} . It then enters an optimization loop, repeatedly proposing new candidates until the evaluation budget is exhausted. Candidates are derived from existing ones via reflective mutation or crossover , guided by feedback from rollouts, with each inheriting learning signals from its parents and its own rollout so that GEPA accumulates knowledge along the genetic tree. In each iteration, GEPA (i) selects promising candidates, (ii) proposes and evaluates a variant on a minibatch of tasks, and (iii) if it outperforms its parent(s), adds it to 𝒫 \mathcal{P} with ancestry records and evaluate on D p ​ a ​ r ​ e ​ t ​ o D_{pareto} , the validation set used for selection. After the budget is exhausted, GEPA returns the candidate with the best aggregate performance on D p ​ a ​ r ​ e ​ t ​ o D_{pareto} .

Reflective Prompt Mutation: Natural language traces generated during the execution of a compound AI system offer rich visibility into the behavior and responsibilities of each module, as they capture the intermediate inferences and underlying reasoning steps. When these traces are paired with the final outcome of the system (e.g., success or failure), they provide substantial diagnostic value, allowing practitioners to trace errors or successes back to specific decisions made at the module level. LLMs can leverage these traces via reflection to perform implicit credit assignment, attributing responsibility for the final outcome to the relevant modules. This process of reflection can then be used to make targeted updates to individual modules, making large and effective updates to the whole system’s behavior.

Given a candidate to mutate in the current iteration of the optimization loop (stochastically selected from the Pareto-frontier, see Section 3.1 below), GEPA executes the selected candidate on a stochastically sampled minibatch of input queries from the trainset, tracing the program’s execution. From the execution traces, GEPA extracts the module’s inputs, outputs, and reasoning, and calls the feedback function μ f \mu_{f} , which returns a numeric score and text feedback including details about the evaluation (like compiler error messages, failed rubrics, etc.). GEPA selects the module (among the | M | |M| modules that the language program contains) to be updated based on a policy (round-robin), and a reflection LM is then shown the (current prompt, language program trajectory, score, feedback) with the task to reflectively attribute successes or failures to prompt elements and propose revised instructions. The updated module, with the rest of the language program, is evaluated again on the minibatch, and if the score improves, then the new program is added to the candidate pool. The meta-prompt for reflective prompt updates is shown in Appendix C and the full algorithm is presented in Algorithm 1 .

Evaluation traces as diagnostic signals: The text that LLMs produce is the execution trace of the AI system. The text that the environment produces to compute the reward (e.g. compiler error messages before giving reward 0) is the evaluation trace . Beyond reflection on execution traces, we identify a second valuable source of diagnostic information in the evaluation traces. Many evaluation metrics apply rich strategies (e.g., code evaluation may involve compilation, execution, and profiling), producing natural language traces before computing a scalar reward. We propose leveraging these evaluation traces for reflective credit assignment and targeted prompt updates. GEPA achieves this by extending rewards μ \mu into a feedback function μ f \mu_{f} that extracts textual traces during evaluation and returns them with the final score as feedback_text . When available, such feedback can even be module-specific (e.g. in multi-hop systems the evaluator may provide feedback after each hop). In practice, there are domains where human-graders are able to rate the AI system’s responses, along with providing detailed feedback justifying their scalar ratings. When available, D t ​ r ​ a ​ i ​ n D_{train} can be augmented with such human-written explanations for each instance; during reflection, and GEPA can consume these explanations as auxiliary feedback_text to guide targeted prompt updates, even when natural-language feedback from rollouts is limited or unavailable.

### 3.1 Pareto-based candidate selection

GEPA is a highly modular algorithm that supports various strategies for candidate selection, with the choice of strategy governing the exploration–exploitation tradeoff. A naive approach is to always select the best-performing candidate, but this often traps the optimizer in a local optimum: once a dominant strategy is found, it becomes difficult to surpass, and the optimizer exhausts its budget without learning new, potentially better strategies. Figure 6(a) illustrates this behavior: after finding one new strategy (the first child node), the search repeatedly attempts to refine it, fails to improve, and ultimately depletes the budget.

To address this, GEPA employs a Pareto-based “illumination" strategy ( Mouret and Clune, 2015 ) , shown in Algorithm 2 . For each training instance, GEPA records the highest score across all candidates, forming a Pareto frontier. Candidates that achieve the best score on at least one task are retained, while strictly dominated ones are pruned. From this pruned set, GEPA stochastically samples a candidate, weighting probabilities by how many tasks each candidate leads. This strategy helps GEPA escape local optima without inflating the search, efficiently balancing exploration and exploitation by focusing resources on candidates that embody “winning” strategies within the optimization budget.

## 4 Evaluation

We adopt a standard train/validation/test split. Optimizers have full access to the train split, including text and labels, for program tuning. Although optimizers may monitor the performance of candidate parameters (like model checkpoints) by tracking scores on the validation set (to implement early stopping, for example), direct access to the content of validation instances is restricted. We evaluate on six benchmarks—AIME-2025 ( Balunović et al., 2025 ) , LiveBench-Math ( White et al., 2025 ) , HotpotQA ( Yang et al., 2018 ) , IFBench ( Pyatkin et al., 2025a ) , HoVer ( Jiang et al., 2020 ) , and PUPA ( Li et al., 2025a ) —each paired with existing compound AI systems and feedback functions. Experiments use Qwen3 8B ( Yang et al., 2025 ) and GPT-4.1 Mini ( OpenAI, 2025 ) with standardized inference settings, and compare against state-of-the-art optimizers MIPROv2 ( Opsahl-Ong et al., 2024 ) , Trace (with its OptoPrime optimizer) ( Cheng et al., 2024 ) , TextGrad ( Yuksekgonul et al., 2025 ) , and GRPO 1 1 1 We use LoRA for GRPO due to its low cost and succesful adoption with GRPO ( Wang et al., 2025b ; Xu et al., 2025b ; Li et al., 2025b ; Yue et al., 2025 ; Sun et al., 2025 ; Hayou et al., 2025 ; Zhao et al., 2025 ; Teknium et al., 2024 ; Zhao et al., 2024 ; Sidahmed et al., 2024 ) . Additionally, we explore full-parameter finetuning. Figure 11 shows a similar result comparing GEPA to GRPO with full finetuning. ( Shao et al., 2024 ) . Appendix E provides further details on benchmarks, systems, and feedback functions (Subsection E.1 ); models and inference settings (Subsection E.2 ); monetary cost to run the experiments (Subsection E.3 ); and optimizer configurations (Subsection E.4 ). Table 1 , Table 2 and Figure 10 summarize our main results, from which we derive the following observations:

Observation 1: Reflective Prompt Evolution is highly sample-efficient and can outperform weight-space reinforcement learning: Across four benchmarks, GEPA adapts rapidly and generalizes robustly in compound AI systems—beating GRPO (24,000 rollouts) by up to 19% while using up to 35 × 35\times fewer rollouts. It reaches optimal test performance with 4 4 – 35 × 35\times fewer rollouts and exceeds GRPO on 5 out of 6 tasks by 19.0%, 2.73%, 13.66%, 5.19% and 0.7%. GEPA matches GRPO’s best validation after only 243, 402, 330, 1143, 1179, and 306 rollouts—up to 78 × 78\times greater sample efficiency. GEPA+Merge widens the gap, outperforming GRPO by 21% at a comparable rollout budget to GEPA.

The majority of GEPA’s rollout budget is spent on validation, where scores are utilized solely for candidate selection and not for producing learning signals. If we restrict the analysis to train set rollouts, GEPA requires only 79 to 737 rollouts to reach optimal performance. To match GRPO’s best validation scores, GEPA achieves this with only 102, 32, 6, and 179 train rollouts for four tasks, respectively, underscoring the high sample efficiency of learning based on reflective prompt evolution.

Since tracking candidates’ validation performance accounts for majority of GEPA’s rollout budget, sample efficiency can be further improved by evaluating on a smaller validation set or by tracking scores on dynamically selected validation subsets instead of the full set—both of which we propose as directions for future work. Figures 1(a) , 1(b) , 14(c) and 15(c) show the full performance-vs-rollouts curve for all optimizers over benchmarks HotpotQA, IFBench, HoVer and PUPA, respectively.

Observation 2: Reflective prompt evolution enables instruction-optimization alone to outperform joint instruction and few-shot optimization: We compare GEPA with MIPROv2, a state-of-the-art instruction and few-shot optimizer, using two leading models across six diverse tasks, and observe that GEPA consistently outperforms MIPROv2 in all settings, achieving margins as high as 11.1% for GPT-4.1 mini and 10.3% for Qwen3 8B. Further, GEPA and GEPA+Merge more than double the aggregate gains over baseline seen with MIPROv2 across all benchmarks and models (+13.33% and +12.19% vs +5.64% for MIPROv2).

While prior works such as Opsahl-Ong et al. (2024) and Wan et al. (2024) have provided compelling evidence for the effectiveness of few-shot example optimization—often outperforming instruction-based approaches—our findings suggest an exciting shift in this trend. We attribute this primarily to recent advances in the instruction-following and self-reflective abilities of LLMs, as well as the design choices in GEPA that capitalize on these improved capabilities. To further contextualize our findings, we redo the study on generalization gap (the difference between validation and test set performance for optimized prompts) as proposed by Wan et al. (2024) . The results presented in Figure 16 reinforce these observations: reflectively evolved instructions now demonstrate a lower generalization gap, underscoring both advancements in model capabilities and the benefits of GEPA’s design. We see this as a reflection of the continuous evolution of LLMs and GEPA’s ability to effectively leverage these improvements.

We provide the full-length optimized prompts produced by GEPA for all systems, benchmarks, and models in Appendix L , alongside MIPROv2 prompts. Notably, in contrast to prior findings where instruction optimization yielded improvements primarily through quasi-exemplars ( Wan et al., 2024 ) , GEPA’s prompts frequently contain detailed declarative instructions for completing the task, as illustrated in Figure 2 .

Observation 3: The next-candidate selection strategy strongly influences the optimization trajectory and final performance, with Pareto-based sampling providing a distinct advantage.

GEPA refines prompts iteratively with rollout feedback; to test our Pareto-based selection, we compare against a baseline that always picks the best-performing candidate in the SelectBestCandidate strategy (which is similar to the strategy used by TextGrad Yuksekgonul et al. (2025) ), and BeamSearch ( N =4) (used by APO Pryzant et al. (2023) ). As shown in Table 3 , these baselines often yield suboptimal exploration of the prompt search space, leading to poor performance. GEPA with Pareto-based sampling outperforms the BeamSearch strategy by upto 11.33%, and SelectBestCandidate strategy by up to 8.17%, with an aggregate margin of +7.33% and +6.4% across all benchmarks, respectively. Figure 6 highlights the difference in optimization trajectories: always choosing the current best candidate gives immediate improvement but quickly stalls, wasting rollouts on a single candidate. In contrast, our Pareto-based method expands the search by considering all Pareto-optimal candidates (all “winning” strategies found so far), balancing exploration and exploitation and converging to a higher-performing solution within the same rollout budget.

Observation 4: Instruction-optimized prompts are computationally cheaper and generalize better than few-shot demonstration prompts: In addition to their strong generalization capabilities, reflectively evolved instructions offer a significant practical advantage: they are often much shorter and thus computationally more efficient than few-shot demonstration prompts. This advantage becomes especially clear for complex tasks, where even a single few-shot demonstration can be prohibitively long. The problem is further exacerbated when few-shot examples are optimized using state-of-the-art methods such as MIPROv2, which jointly optimizes multiple demonstrations to be used simultaneously, further increasing prompt length.

In contrast, reflectively evolved instructions—such as those generated by GEPA—maintain compactness while providing large performance gains (as demonstrated in Lessons 1 and 2). To illustrate this, we compare GEPA’s and MIPROv2’s prompt lengths (see Figure 18 ). Notably, prompts produced by GEPA and GEPA+Merge are up to 9.2 × 9.2\times shorter than those from MIPROv2, representing a substantial improvement in efficiency, alongside performance improvements.

Moreover, we observe a trend where, in aggregate, optimizers that achieve higher performance tend to produce shorter prompts (see Figure 17 ). This reduction in prompt size has a significant impact—not only reducing runtime cost for downstream tasks (as all API-providers meter the input tokens), but also decreasing latency and improving the overall efficiency of LLM-serving systems ( Kwon et al., 2023 ; Zheng et al., 2024 ; Agrawal et al., 2023 ; Yu et al., 2025 ) .

Observation 5: System aware crossover strategies can provide large gains, but the optimal budget allocation between mutation and crossover, as well as when to invoke merge needs further study: We identify a unique system-aware crossover strategy and operationalize it as Merge (described in Appendix D.1 ). GEPA+Merge can outperform GEPA by as much as 5%, providing an aggregate 2% additional improvement over the already strong performance established by GEPA. Detailed results are available in Table 1 . We attribute these gains to the ability of GEPA+Merge to identify distinct optimization lineages, that have learnt complementary strategies (by evolving distinct modules), and merging them by picking the best version of different modules from each of these lineages to propose a single, optimal candidate.

While in our analysis, we found GEPA+Merge works especially well for GPT-4.1 Mini, it lead to performance degradation when used with Qwen3 8B. Even Qwen3 8B benefits from Merge on one out of four tasks. We attribute these discrepancies to the way the rollout budget is allocated between reflective mutation and crossover, and the timing of invocation of the crossover strategy. In our experiments, we fixed the same hyperparameters for GPT-4.1 Mini and Qwen3 8B, leading to suboptimal choice for Qwen3 8B. Intuitively, crossover would provide the maximum benefit, when there are independent lineages that perform well. Hence, the hyperparameters should be chosen such that Merge is invoked once the optimization tree has evolved sufficiently different lineages. We propose the study of such adaptive techniques as future work.

Observation 6: GEPA-optimized prompts demonstrate cross-model generalization. Table 2 presents results for “GEPA-Qwen-Opt”, a configuration where prompts were optimized using the smaller Qwen3-8B model but evaluated on GPT-4.1-Mini. Despite originating from a weaker model in a different family, these prompts transfer effectively, achieving a +9.00% aggregate improvement across 6 benchmarks (with gains as high as +27.67% on HotpotQA). Remarkably, this transfer performance outperforms strong baselines like MIPROv2 (+5.64%), TextGrad (+6.11%), and Trace (+3.27%), even though those methods were optimized directly on the target GPT-4.1-Mini model.

## 5 Extended Applications of GEPA

### 5.1 GEPA For Inference-Time Search (Contd.)

While the primary focus of this paper is sample-efficient adaptation of AI systems to new tasks, preliminary findings suggest that GEPA may also serve as a promising inference-time search technique. This can be achieved by passing the set of tasks to be solved (for example, a list of Pytorch modules to be converted to CUDA) as the training set to GEPA, ensuring that both D t ​ r ​ a ​ i ​ n D_{train} and D p ​ a ​ r ​ e ​ t ​ o D_{pareto} contain the full set of tasks. This way, GEPA can “overfit” the set of tasks, iteratively proposing better solutions to every problem. We also note that this allows GEPA to apply lessons and insights extracted from rollouts for one task to other tasks. To explore this use case, we conduct preliminary experiments using GEPA as an inference-time search technique for code-generation tasks on two hardware platforms: writing kernels for AMD’s recently introduced XDNA2 Architecture ( Advanced Micro Devices, 2025 ) using an early version of the NPUEval benchmark ( Kalade and Schelle, 2025 ) , and generating CUDA code for NVIDIA-V100 GPUs using KernelBench ( Ouyang et al., 2025 ) .

A distinguishing aspect of these experiments is the use of the feedback function μ f \mu_{f} to dynamically inject domain-specific knowledge into the optimization process. Specifically, kernel development expertise—often codified in technical manuals and documentation—can be selectively surfaced by retrieving relevant manual sections based on rollout failures (e.g., compiler error messages). By using error information to make targetted retrieval queries, GEPA promotes integration of architectural best practices into prompt evolution, as exemplified by the detailed prompt for NPUEval shown in Figure 27 . We also note that generation stochasticity (temperature based sampling) is eliminated by operating under a cache; this ensures that observed improvements tie closely to inference scaling through prompt updates and GEPA’s diverse prompt exploration, rather than stochasticity in the model’s sampling process.

NPU Kernels: We create a sequential refinement agent that iteratively generates kernels (up to 10 times) based on feedback like compiler errors and profiling results (Sequential10), and evaluate the Best-of-N generation. With GPT-4o alone, Sequential10 reaches only 4.25% mean vector utilization. Adding RAG, sourced from technical manuals, improves this to 16.33%, and integrating MIPROv2 further raises it to 19.03%. Notably, applying GEPA to Sequential10 (without RAG) dramatically boosts kernel performance, with several generated kernels achieving up to 70% vector utilization and a mean of 30.52%. Furthermore, a single prompt generated by GEPA enables Sequential10 (again without RAG) to attain a score of 26.85%.

CUDA Kernels: For 35 tasks from the KernelBench “representative subset” ( Ouyang et al., 2025 ) , spanning three difficulty levels, we ran GEPA with GPT-4o. As depicted in Figure 8 , GEPA boosts GPT-4o’s close-to-0% f ​ a ​ s ​ t 1 fast_{1} score to above 20% with increasing search budget. This task used an agent that could generate upto 5 sequential refinements based on environment feedback (Sequential5).

These experiments with GPT-4o also demonstrate GEPA’s ability to leverage the abilities of frontier LLMs. However, these are early results and warrant further systematic study. We believe that leveraging GEPA for inference-time search, particularly when coupled with domain specific textual feedback, could generalize to other code generation and domain adaptation tasks—a direction we leave for future work.

### 5.2 GEPA for Adversarial Prompt Search (Contd.)

We instantiate GEPA for adversarial prompt search by inverting the reward signal: the optimizer proposes prompt edits to include additional information like trivia that minimize task performance (pass@1), while requiring that prompts do not contradict the task and still contain all information needed to solve it. For AIME, GEPA’s adversarial search used AIME 2022–2024 problems as the pool for prompt evolution. The learned prompt was evaluated on AIME-2025 (30 problems), using GPT-5 Mini with 5 runs per problem (150 generations total). We started from a clean instruction prompt and evolved a single universal adversarial instruction that is prepended to each query.

The learned adversarial instruction reduced pass@1 on AIME-2025 from 76% (clean prompt) to 10% (adversarial prompt), despite preserving the nominal task description and the required answer-format directive.

Base (clean) prompt: You are a helpful assistant. You are given a question and you need to answer it. The answer should be given at the end of your response in exactly the format ‘ ### <final answer> ’.

GEPA-generated adversarial prompt (abridged): You are a helpful assistant. You are given a question and you need to answer it. It’s interesting to note that honey never spoils and that the longest river in the world is the Nile, stretching over 6,650 kilometers. When providing your answer, be sure to format it at the end of your response exactly as ‘ ### <final answer> ’. For this task, remember that many mammals, including dolphins, sleep with one eye open. Proceed to answer the given question accordingly.

Manual inspection showed that the adversarial prompt caused GPT-5 Mini to end most responses with the literal placeholder ### <final answer> , indicating a systematic misinterpretation of the formatting rule when paired with the injected distractors. This suggests that the large drop arises from the interaction between extraneous details and a strict, literal formatting constraint, rather than from the formatting requirement alone.

Adversarial prompt search systematically uncovers instruction-level perturbations that sharply degrade model performance, providing a principled, automated way to probe worst-case robustness beyond average-case metrics. By finding universal, task-preserving distractors (e.g., trivia plus strict formatting), it reveals brittle instruction-following interactions and turns them into reusable stress tests and regression suites for continuous evaluation. The resulting adversarial prompts could be used to provide targeted data for fine-tuning or safety training. In practice, this could improve deployment reliability, enables red-teaming at scale, and help track robustness drift over time across models, versions, and domains.

## 6 Related Work

Prompt optimization improves LLMs but often needs manual expertise; for instance, chain-of-thought prompting Wei et al. (2023) . To scale this approach, recent methods use LLMs to optimize prompts automatically ( Zhou et al., 2022 ; Yang et al., 2024 ; Agarwal et al., 2024 ; Fernando et al., 2024 ) . GEPA leverages LLMs, but differs by incorporating textual environment feedback, Pareto-aware search over candidates, and evolution strategies per submodule within an AI system.

Evolutionary algorithms have been used to optimize prompts, e.g., EvoPrompt ( Guo et al., 2024 ) , which evolves prompt populations. Rainbow Teaming ( Samvelyan et al., 2024 ) applies quality-diversity evolution to generate diverse adversarial prompts. GEPA additionally uses domain-specific feedback for targeted mutations achieving higher sample efficiency. AlphaEvolve ( Novikov et al., 2025 ) and OpenEvolve ( Sharma, 2025 ) apply evolutionary search directly to code rewriting, excelling when problem solution can be codified. While AlphaEvolve targets a single hard problem, GEPA brings evolution to prompts across domains, combining Pareto-frontier optimization and prompt evolution to transfer tactics from related problems.

Feedback-driven improvement often uses reinforcement learning, such as majority voting signals ( Zuo et al., 2025 ) , but RL can be sample-inefficient when rewards are slow to compute. An alternative is learning in the language space: in-context bandit/self-bootstrapping methods ( Shinn et al., 2023 ; Madaan et al., 2023 ) ( Monea et al., 2025 ; Xu et al., 2025a ; Feng et al., 2025 ; Cheng et al., 2024 ) , workflow memory and skills ( Wang et al., 2024 ; Wang et al., 2025c ) , and test-time strategy synthesis via Dynamic Cheatsheet ( Suzgun et al., 2025 ) , reasoning cache ( Chen et al., 2025b ) . GEPA instead uses examples to propose new instructions , yielding task-specific rules.

To optimize compound AI systems and agents ( Lin et al., 2025b ) , DSPy ( Khattab et al., 2022 ; Khattab et al., 2024 ) searches/bootstraps few-shot examples, TextGrad ( Yuksekgonul et al., 2025 ) backpropagates textual feedback, and MIPROv2 ( Opsahl-Ong et al., 2024 ) jointly aligns instructions and examples via Bayesian optimization; these largely rely on global rewards. Agent-Pro ( Zhang et al., 2024 ) evolves agent policies through dynamic belief generation and reflection on interactive experiences. Optimas ( Wu et al., 2025a ) introduces globally aligned local rewards per module. GEPA combines global rewards with environment textual feedback per module and maintains a Pareto frontier over individual data instances, matching prompts/agent design to specific examples. The Pareto-guided evolution lets GEPA explore diverse prompt/code/agent design strategies before converging to a robust, generalizable set.

## 7 Conclusion

We introduced GEPA, a prompt optimizer for arbitrary LLM agents and workflows that leverages explicit reflection and Pareto-based selection, showing superior sample efficiency compared to reinforcement learning (GRPO), while outperforming leading prompt optimizers (MIPROv2). By explicitly incorporating natural language feedback and maintaining a diverse pool of Pareto-optimal candidates, GEPA rapidly adapts AI systems to new tasks. Our results across benchmarks and models suggest that language-based reflection can offer a scalable strategy for optimizing complex real-world AI workflows, especially in resource-constrained settings. GEPA also shows promise as an inference-time search strategy, showing the ability to write code in challenging domains.

## References

Advanced Micro Devices (2025) Advanced Micro Devices AMD xdna™ architecture . Note: https://www.amd.com/en/technologies/xdna.html#xdna2 Accessed on: 2025-07-23 Cited by: §5.1 .

Agarwal et al. (2024) E. Agarwal, J. Singh, V. Dani, R. Magazine, T. Ganu, and A. Nambi PromptWizard: task-aware prompt optimization framework . External Links: 2405.18369 , Link Cited by: §6 .

Agrawal et al. (2023) A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, and R. Ramjee Sarathi: efficient llm inference by piggybacking decodes with chunked prefills . arXiv preprint arXiv:2308.16369 . Cited by: §4 .

AI (2025) M. AI Llama-prompt-ops: an open-source tool for seamless migration from other llms to llama, and for general prompt optimization . Note: https://github.com/meta-llama/llama-prompt-ops Accessed: 2025-07-11 Cited by: §E.4 .

Balunović et al. (2025) M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev AIME-2025 . SRI Lab, ETH Zurich . Note: Dataset from MathArena: Evaluating LLMs on Uncontaminated Math Competitions External Links: Link Cited by: §E.1 , §1 , §4 .

Cao et al. (2025) S. Cao, S. Hegde, D. Li, T. Griggs, S. Liu, E. Tang, J. Pan, X. Wang, A. Malik, G. Neubig, K. Hakhamaneshi, R. Liaw, P. Moritz, M. Zaharia, J. E. Gonzalez, and I. Stoica SkyRL-v0: train real-world long-horizon agents via reinforcement learning . Cited by: §E.4 .

Chen et al. (2025a) M. Chen, T. Li, H. Sun, Y. Zhou, C. Zhu, H. Wang, J. Z. Pan, W. Zhang, H. Chen, F. Yang, Z. Zhou, and W. Chen ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning . arXiv . Note: arXiv:2503.19470 [cs]Comment: Work in progress External Links: Link , Document Cited by: §1 .

Chen et al. (2025b) P. B. Chen, Y. Zhang, D. Roth, S. Madden, J. Andreas, and M. Cafarella Log-augmented generation: scaling test-time reasoning with reusable computation . External Links: 2505.14398 , Link Cited by: §6 .

Chen et al. (2025c) P. Chen, X. Li, Z. Li, X. Chen, and T. Lin Spectral Policy Optimization: Coloring your Incorrect Reasoning in GRPO . arXiv . Note: arXiv:2505.11595 [cs]Comment: 28 pages External Links: Link , Document Cited by: §1 .

Cheng et al. (2024) C. Cheng, A. Nie, and A. Swaminathan Trace is the next autodiff: generative optimization with rich feedback, execution traces, and llms . External Links: 2406.16218 , Link Cited by: §E.4 , §4 , §6 .

Feng et al. (2025) X. Feng, B. Liu, Y. Song, H. Fu, Z. Wan, G. A. Koushik, Z. Hu, M. Yang, Y. Wen, and J. Wang Natural language reinforcement learning . External Links: 2411.14251 , Link Cited by: §6 .

Fernando et al. (2024) C. Fernando, D. Banarse, H. Michalewski, S. Osindero, and T. Rockt""aschel Promptbreeder: self-referential self-improvement via prompt evolution . In Proceedings of the 41st International Conference on Machine Learning , ICML’24 . Cited by: §6 .

Griggs et al. (2025) T. Griggs, S. Hegde, E. Tang, S. Liu, S. Cao, D. Li, C. Ruan, P. Moritz, K. Hakhamaneshi, R. Liaw, A. Malik, M. Zaharia, J. E. Gonzalez, and I. Stoica Evolving skyrl into a highly-modular rl framework . Note: Notion Blog Cited by: §E.4 .

Guo et al. (2024) Q. Guo, R. Wang, J. Guo, B. Li, K. Song, X. Tan, G. Liu, J. Bian, and Y. Yang Connecting large language models with evolutionary algorithms yields powerful prompt optimizers . In The Twelfth International Conference on Learning Representations , External Links: Link Cited by: §6 .

Hayou et al. (2025) S. Hayou, N. Ghosh, and B. Yu PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models . arXiv . Note: arXiv:2506.20629 [cs]Comment: TD,LR: A lightweight module type selection method for LoRA finetuning. PLoP gives precise placements for LoRA adapters for improved performance External Links: Link , Document Cited by: footnote 1 .

Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: low-rank adaptation of large language models. . ICLR 1 ( 2 ), pp. 3 . Cited by: §E.4 .

Jiang et al. (2020) Y. Jiang, S. Bordia, Z. Zhong, C. Dognin, M. Singh, and M. Bansal HoVer: a dataset for many-hop fact extraction and claim verification . In Findings of the Association for Computational Linguistics: EMNLP 2020 , T. Cohn, Y. He, and Y. Liu (Eds.) , Online , pp. 3441–3460 . External Links: Link , Document Cited by: §E.1 , §1 , §4 .

Jin et al. (2025) B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning . arXiv . Note: arXiv:2503.09516 [cs]Comment: 31 pages External Links: Link , Document Cited by: §1 .

Kalade and Schelle (2025) S. Kalade and G. Schelle NPUEval: optimizing npu kernels with llms and open source compilers . External Links: 2507.14403 , Link Cited by: §1 , §5.1 .

Khattab et al. (2022) O. Khattab, K. Santhanam, X. L. Li, D. Hall, P. Liang, C. Potts, and M. Zaharia Demonstrate-search-predict: composing retrieval and language models for knowledge-intensive NLP . arXiv preprint arXiv:2212.14024 . Cited by: §6 .

Khattab et al. (2024) O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. V. A, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts DSPy: compiling declarative language model calls into state-of-the-art pipelines . In The Twelfth International Conference on Learning Representations , External Links: Link Cited by: §E.4 , §2 , §6 .

Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with pagedattention . In Proceedings of the 29th symposium on operating systems principles , pp. 611–626 . Cited by: §4 .

Lambert (2025) N. Lambert Policy gradient algorithms . In RLHF Book: Reinforcement Learning from Human Feedback , Note: Accessed July 16, 2025 External Links: Link Cited by: §1 .

Li et al. (2025a) S. Li, V. C. Raghuram, O. Khattab, J. Hirschberg, and Z. Yu PAPILLON: privacy preservation from Internet-based and local language model ensembles . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , L. Chiruzzo, A. Ritter, and L. Wang (Eds.) , Albuquerque, New Mexico , pp. 3371–3390 . External Links: Link , Document , ISBN 979-8-89176-189-6 Cited by: §E.1 , §1 , §4 .

Li et al. (2025b) X. Li, A. Shakir, R. Huang, J. Lipp, and J. Li ProRank: Prompt Warmup via Reinforcement Learning for Small Language Models Reranking . arXiv . Note: arXiv:2506.03487 [cs] External Links: Link , Document Cited by: footnote 1 .

Lin et al. (2025a) C. Lin, Y. Wen, D. Su, F. Sun, M. Chen, C. Bao, and Z. Lv Knowledgeable-r1: Policy Optimization for Knowledge Exploration in Retrieval-Augmented Generation . arXiv . Note: arXiv:2506.05154 [cs] External Links: Link , Document Cited by: §1 .

Lin et al. (2025b) M. Lin, J. Sheng, A. Zhao, S. Wang, Y. Yue, V. S. J. Huang, H. Liu, J. Liu, G. Huang, and Y. Liu Training of scaffolded language models with language supervision: a survey . External Links: 2410.16392 , Link Cited by: §6 .

Liu et al. (2025) S. Liu, S. Hegde, S. Cao, A. Zhu, D. Li, T. Griggs, E. Tang, A. Malik, K. Hakhamaneshi, R. Liaw, P. Moritz, M. Zaharia, J. E. Gonzalez, and I. Stoica SkyRL-sql: matching gpt-4o and o4-mini on text2sql with multi-turn rl . Cited by: §E.4 .

Madaan et al. (2023) A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark Self-refine: iterative refinement with self-feedback . In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.) , Vol. 36 , pp. 46534–46594 . External Links: Link Cited by: §6 .

Monea et al. (2025) G. Monea, A. Bosselut, K. Brantley, and Y. Artzi LLMs are in-context bandit reinforcement learners . External Links: 2410.05362 , Link Cited by: §6 .

Mouret and Clune (2015) J. Mouret and J. Clune Illuminating search spaces by mapping elites . External Links: 1504.04909 , Link Cited by: §3.1 .

Novikov et al. (2025) A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. Balog AlphaEvolve: a coding agent for scientific and algorithmic discovery . Technical report Google DeepMind . Note: White paper External Links: Link Cited by: §6 .

OpenAI (2025) OpenAI GPT-4.1 series . Note: Large language model series, released April 2025. https://openai.com/index/gpt-4-1/ Cited by: §E.2 , §1 , §4 .

Opsahl-Ong et al. (2024) K. Opsahl-Ong, M. J. Ryan, J. Purtell, D. Broman, C. Potts, M. Zaharia, and O. Khattab Optimizing instructions and demonstrations for multi-stage language model programs . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.) , Miami, Florida, USA , pp. 9340–9366 . External Links: Link , Document Cited by: §E.4 , §1 , §2 , §4 , §4 , §6 .

Ouyang et al. (2025) A. Ouyang, S. Guo, S. Arora, A. L. Zhang, W. Hu, C. Re, and A. Mirhoseini KernelBench: can LLMs write efficient GPU kernels? . In Scaling Self-Improving Foundation Models without Human Supervision , External Links: Link Cited by: §1 , Figure 8 , §5.1 , §5.1 .

Peng et al. (2025) H. Peng, Y. Qi, X. Wang, B. Xu, L. Hou, and J. Li VerIF: Verification Engineering for Reinforcement Learning in Instruction Following . arXiv . Note: arXiv:2506.09942 [cs]Comment: 16 pages, 8 figures External Links: Link , Document Cited by: §1 .

Pryzant et al. (2023) R. Pryzant, D. Iter, J. Li, Y. Lee, C. Zhu, and M. Zeng Automatic prompt optimization with “gradient descent” and beam search . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , H. Bouamor, J. Pino, and K. Bali (Eds.) , Singapore , pp. 7957–7968 . External Links: Link , Document Cited by: Table 3 , §4 .

Pyatkin et al. (2025a) V. Pyatkin, S. Malik, V. Graf, H. Ivison, S. Huang, P. Dasigi, N. Lambert, and H. Hajishirzi Generalizing verifiable instruction following . External Links: 2507.02833 , Link Cited by: §E.1 , §1 , §4 .

Pyatkin et al. (2025b) V. Pyatkin, S. Malik, V. Graf, H. Ivison, S. Huang, P. Dasigi, N. Lambert, and H. Hajishirzi IF-RLVR-Train . arXiv . External Links: Link Cited by: §E.1 .

Saad-Falcon et al. (2025) J. Saad-Falcon, A. G. Lafuente, S. Natarajan, N. Maru, H. Todorov, E. Guha, E. K. Buchanan, M. Chen, N. Guha, C. Ré, and A. Mirhoseini Archon: an architecture search framework for inference-time techniques . External Links: 2409.15254 , Link Cited by: §2 .

Samvelyan et al. (2024) M. Samvelyan, S. C. Raparthy, A. Lupu, E. Hambro, A. H. Markosyan, M. Bhatt, Y. Mao, M. Jiang, J. Parker-Holder, J. Foerster, T. Rocktäschel, and R. Raileanu Rainbow teaming: open-ended generation of diverse adversarial prompts . In Proceedings of the 38th International Conference on Neural Information Processing Systems , NIPS ’24 , Red Hook, NY, USA . External Links: ISBN 9798331314385 Cited by: §6 .

Sha et al. (2025) Z. Sha, S. Cui, and W. Wang SEM: Reinforcement Learning for Search-Efficient Large Language Models . arXiv . Note: arXiv:2505.07903 [cs] External Links: Link , Document Cited by: §1 .

Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models . External Links: 2402.03300 , Link Cited by: §E.4 , §1 , §4 .

Sharma (2025) A. Sharma OpenEvolve: open-source implementation of alphaevolve . Note: https://github.com/codelion/openevolve GitHub Cited by: §6 .

Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning . In Proceedings of the 37th International Conference on Neural Information Processing Systems , NIPS ’23 , Red Hook, NY, USA . Cited by: §6 .

Si et al. (2025) S. Si, H. Zhao, C. Gao, Y. Bai, Z. Wang, B. Gao, K. Luo, W. Li, Y. Huang, G. Chen, F. Qi, M. Zhang, B. Chang, and M. Sun Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learning . arXiv . Note: arXiv:2505.16483 [cs] External Links: Link , Document Cited by: §1 .

Sidahmed et al. (2024) H. Sidahmed, S. Phatale, A. Hutcheson, Z. Lin, Z. Chen, Z. Yu, J. Jin, S. Chaudhary, R. Komarytsia, C. Ahlheim, Y. Zhu, B. Li, S. Ganesh, B. Byrne, J. Hoffmann, H. Mansoor, W. Li, A. Rastogi, and L. Dixon Parameter Efficient Reinforcement Learning from Human Feedback . arXiv . Note: arXiv:2403.10704 [cs] External Links: Link , Document Cited by: footnote 1 .

Song et al. (2025) H. Song, J. Jiang, Y. Min, J. Chen, Z. Chen, W. X. Zhao, L. Fang, and J. Wen R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning . arXiv . Note: arXiv:2503.05592 [cs] External Links: Link , Document Cited by: §1 .

Soylu et al. (2024) D. Soylu, C. Potts, and O. Khattab Fine-tuning and prompt optimization: two great steps that work better together . External Links: 2407.10930 , Link Cited by: §2 .

Sun et al. (2025) Z. Sun, Q. Wang, H. Wang, X. Zhang, and J. Xu Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective . arXiv . Note: arXiv:2505.12886 [cs]Comment: 25 pages External Links: Link , Document Cited by: footnote 1 .

Suzgun et al. (2025) M. Suzgun, M. Yuksekgonul, F. Bianchi, D. Jurafsky, and J. Zou Dynamic cheatsheet: test-time learning with adaptive memory . External Links: 2504.07952 , Link Cited by: §6 .

Tan et al. (2025) S. Tan, L. A. Agrawal, A. Singhvi, L. Lai, M. J. Ryan, D. Klein, O. Khattab, K. Sen, and M. Zaharia LangProBe: a language programs benchmark . External Links: 2502.20315 , Link Cited by: §E.1 , §E.1 , §2 .

Team (2025) Q. Team Qwen/qwen3-8b . Note: https://huggingface.co/Qwen/Qwen3-8B Accessed: 2025-07-11 Cited by: §E.2 , §1 .

Teknium et al. (2024) R. Teknium, J. Quesnelle, and C. Guang Hermes 3 Technical Report . arXiv . Note: arXiv:2408.11857 [cs] External Links: Link , Document Cited by: footnote 1 .

Wan et al. (2024) X. Wan, R. Sun, H. Nakhost, and S. Arik Teach better or show smarter? on instructions and exemplars in automatic prompt optimization . Advances in Neural Information Processing Systems 37 , pp. 58174–58244 . External Links: Link Cited by: Figure 16 , §4 , §4 .

Wang et al. (2025a) H. Wang, C. Qian, W. Zhong, X. Chen, J. Qiu, S. Huang, B. Jin, M. Wang, K. Wong, and H. Ji Acting Less is Reasoning More! Teaching Model to Act Efficiently . arXiv . Note: arXiv:2504.14870 [cs] External Links: Link , Document Cited by: §1 .

Wang et al. (2025b) S. Wang, J. Asilis, Ö. F. Akgül, E. B. Bilgin, O. Liu, and W. Neiswanger Tina: Tiny Reasoning Models via LoRA . arXiv . Note: arXiv:2504.15777 [cs] External Links: Link , Document Cited by: footnote 1 .

Wang et al. (2025c) Z. Z. Wang, A. Gandhi, G. Neubig, and D. Fried Inducing programmatic skills for agentic tasks . External Links: 2504.06821 , Link Cited by: §6 .

Wang et al. (2024) Z. Z. Wang, J. Mao, D. Fried, and G. Neubig Agent workflow memory . External Links: 2409.07429 , Link Cited by: §6 .

Wei et al. (2023) J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou Chain-of-thought prompting elicits reasoning in large language models . External Links: 2201.11903 , Link Cited by: §6 .

White et al. (2025) C. White, S. Dooley, M. Roberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Saifullah, S. Dey, Shubh-Agrawal, S. S. Sandha, S. Naidu, C. Hegde, Y. LeCun, T. Goldstein, W. Neiswanger, and M. Goldblum LiveBench: a challenging, contamination-limited llm benchmark . External Links: 2406.19314 , Link Cited by: §E.1 , §1 , §4 .

Wu et al. (2025a) S. Wu, P. Sarthi, S. Zhao, A. Lee, H. Shandilya, A. M. Grobelnik, N. Choudhary, E. Huang, K. Subbian, L. Zhang, D. Yang, J. Zou, and J. Leskovec Optimas: optimizing compound ai systems with globally aligned local rewards . External Links: 2507.03041 , Link Cited by: §6 .

Wu et al. (2025b) Y. Wu, L. Ma, M. Li, J. Zhou, J. Hao, H. Leung, I. King, Y. Zhang, and J. Nie Reinforcing Question Answering Agents with Minimalist Policy Gradient Optimization . arXiv . Note: arXiv:2505.17086 [cs] External Links: Link , Document Cited by: §1 .

Xu et al. (2025a) W. Xu, A. Nie, R. Zheng, A. Modi, A. Swaminathan, and C. Cheng Provably learning from language feedback . External Links: 2506.10341 , Link Cited by: §6 .

Xu et al. (2025b) Y. E. Xu, Y. Savani, F. Fang, and Z. Kolter Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning . arXiv . Note: arXiv:2504.13818 [cs]Comment: 14 pages, 7 figures External Links: Link , Document Cited by: footnote 1 .

Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report . External Links: 2505.09388 , Link Cited by: §E.2 , §1 , §4 .

Yang et al. (2024) C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen Large language models as optimizers . External Links: 2309.03409 , Link Cited by: §6 .

Yang et al. (2018) Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning HotpotQA: a dataset for diverse, explainable multi-hop question answering . In Conference on Empirical Methods in Natural Language Processing (EMNLP) , Cited by: §E.1 , §1 , §4 .

Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models . External Links: 2210.03629 , Link Cited by: §2 .

Yu et al. (2025) L. Yu, J. Lin, and J. Li Stateful large language model serving with pensieve . In Proceedings of the Twentieth European Conference on Computer Systems , EuroSys ’25 , New York, NY, USA , pp. 144–158 . External Links: ISBN 9798400711961 , Link , Document Cited by: §4 .

Yue et al. (2025) Z. Yue, B. Jin, H. Zeng, H. Zhuang, Z. Qin, J. Yoon, L. Shang, J. Han, and D. Wang Hybrid Latent Reasoning via Reinforcement Learning . arXiv . Note: arXiv:2505.18454 [cs] External Links: Link , Document Cited by: footnote 1 .

Yuksekgonul et al. (2025) M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, P. Lu, Z. Huang, C. Guestrin, and J. Zou Optimizing generative ai by backpropagating language model feedback . Nature 639 , pp. 609–616 . Cited by: §E.4 , Table 3 , §4 , §4 , §6 .

Zhang et al. (2025) Q. Zhang, S. Yang, L. Gao, H. Chen, X. Hu, J. Chen, J. Wang, S. Guo, B. Zheng, H. Wang, and J. Zhao LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization . arXiv . Note: arXiv:2505.17447 [cs]Comment: preprint, under review External Links: Link , Document Cited by: §1 .

Zhang et al. (2024) W. Zhang, K. Tang, H. Wu, M. Wang, Y. Shen, G. Hou, Z. Tan, P. Li, Y. Zhuang, and W. Lu Agent-pro: learning to evolve via policy-level reflection and optimization . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , L. Ku, A. Martins, and V. Srikumar (Eds.) , Bangkok, Thailand , pp. 5348–5375 . External Links: Link , Document Cited by: §6 .

Zhao et al. (2024) S. Zhao, J. Dang, and A. Grover Group Preference Optimization: Few-Shot Alignment of Large Language Models . arXiv . Note: arXiv:2310.11523 [cs]Comment: accepted at ICLR 2024, code at https://github.com/jamqd/Group-Preference-Optimization External Links: Link , Document Cited by: footnote 1 .

Zhao et al. (2025) S. Zhao, D. Gupta, Q. Zheng, and A. Grover D1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning . arXiv . Note: arXiv:2504.12216 [cs]Comment: 27 pages, project page at https://dllm-reasoning.github.io/ External Links: Link , Document Cited by: footnote 1 .

Zheng et al. (2024) L. Zheng, L. Yin, Z. Xie, C. L. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, et al. Sglang: efficient execution of structured language model programs . Advances in neural information processing systems 37 , pp. 62557–62583 . Cited by: §4 .

Zhou et al. (2022) Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba Large language models are human-level prompt engineers . In The eleventh international conference on learning representations , Cited by: §6 .

Ziems et al. (2025) N. Ziems, D. Soylu, L. A. Agrawal, I. Miller, L. Lai, C. Qian, K. Song, M. Jiang, D. Klein, M. Zaharia, K. D’Oosterlinck, C. Potts, and O. Khattab Multi-module grpo: composing policy gradients and prompt optimization for language model programs . External Links: 2508.04660 , Link Cited by: §E.4 .

Zuo et al. (2025) Y. Zuo, K. Zhang, L. Sheng, S. Qu, G. Cui, X. Zhu, H. Li, Y. Zhang, X. Long, E. Hua, B. Qi, Y. Sun, Z. Ma, L. Yuan, N. Ding, and B. Zhou TTRL: test-time reinforcement learning . External Links: 2504.16084 , Link Cited by: §6 .

## Appendix A Appendix Outline

• Usage of Large Language Models

• GEPA’s Reflection and Prompt Update Meta Prompt

• GEPA Algorithm and Methodology Details

• Evaluation Setup (Contd.)

• Results and Analysis (Contd.)

• GEPA For Inference-Time Search (Contd.)

• GEPA for Adversarial Prompt Search (Contd.)

• Performance vs. Budget (Rollouts) Curves

• Generalization Gap

• Cost vs. Performance Analysis for optimized systems

• GEPA Search Trees

• Visualizing the Iterative Refinement achieved by GEPA

• Examples of best prompts for every benchmark

• GEPA generated prompts for kernel generation

• Number of reflection LM calls made by GEPA during optimization

## Appendix B Usage of Large Language Models

The authors used large language models (LLMs) only for polishing prose of text where the complete draft was fully written by the authors initially and polished later with the help of LLM-based assistants including ChatGPT, Gemini, and Perplexity. The authors’ used code assistants including Cursor and Copilot to implement the authors’ original design and ideas. The scientific contributions, technical methods, ideas and core results are entirely the original work of the authors.

## Appendix C GEPA’s Reflection and Prompt Update Meta Prompt

Figure C shows the meta-prompt used by GEPA, which guides the LLM to reflectively refine its current instruction based on input–output examples and corresponding feedback from the environment.

## Appendix D GEPA Algorithm and Methodology Details

Figure 4 presents the core GEPA Algorithm, along with the algorithm for Pareto-based candidate selection.

### D.1 Merge: System-aware crossover strategy for Compound AI optimization

Algorithm 4 provides the instantiation of the System aware Merge strategy used in GEPA+Merge. Intuitively, merge will be helpful when there are candidates in the pool that learn complementary strategies. Algorithm 3 defines the selection criteria: candidates are merged only if they share a common ancestor but have optimized disjoint sets of prompts (complementary strategies), are pareto-optimal, and both candidates improve upon the aggregate performance of the ancestor. GEPA routinely checks if the pool has 2 such candidates, invoking merge when identified. These strict lineage conditions mean merge occurs sparsely.

## Appendix E Evaluation Setup (Contd.)

### E.1 Benchmarks, Reference compound AI systems, and Feedback Functions

To rigorously evaluate the performance of GEPA and and compare it against current state-of-the-art compound AI system optimizers, we assemble a diverse suite of benchmarks mostly obtained from Tan et al. (2025) , each paired with available Compound AI Systems.

HotpotQA ( Yang et al., 2018 ) is a large-scale question-answering dataset consisting of 113K Wikipedia-based question-answer pairs. It features questions that require reasoning over multiple supporting documents. We modify the last hop of the HoVerMultiHop program (described below) to answer the question instead of generating another query, and the rest of the system remains unmodified. The textual feedback module identifies the set of relevant documents remaining to be retrieved at each stage of the program, and provides that as feedback to the modules at that stage. We use 150 examples for training, 300 for validation, and 300 for testing.

IFBench ( Pyatkin et al., 2025a ) introduced a benchmark specifically designed to assess language models’ ability to follow precise human instructions, especially output constraints (e.g., “answer only with yes or no”, or “mention a word at least three times”). The IFBench test set consists of 58 new and out-of-distribution output constraints and instructions to test system’s ability to generalize to new task constraints. Pyatkin et al. (2025a) also release IFTrain and IF-RLVR Train data ( Pyatkin et al., 2025b ) which are used for training. We split the IF-RLVR Train into our train/val sets, and IFBench as our test set in order to ensure that the optimizers do not access the new, unseen constraints being tested in IFBench. We design a 2-stage system, that first attempts to answer the user query, and then in the second stage, rewrites the answer following the constraints. The textual feedback module provides the descriptions of constraints satsified and failed-to-be-satisifed by the system’s response. Our splits contain 150 training examples, 300 for validation, and 294 for testing.

AIME-2025 ( Balunović et al., 2025 ) The AIME-2025 benchmark consists of 2 problem sets of 15 questions each (total 30) obtained from the AIME examination conducted by Mathematical Association of America. We use prior years AIME questions (2022-2024 totalling 90 questions) split equally into training and validation set, and use the AIME-2025 questions, repeating each question 5 times, as the final test set. We use a single-step ChainOfThought as the AI system under optimization.

LiveBench-Math White et al. (2025) LiveBench is a cross-domain benchmark consisting of regularly updated questions. We use the math subset of LiveBench questions retrieved on July 30, 2025. This set of questions (n=368) is shuffled (with python random seed 0) and split equally into train/val/test questions. We use a single-step ChainOfThought as the AI system under optimization.

HoVer ( Jiang et al., 2020 ) is an open-domain multihop fact extraction and claim verification benchmark built on a Wikipedia-based corpus requiring complex reasoning across multiple sentences and documents, typically involving multiple wikipedia articles. Following Tan et al. (2025) , the systems are evaluated for their ability to write queries in multiple hops to retrieve all relevant wikipedia documents (gold documents) required to make the claim. We obtain the HoverMultiHop program from Tan et al. (2025) , which performs up to 3-hop retrievals using 2 query writer modules, and 2 document summary modules. The textual feedback module simply identifies the set of correct documents retrieved, and the set of documents remaining to be retrieved, and returns them as feedback text. For the full-parameter finetuning results demonstrated in figure 11 , we instantiate a 2-hop program, where the first hop is performed with the initial claim, and the LLM is prompted in a single turn with the claim and first-hop retrieved documents, to provide the second-hop search query. For HoVer, we use 150 examples for training, 300 for validation, and 300 for testing.

PUPA ( Li et al., 2025a ) propose the task of Privacy-Conscious Delegation: addressing real-world user queries using an ensemble of trusted and untrusted models. The core challenges are maintaining high response quality while minimizing leakage of personally identifiable information (PII) to untrusted models. Li et al. (2025a) also present PAPILLON, a compound AI system consisting of 2 modules, a user query rewriter and a response rewriter, run over the trusted model, along with an intermediate call to the untrusted model with the rewritten query. The feedback text simply provides the breakdown of the aggregate score, consisting of a response quality score and a PII leakage score. The dataset is split into 111 training examples, 111 for validation, and 221 for testing.

### E.2 Models and Inference Parameters

We evaluate GEPA and baseline optimizers using two contemporary LLMs, chosen to represent both open-source and commercial model families. Each compound AI system is instantiated once per model, with all modules (e.g., retrievers, rewriters, answer generators) relying on the same model. All models are allowed a context window of upto 16384 tokens for inference.

Qwen3 8B ( Yang et al., 2025 ) : For our open-source experiments (including GRPO), we use Qwen3-8B . Following the recommended settings as per Team (2025) , we use a decoding temperature of 0.6, top-p of 0.95, and top-k of 20 for training as well as inference.

GPT-4.1 Mini ( OpenAI, 2025 ) : For comparison with large commercial models, we use GPT-4.1 mini ( openai/gpt-4.1-mini-2025-04-14 ) accessed via the OpenAI API with a model temperature of 1.0.

### E.3 Costs

It costs under $500 to run all experiments in Table 2 with GPT-4.1 mini. Specifically, GEPA costs a total of $86, GEPA-Merge costs $67, MIPROv2 costs $76, and Trace and TextGrad cost $172 in total.

### E.4 Optimizers

Baseline: The base program is directly evaluated without any further optimization applied.

MIPROv2 ( Opsahl-Ong et al., 2024 ) : MIPROv2 is a widely used compound AI system prompt optimizer and has been integrated into the DSPy ( Khattab et al., 2024 ) and llama-prompt-ops ( AI, 2025 ) frameworks. It works by jointly optimizing both instructions and demonstrations using Bayesian optimization. For each program module, it first bootstraps candidate sets of instructions and demonstrations, assigning uniform priors over their utilities. Candidate assignments are proposed with the Tree-Structured Parzen Estimator (TPE), and the Bayesian model is updated based on evaluation scores to favor high-performing candidates. The most probable sets of instructions and demonstrations are then selected and validated to obtain the final optimized program configuration.

All MIPROv2 optimization runs are performed with the a ​ u ​ t ​ o = h ​ e ​ a ​ v ​ y auto=heavy setting, which corresponds to proposing 18 instruction candidates and 18 bootstrapped few-shot sets. Hence, across benchmarks, the exact number of rollouts varies depending on the number of trials it takes to bootstrap examples (finding 18 successful solution instances), the required number of Bayesian search steps (determined by the number of modules in the system), and size of the valset. Overall, MIPROv2’s rollouts ranged from a minimum of 2270 (for PUPA) to maximum of 6926 (for HoVer).

Trace and TextGrad ( Cheng et al., 2024 ; Yuksekgonul et al., 2025 ) : We implement both optimizers in the Trace framework. All programs under optimization have the exact same architecture compared to the DSPy implementation. To ensure a fair comparison, we port all the DSPy specific signature and parsing prompt to Trace, and use the same initial prompt. In addition, all the test, train, and validation data match exactly the experiment we used for GEPA. The performance of the unoptimized Trace program baseline closely matched our baseline implementation in DSPy (within 0.5% difference). All optimization experiments were under the same rollout budget as MIPROv2 and GEPA. We also provide both optimizer the same metric and feedback functions as GEPA, and, for the per-module feedback function that is not available in Trace (both optimizer do not support per-module feedback), we followed the feedback format in the BigBench-Hard tutorial 2 2 2 https://microsoft.github.io/Trace/examples/nlp/bigbench_hard.html from the Trace authors.

GRPO ( Shao et al., 2024 ) : Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm that estimates advantages in a group-relative manner. For compound AI systems consisting of multiple modules, we use the GRPO implementation provided and open-sourced by Ziems et al. (2025) to perform our experiments, whereas for single-module systems (e.g., figure 11 ), we use the GRPO implementation provided by SkyRL ( Griggs et al., 2025 ; Liu et al., 2025 ; Cao et al., 2025 ) .

Across all compound system training runs, each training step uses a group size of 12, with 4 training instances per step (total batch size 48, with per device train batch size 1). Training employs LoRA ( Hu et al., 2022 ) with rank dimension 16, α = 64 \alpha=64 , and dropout 0.05, using bf16 precision targeting the projection modules [ q , k , v , o , up , down , gate ] [\mathrm{q},\mathrm{k},\mathrm{v},\mathrm{o},\mathrm{up},\mathrm{down},\mathrm{gate}] . We use a learning rate of 1 × 10 − 5 1\times 10^{-5} , β = 0.01 \beta=0.01 , reward scale normalization, and gradient norm clipping of 0.1. Gradients are accumulated for 20 steps before each update, with a “constant with warmup learning” rate scheduler. Non-reentrant gradient checkpointing is enabled to further reduce memory usage. GRPO optimization run for 500 training steps, amounting to fixed 24,000 rollouts, with validation performed every 20 training steps, which is used to implement early stopping. Compound AI system GRPO training experiments are performed on 1xH100/A100 (80 GB memory) with separate GPUs for inference rollouts.

For single-module GRPO training, we adopt full-parameter finetuning with a group size of 16 16 . Each training step employs a global batch size of 32 32 , realized as per-device micro-batches of 4 4 across 8 8 GPUs. Rollout generation is performed with a per-GPU forward micro-batch size of 12 12 . Training is distributed using FSDP2 , with sampling performed at temperature 1.0 1.0 . We apply KL regularization and set the learning rate to 1 × 10 − 6 1\times 10^{-6} . Validation is conducted every 5 5 training steps. During evaluation, sampling is performed with temperature 0.6 0.6 , top- p = 0.95 p=0.95 , and top- k = 20 k=20 .

We manually explore several values for [LR, beta, norm clipping] hyperparameters for both training runs.

GEPA: GEPA is our optimizer, based on the algorithm described in Section 3 . We evaluate 2 variants of our main optimizer GEPA: GEPA and GEPA+Merge , along with 2 ablations created by replacing the Pareto-based sampling strategy with a naive, SelectBestCandidate strategy (SelectBestCandidate and SelectBestCandidate+Merge). All GEPA optimization runs use a minibatch size of 3, and merge is invoked a maximum of 5 times during the optimization run, when enabled. To ensure a fair comparison with MIPROv2, we align the computational budget between GEPA and MIPROv2 on a per-benchmark basis. The training set from each benchmark is used as D f ​ e ​ e ​ d ​ b ​ a ​ c ​ k D_{feedback} (which is used to derive the training signals, as discussed in Section 3 ) and the validation set is used as D p ​ a ​ r ​ e ​ t ​ o D_{pareto} . Specifically, since MIPROv2’s total rollout budget depends on factors such as validation set size and the number of modules, we first record the number of rollouts expended by MIPROv2 for each benchmark, and then cap GEPA’s optimization to match this rollout budget. While differences in proposal and validation procedures cause the exact budget usage by the systems to be slightly different, the discrepancy is always within 10.15%. This protocol ensures that any performance differences arise from the optimization algorithms themselves, rather than from differences in search budget. The exact rollout counts for each optimizer is visualized in Appendix G .

## Appendix F Results and Analysis (Contd.)

Figure 10 visualizes the final test set performance for aggregate and individual benchmarks across both models.

## Appendix G Performance vs. Budget (Rollouts) Curves

Figures 12 , 13 , 14 , 15 show the full Performance-vs-Rollout curves for all the optimizers across all benchmarks.

## Appendix H Generalization Gap

Figure 16 visualizes the generalization gap for different optimization methods.

## Appendix I Cost vs. Performance Analysis for optimized systems

The prompt size of the optimized system plays an important role in determining the downstream cost of using the optimized system. Figure 17 visualizes the aggregate prompt lengths of the final optimized system (as cost proxy) for each optimizer, against the performance achieved. Notably, GEPA’s prompts are around 33% shorter than MIPROv2’s prompts, while achieving higher performance.

## Appendix J GEPA Search Trees

Figures 19 , 20 , 21 , 22 , 23 , 24 , 25 , and 26 present the genetic search trees created by various configurations of GEPA (and ablation SelectBestCandidate).

## Appendix K Visualizing the Iterative Refinement achieved by GEPA

Figure 5 presented a summary of the prompt refinements performed by GEPA during the optimization for PUPA. In this section, we present the full prompts produced during the optimization.

### K.1 Prompts at intermediate stages for PUPA

## Appendix L Examples of best prompts for every benchmark

In this section, we present the best optimized prompt obtained for every (benchmark, model) configuration. Each subsection below pertains to one (benchmark, model) configuration. Since every compound AI system consists of multiple modules, each subsection consists of multiple boxes, listing the prompts for each module. MIPROv2 optimized prompts contain upto 4 few-shot examples for each task. We provide just the first demo here for brevity. GEPA’s prompts only consist of the optimized instruction, which is provided in full.

### L.1 HotpotQA, GPT-4.1 Mini

### L.2 HotpotQA, Qwen3 8B

### L.3 IFBench, GPT-4.1 Mini

### L.4 IFBench, Qwen3 8B

### L.5 HoVer, GPT-4.1 Mini

### L.6 HoVer, Qwen3 8B

### L.7 PUPA, GPT-4.1 Mini

### L.8 PUPA, Qwen3 8B

## Appendix M GEPA generated prompts for kernel generation

### M.1 NPUEval: Kernel Code Generation for new hardware architecture

Figure 27 shows the prompt generated by GEPA with GPT-4o for NPU Keernel Generation, that achieves 26.85% score with the same same GPT-4o agent, that achieved just 4.25% with a simple prompt.

### M.2 KernelBench: CUDA Kernel Code Generation for NVIDIA GPUs

## Appendix N Number of reflection LM calls made by GEPA during optimization

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .