Dr. Zero: Self-Evolving Search Agents without Training Data — part 1/3

Cleaned-text code points [0, 14584). Bibliography omitted here; source.md includes it.

# Dr. Zero: Self-Evolving Search Agents without Training Data

###### Abstract

As high-quality data becomes increasingly difficult to obtain, self-evolution without curated training data has emerged as a promising paradigm. This approach allows large language models (LLMs) to autonomously generate and solve complex problems, thereby improving their reasoning capabilities. However, multi-turn search agents struggle in this setting due to limited question diversity and the substantial compute required for multi-step reasoning and tool use. In this work, we introduce Dr. Zero, a framework that enables search agents to effectively self-evolve without human-annotated training data, relying solely on an external search engine as their knowledge environment. In particular, we design a self-evolution feedback loop where a proposer generates structurally diverse questions to train a solver initialized from the same base model. As the solver evolves, it incentivizes the proposer to produce increasingly difficult yet solvable tasks, thus establishing an automated curriculum to refine both agents. To enhance training efficiency, we also introduce hop-grouped relative policy optimization (HRPO). This method clusters structurally similar questions to construct group-level baselines, effectively minimizing the sampling overhead in evaluating each query’s individual difficulty and solvability. Consequently, HRPO significantly reduces the compute requirements for proposer training and reward estimation without compromising performance or stability. Extensive experimental results demonstrate that Dr. Zero matches or surpasses fully supervised search agents on several question answering benchmarks, showing that strong agentic search and evidence-grounded reasoning can emerge solely through self-evolution.

## 1 Introduction

Self-evolving language agents have emerged as a promising paradigm to autonomously enhance model performance by iteratively proposing problems, bootstrapping solutions, and learning from prior experiences ( Liu et al., 2025a ; Zhang et al., 2025a ) . However, existing methods typically rely on extensive curated questions (prompts) to drive exploration, maintaining the need for the costly data curation process ( Chen et al., 2024 ; Wang et al., 2025a ) . To overcome this bottleneck, an automated proposer can be employed to generate synthetic training data, thereby making self-evolution possible with minimal or even zero human input ( Zhao et al., 2025 ; Chen et al., 2025 ) . As shown in Figure 1 , Huang et al. (2025a) design a proposer-solver co-evolution framework to iteratively bootstrap questions and rationales, thereby achieving meaningful performance gains without access to any human-annotated datasets.

Nevertheless, self-evolving language agents primarily focus on mathematical or specific reasoning tasks ( Zhao et al., 2025 ; Huang et al., 2025a ; Wang et al., 2025a ) . In these constrained domains, questions are often narrowly defined, allowing agents to achieve considerable gains even with limited or low-diversity training data ( Zuo et al., 2025 ; Wang et al., 2025c ) . Yet for open-domain questions, self-evolving search agents remain substantially under-explored. While some approaches exist, they typically rely on human questions, extensive contexts or ground truth annotations ( Jiang et al., 2025 ; Sun et al., 2025 ; Lu et al., 2025 ) . To address this limitation, we focus on self-evolution without curated QA training data for learning search agents. Distinct from prior work, we eliminate the need for human-written training questions or answer annotations, while using an external search engine and its indexed corpus as the knowledge environment for the agent’s self-evolution.

We analyze existing data-free frameworks ( Huang et al., 2025a ; Chen et al., 2025 ) and identify two primary limitations in proposer training: (1) the trained proposer suffers from limited diversity, exhibiting a bias toward simple, one-hop questions (see Section .2 ); and (2) their proposer objectives do not explicitly adapt query difficulty as the solver’s capabilities advance. As a result, such approaches yield moderate performance gains on trivial one-hop tasks but struggle to match supervised baselines on complex multi-hop queries (see Section 4 ). Moreover, the standard group relative policy optimization (GRPO) ( Shao et al., 2024 ) significantly increases training compute in self-evolution as it requires nested sampling: generating multiple queries and subsequently producing multiple responses for each question. For instance, Chen et al. (2025) propose a unified model that acts as both proposer and solver for simple math and coding tasks. However, its reliance on nested sampling renders this approach computationally prohibitive for search agents and creates bottlenecks during complex reasoning and search interactions.

In this paper, we investigate self-evolving search agents without curated QA training data. We propose D eep R esearch- Zero (Dr. Zero), a framework that leverages external search engines to improve both proposer and solver performance without human-written training questions or answers. Specifically, we introduce a multi-turn tool-use rollout pipeline that enables the trained proposer to improve question generation and produce complex, multi-hop questions. Furthermore, we design a difficulty-guided reward that incentivizes the proposer to utilize the search engine and generate challenging yet verifiable queries. As computing the reward involves sampling multiple reasoning trajectories from the solver, we also propose hop-grouped relative policy optimization (HRPO), which clusters structurally similar questions to provide a more robust baseline for advantage estimation. Thus, Dr. Zero avoids nested sampling by generating one question per seed prompt and estimating its baseline across questions with the same hop structure, rather than sampling multiple question candidates from each seed prompt as in standard GRPO. Combined with further training and inference optimizations, Dr. Zero achieves substantial gains compared to the base LLM. Additionally, through multiple training iterations, the model matches or even outperforms supervised RL baselines, showing that complex reasoning and search capabilities can be induced without human-annotated training questions in the evaluated short-form, verifiable QA setting. We summarize our contributions as follows: 1. We propose D eep R esearch- Zero (Dr. Zero), a unified framework that leverages external search engines to autonomously improve both proposer and solver performance. Our approach integrates a refined multi-turn tool-use pipeline with a difficulty-guided reward to produce complex, multi-hop questions.

2. We introduce hop-grouped relative policy optimization (HRPO), a novel optimization method that clusters structurally similar questions to provide a robust group-level baseline for advantage estimation. This optimization ensures effective training while eliminating the need for expensive nested sampling in self-evolution.

3. We empirically demonstrate the effectiveness of Dr. Zero through extensive experiments, showing that the proposed self-evolution paradigm yields significant performance gains. Without curated QA training data, our search agents match the aggregate performance of a fully supervised search agent at the 3B scale and outperform supervised baselines on several individual benchmarks.

## 2 Related Work

### 2.1 Reinforcement Learning

Reinforcement learning (RL) improves agent performance by learning from previous experience and maximizing cumulative rewards ( Sutton et al., 1998 ) . In the context of LLMs, RL is frequently implemented using policy gradient algorithms ( Sutton et al., 1999 ; Ouyang et al., 2022 ) . For example, actor-critic methods such as PPO employ a learned critic to estimate a value baseline, thereby reducing gradient variance ( Mnih et al., 2016 ; Schulman et al., 2017 ) . A simpler offline alternative is direct preference optimization (DPO) ( Rafailov et al., 2023 ) , which directly optimizes language models on pairwise preference data. Recently, group-based methods like GRPO have been adopted for their ability to construct low-variance baselines from multiple responses ( Ahmadian et al., 2024 ; Shao et al., 2024 ; Hu, 2025 ) . Building upon these algorithmic improvements, LLMs have demonstrated significant potential across diverse model architectures as well as reasoning-intensive tasks ( Yu et al., 2025 ; Huang et al., 2025b ; Zheng et al., 2025a ) .

### 2.2 Search-Augmented LLMs

Search and retrieval augmentation can improve language modeling by integrating external knowledge ( Lewis et al., 2020 ; Guu et al., 2020 ) . Furthermore, iteratively retrieving relevant documents can substantially enhance LLM performance on complex questions ( Yoran et al., 2024 ; Yue et al., 2024 ; Wang et al., 2025b ) . A notable example is IRCoT, where Trivedi et al. (2023) exploit multi-step retrieval to optimize answer accuracy on knowledge-intensive tasks. In agentic LLMs, search engines can be incorporated as optional tools to enable adaptive retrieval and reasoning, leading to enhanced multi-hop reasoning performance ( Jin et al., 2025 ; Zheng et al., 2025b ; Zhang et al., 2025b ) . Nevertheless, existing methods primarily focus on supervised, verifiable settings and rely on extensive human queries and annotations for training. Concurrent with our work, Lu et al. (2025) propose using self-play to improve search-augmented LLMs. However, their method still depends on ground truth labels and numerous human-written examples. In contrast, we use no curated QA training data and rely on external search as the knowledge environment for self-improvement.

### 2.3 Self-Evolving LLMs

Self-evolving LLMs autonomously enhance model capabilities by iteratively generating and learning from their own experiences ( Pang et al., 2024 ; Liu et al., 2025a ) . Early approaches utilize self-play mechanisms where the model acts as both the generator and the evaluator to refine its policy without human annotations ( OpenAI et al., 2021 ; Chen et al., 2024 ; Wu et al., 2024 ) . For instance, self-rewarding LLMs ( Yuan et al., 2024 ) employ iterative training loops where the model judges its own outputs to construct preference data for optimization. To further eliminate the reliance on external prompts, recent works have introduced data-free self-evolution frameworks ( Zhao et al., 2025 ; Chen et al., 2025 ; Liu et al., 2025b ; Xia et al., 2025 ) . Notably, R-Zero ( Huang et al., 2025a ) adopts a proposer-solver design to co-evolve their performance from scratch, effectively creating a self-improving curriculum. Although promising, existing methods are computationally intensive for multi-turn search agents and often underperform on open-domain questions due to the lack of diverse data and tool integration. To bridge this gap, we introduce Dr. Zero, an efficient self-evolution framework without curated QA training data that enables search agents to match or even exceed the performance of strong supervised baselines.

## 3 Methodology

### 3.1 Setup

We employ a proposer-solver self-evolution framework where both models function as search agents capable of leveraging external knowledge. Equipped with the external search engine ℛ \mathcal{R} , the proposer π θ \pi_{\theta} and the solver π ϕ \pi_{\phi} are trained to maximize their respective expected rewards: Proposer : \displaystyle\mathrm{Proposer:} 𝔼 ( x , y ) ∼ π θ ( ⋅ | ℛ ) , { y ^ i } i = 1 n ∼ π ϕ ( ⋅ | x , ℛ ) [ r ( y , { y ^ i } i = 1 n ) ] , \displaystyle\mathbb{E}_{\begin{subarray}{c}(x,y)\sim\pi_{\theta}(\cdot|\mathcal{R}),\{\hat{y}_{i}\}_{i=1}^{n}\end{subarray}\sim\pi_{\phi}(\cdot|x,\mathcal{R})}[r(y,\{\hat{y}_{i}\}_{i=1}^{n})], (1) Solver : \displaystyle\mathrm{Solver:} 𝔼 ( x , y ) ∼ π θ ( ⋅ | ℛ ) , y ^ ∼ π ϕ ( ⋅ | x , ℛ ) ​ [ 𝕀 ⁡ ( y = y ^ ) ] , \displaystyle\mathbb{E}_{\begin{subarray}{c}(x,y)\sim\pi_{\theta}(\cdot|\mathcal{R}),\hat{y}\sim\pi_{\phi}(\cdot|x,\mathcal{R})\end{subarray}}[\mathbb{I}(y=\hat{y})], where r r denotes the proposer reward and 𝕀 \mathbb{I} is the indicator function. In contrast to the solver’s simple outcome-based reward, the proposer reward is defined over the distribution of predicted answers (i.e., { y ^ i } i = 1 n \{\hat{y}_{i}\}_{i=1}^{n} ). If all predictions are correct, the question is considered trivial, whereas if none are correct, the question is likely too difficult for the solver. To enable self-evolution of π θ \pi_{\theta} and π ϕ \pi_{\phi} , we iteratively optimize both components in a symbiotic loop: the proposer learns to synthesize diverse and challenging questions, while the solver enhances its reasoning abilities by learning from these questions. The improved solver performance subsequently encourages the proposer to generate increasingly complex queries, forming a continuously evolving curriculum (see Figure 2 ). We initialize both models from the same base LLM and rely exclusively on the search tool ( ℛ \mathcal{R} ) for external knowledge. To adhere to our setting without curated QA training data, we avoid utilizing any demonstrations, human-written questions or annotated answers in our framework.

### 3.2 Proposer Training

Existing self-evolution methods primarily target specialized domains (e.g., math, coding) to enhance LLM performance without external data ( Zhao et al., 2025 ; Huang et al., 2025a ; Chen et al., 2025 ) . However, for open-domain question answering, we find that such methods tend to generate structurally homogeneous one-hop queries, which limits solver performance gains to simple reasoning questions. Even when equipped with a multi-turn search tool, the solver yields only marginal improvements on multi-hop queries (see Section 4 ). Furthermore, since the proposer reward necessitates multiple solver predictions to assess query difficulty and solvability, optimizing the proposer with a standard RL algorithm like GRPO becomes highly inefficient ( Shao et al., 2024 ) . Generating m m candidate QA pairs from each seed prompt and evaluating each with n n solver predictions requires m + m ​ n = m ⁡ ( n + 1 ) m+mn=m(n+1) rollouts. In our implementation, this corresponds to 4 proposer rollouts and 4 × 4 4\times 4 solver rollouts (20 total). Combined with the high latency of multi-turn rollouts, this scaling bottleneck renders existing approaches impractical for self-evolving agents that require complex tool interactions.