# Dr. Zero: Self-Evolving Search Agents without Training Data

###### Abstract

As high-quality data becomes increasingly difficult to obtain, self-evolution without curated training data has emerged as a promising paradigm. This approach allows large language models (LLMs) to autonomously generate and solve complex problems, thereby improving their reasoning capabilities. However, multi-turn search agents struggle in this setting due to limited question diversity and the substantial compute required for multi-step reasoning and tool use. In this work, we introduce Dr. Zero, a framework that enables search agents to effectively self-evolve without human-annotated training data, relying solely on an external search engine as their knowledge environment. In particular, we design a self-evolution feedback loop where a proposer generates structurally diverse questions to train a solver initialized from the same base model. As the solver evolves, it incentivizes the proposer to produce increasingly difficult yet solvable tasks, thus establishing an automated curriculum to refine both agents. To enhance training efficiency, we also introduce hop-grouped relative policy optimization (HRPO). This method clusters structurally similar questions to construct group-level baselines, effectively minimizing the sampling overhead in evaluating each query’s individual difficulty and solvability. Consequently, HRPO significantly reduces the compute requirements for proposer training and reward estimation without compromising performance or stability. Extensive experimental results demonstrate that Dr. Zero matches or surpasses fully supervised search agents on several question answering benchmarks, showing that strong agentic search and evidence-grounded reasoning can emerge solely through self-evolution.

## 1 Introduction

Self-evolving language agents have emerged as a promising paradigm to autonomously enhance model performance by iteratively proposing problems, bootstrapping solutions, and learning from prior experiences ( Liu et al., 2025a ; Zhang et al., 2025a ) . However, existing methods typically rely on extensive curated questions (prompts) to drive exploration, maintaining the need for the costly data curation process ( Chen et al., 2024 ; Wang et al., 2025a ) . To overcome this bottleneck, an automated proposer can be employed to generate synthetic training data, thereby making self-evolution possible with minimal or even zero human input ( Zhao et al., 2025 ; Chen et al., 2025 ) . As shown in Figure 1 , Huang et al. (2025a) design a proposer-solver co-evolution framework to iteratively bootstrap questions and rationales, thereby achieving meaningful performance gains without access to any human-annotated datasets.

Nevertheless, self-evolving language agents primarily focus on mathematical or specific reasoning tasks ( Zhao et al., 2025 ; Huang et al., 2025a ; Wang et al., 2025a ) . In these constrained domains, questions are often narrowly defined, allowing agents to achieve considerable gains even with limited or low-diversity training data ( Zuo et al., 2025 ; Wang et al., 2025c ) . Yet for open-domain questions, self-evolving search agents remain substantially under-explored. While some approaches exist, they typically rely on human questions, extensive contexts or ground truth annotations ( Jiang et al., 2025 ; Sun et al., 2025 ; Lu et al., 2025 ) . To address this limitation, we focus on self-evolution without curated QA training data for learning search agents. Distinct from prior work, we eliminate the need for human-written training questions or answer annotations, while using an external search engine and its indexed corpus as the knowledge environment for the agent’s self-evolution.

We analyze existing data-free frameworks ( Huang et al., 2025a ; Chen et al., 2025 ) and identify two primary limitations in proposer training: (1) the trained proposer suffers from limited diversity, exhibiting a bias toward simple, one-hop questions (see Section .2 ); and (2) their proposer objectives do not explicitly adapt query difficulty as the solver’s capabilities advance. As a result, such approaches yield moderate performance gains on trivial one-hop tasks but struggle to match supervised baselines on complex multi-hop queries (see Section 4 ). Moreover, the standard group relative policy optimization (GRPO) ( Shao et al., 2024 ) significantly increases training compute in self-evolution as it requires nested sampling: generating multiple queries and subsequently producing multiple responses for each question. For instance, Chen et al. (2025) propose a unified model that acts as both proposer and solver for simple math and coding tasks. However, its reliance on nested sampling renders this approach computationally prohibitive for search agents and creates bottlenecks during complex reasoning and search interactions.

In this paper, we investigate self-evolving search agents without curated QA training data. We propose D eep R esearch- Zero (Dr. Zero), a framework that leverages external search engines to improve both proposer and solver performance without human-written training questions or answers. Specifically, we introduce a multi-turn tool-use rollout pipeline that enables the trained proposer to improve question generation and produce complex, multi-hop questions. Furthermore, we design a difficulty-guided reward that incentivizes the proposer to utilize the search engine and generate challenging yet verifiable queries. As computing the reward involves sampling multiple reasoning trajectories from the solver, we also propose hop-grouped relative policy optimization (HRPO), which clusters structurally similar questions to provide a more robust baseline for advantage estimation. Thus, Dr. Zero avoids nested sampling by generating one question per seed prompt and estimating its baseline across questions with the same hop structure, rather than sampling multiple question candidates from each seed prompt as in standard GRPO. Combined with further training and inference optimizations, Dr. Zero achieves substantial gains compared to the base LLM. Additionally, through multiple training iterations, the model matches or even outperforms supervised RL baselines, showing that complex reasoning and search capabilities can be induced without human-annotated training questions in the evaluated short-form, verifiable QA setting. We summarize our contributions as follows: 1. We propose D eep R esearch- Zero (Dr. Zero), a unified framework that leverages external search engines to autonomously improve both proposer and solver performance. Our approach integrates a refined multi-turn tool-use pipeline with a difficulty-guided reward to produce complex, multi-hop questions.

2. We introduce hop-grouped relative policy optimization (HRPO), a novel optimization method that clusters structurally similar questions to provide a robust group-level baseline for advantage estimation. This optimization ensures effective training while eliminating the need for expensive nested sampling in self-evolution.

3. We empirically demonstrate the effectiveness of Dr. Zero through extensive experiments, showing that the proposed self-evolution paradigm yields significant performance gains. Without curated QA training data, our search agents match the aggregate performance of a fully supervised search agent at the 3B scale and outperform supervised baselines on several individual benchmarks.

## 2 Related Work

### 2.1 Reinforcement Learning

Reinforcement learning (RL) improves agent performance by learning from previous experience and maximizing cumulative rewards ( Sutton et al., 1998 ) . In the context of LLMs, RL is frequently implemented using policy gradient algorithms ( Sutton et al., 1999 ; Ouyang et al., 2022 ) . For example, actor-critic methods such as PPO employ a learned critic to estimate a value baseline, thereby reducing gradient variance ( Mnih et al., 2016 ; Schulman et al., 2017 ) . A simpler offline alternative is direct preference optimization (DPO) ( Rafailov et al., 2023 ) , which directly optimizes language models on pairwise preference data. Recently, group-based methods like GRPO have been adopted for their ability to construct low-variance baselines from multiple responses ( Ahmadian et al., 2024 ; Shao et al., 2024 ; Hu, 2025 ) . Building upon these algorithmic improvements, LLMs have demonstrated significant potential across diverse model architectures as well as reasoning-intensive tasks ( Yu et al., 2025 ; Huang et al., 2025b ; Zheng et al., 2025a ) .

### 2.2 Search-Augmented LLMs

Search and retrieval augmentation can improve language modeling by integrating external knowledge ( Lewis et al., 2020 ; Guu et al., 2020 ) . Furthermore, iteratively retrieving relevant documents can substantially enhance LLM performance on complex questions ( Yoran et al., 2024 ; Yue et al., 2024 ; Wang et al., 2025b ) . A notable example is IRCoT, where Trivedi et al. (2023) exploit multi-step retrieval to optimize answer accuracy on knowledge-intensive tasks. In agentic LLMs, search engines can be incorporated as optional tools to enable adaptive retrieval and reasoning, leading to enhanced multi-hop reasoning performance ( Jin et al., 2025 ; Zheng et al., 2025b ; Zhang et al., 2025b ) . Nevertheless, existing methods primarily focus on supervised, verifiable settings and rely on extensive human queries and annotations for training. Concurrent with our work, Lu et al. (2025) propose using self-play to improve search-augmented LLMs. However, their method still depends on ground truth labels and numerous human-written examples. In contrast, we use no curated QA training data and rely on external search as the knowledge environment for self-improvement.

### 2.3 Self-Evolving LLMs

Self-evolving LLMs autonomously enhance model capabilities by iteratively generating and learning from their own experiences ( Pang et al., 2024 ; Liu et al., 2025a ) . Early approaches utilize self-play mechanisms where the model acts as both the generator and the evaluator to refine its policy without human annotations ( OpenAI et al., 2021 ; Chen et al., 2024 ; Wu et al., 2024 ) . For instance, self-rewarding LLMs ( Yuan et al., 2024 ) employ iterative training loops where the model judges its own outputs to construct preference data for optimization. To further eliminate the reliance on external prompts, recent works have introduced data-free self-evolution frameworks ( Zhao et al., 2025 ; Chen et al., 2025 ; Liu et al., 2025b ; Xia et al., 2025 ) . Notably, R-Zero ( Huang et al., 2025a ) adopts a proposer-solver design to co-evolve their performance from scratch, effectively creating a self-improving curriculum. Although promising, existing methods are computationally intensive for multi-turn search agents and often underperform on open-domain questions due to the lack of diverse data and tool integration. To bridge this gap, we introduce Dr. Zero, an efficient self-evolution framework without curated QA training data that enables search agents to match or even exceed the performance of strong supervised baselines.

## 3 Methodology

### 3.1 Setup

We employ a proposer-solver self-evolution framework where both models function as search agents capable of leveraging external knowledge. Equipped with the external search engine ℛ \mathcal{R} , the proposer π θ \pi_{\theta} and the solver π ϕ \pi_{\phi} are trained to maximize their respective expected rewards: Proposer : \displaystyle\mathrm{Proposer:} 𝔼 ( x , y ) ∼ π θ ( ⋅ | ℛ ) , { y ^ i } i = 1 n ∼ π ϕ ( ⋅ | x , ℛ ) [ r ( y , { y ^ i } i = 1 n ) ] , \displaystyle\mathbb{E}_{\begin{subarray}{c}(x,y)\sim\pi_{\theta}(\cdot|\mathcal{R}),\{\hat{y}_{i}\}_{i=1}^{n}\end{subarray}\sim\pi_{\phi}(\cdot|x,\mathcal{R})}[r(y,\{\hat{y}_{i}\}_{i=1}^{n})], (1) Solver : \displaystyle\mathrm{Solver:} 𝔼 ( x , y ) ∼ π θ ( ⋅ | ℛ ) , y ^ ∼ π ϕ ( ⋅ | x , ℛ ) ​ [ 𝕀 ⁡ ( y = y ^ ) ] , \displaystyle\mathbb{E}_{\begin{subarray}{c}(x,y)\sim\pi_{\theta}(\cdot|\mathcal{R}),\hat{y}\sim\pi_{\phi}(\cdot|x,\mathcal{R})\end{subarray}}[\mathbb{I}(y=\hat{y})], where r r denotes the proposer reward and 𝕀 \mathbb{I} is the indicator function. In contrast to the solver’s simple outcome-based reward, the proposer reward is defined over the distribution of predicted answers (i.e., { y ^ i } i = 1 n \{\hat{y}_{i}\}_{i=1}^{n} ). If all predictions are correct, the question is considered trivial, whereas if none are correct, the question is likely too difficult for the solver. To enable self-evolution of π θ \pi_{\theta} and π ϕ \pi_{\phi} , we iteratively optimize both components in a symbiotic loop: the proposer learns to synthesize diverse and challenging questions, while the solver enhances its reasoning abilities by learning from these questions. The improved solver performance subsequently encourages the proposer to generate increasingly complex queries, forming a continuously evolving curriculum (see Figure 2 ). We initialize both models from the same base LLM and rely exclusively on the search tool ( ℛ \mathcal{R} ) for external knowledge. To adhere to our setting without curated QA training data, we avoid utilizing any demonstrations, human-written questions or annotated answers in our framework.

### 3.2 Proposer Training

Existing self-evolution methods primarily target specialized domains (e.g., math, coding) to enhance LLM performance without external data ( Zhao et al., 2025 ; Huang et al., 2025a ; Chen et al., 2025 ) . However, for open-domain question answering, we find that such methods tend to generate structurally homogeneous one-hop queries, which limits solver performance gains to simple reasoning questions. Even when equipped with a multi-turn search tool, the solver yields only marginal improvements on multi-hop queries (see Section 4 ). Furthermore, since the proposer reward necessitates multiple solver predictions to assess query difficulty and solvability, optimizing the proposer with a standard RL algorithm like GRPO becomes highly inefficient ( Shao et al., 2024 ) . Generating m m candidate QA pairs from each seed prompt and evaluating each with n n solver predictions requires m + m ​ n = m ⁡ ( n + 1 ) m+mn=m(n+1) rollouts. In our implementation, this corresponds to 4 proposer rollouts and 4 × 4 4\times 4 solver rollouts (20 total). Combined with the high latency of multi-turn rollouts, this scaling bottleneck renders existing approaches impractical for self-evolving agents that require complex tool interactions.

Motivated by these limitations, we propose hop-grouped relative policy optimization (HRPO) to train the proposer model. Instead of sampling multiple responses for a single seed prompt, HRPO calculates advantages by grouping structurally similar questions across prompts. Specifically, we cluster the generated QA pairs by their cross-hop complexity, denoted by the number of hops h ∈ ℋ h\in\mathcal{H} . Questions with fewer hops are typically simpler, whereas higher-hop questions demand extensive search and multi-step reasoning. Here, hop count is used only as a structural grouping variable; empirical question difficulty is determined by the solver pass rate in Equation 4 . This hop-specific normalization of returns produces low-variance advantage estimates while avoiding the computational cost of sampling multiple candidate questions per seed prompt. With five solver predictions, HRPO requires one proposer rollout and five solver rollouts (6 total). HRPO can be formulated with: 𝒥 ⁡ ( θ ) \displaystyle\mathcal{J}(\theta) = 𝔼 { ( x i , y i ) ∼ π θ ( ⋅ | ℛ ) , { y ^ i , k } k = 1 n ∼ π ϕ ( ⋅ | x i , ℛ ) } i = 1 N \displaystyle=\mathbb{E}_{\begin{subarray}{c}\{(x_{i},y_{i})\sim\pi_{\theta}(\cdot|\mathcal{R}),\{\hat{y}_{i,k}\}_{k=1}^{n}\sim\pi_{\phi}(\cdot|x_{i},\mathcal{R})\}_{i=1}^{N}\end{subarray}} (2) [ 1 N ​ ∑ h ∈ ℋ ∑ i ∈ ℐ h log ⁡ π θ ​ ( x i , y i | ℛ ) ​ A i , h ] − β ​ 𝔻 K ​ L , \displaystyle[\frac{1}{N}\sum_{h\in\mathcal{H}}\sum_{i\in\mathcal{I}_{h}}\log\pi_{\theta}(x_{i},y_{i}|\mathcal{R})A_{i,h}]-\beta\mathbb{D}_{KL}, where N N denotes the size of the sampled batch, and β \beta is the hyperparameter controlling the KL regularizer. A i , h A_{i,h} denotes the advantage of the generated QA pair ( x i , y i ) (x_{i},y_{i}) , computed by standardizing the solver’s reward scores over all h h -hop questions: A i , h = r i − 𝔼 j ∈ ℐ h ​ [ r j ] 𝕍 ​ ar j ∈ ℐ h ​ [ r j ] + δ . A_{i,h}=\frac{r_{i}-\mathbb{E}_{j\in\mathcal{I}_{h}}[r_{j}]}{\sqrt{\mathbb{V}\text{ar}_{j\in\mathcal{I}_{h}}[r_{j}]}+\delta}. (3) For optimal proposer performance and training efficiency, we adopt a strictly on-policy framework and omit ratio clipping. While single-response methods like REINFORCE++ reduce sampling costs, we find that a global baseline becomes unstable when processing diverse query structures. This mismatch induces high variance in policy gradients, frequently leading to training failures. In contrast, HRPO mitigates this issue by computing relative advantages among structurally aligned trajectories.

To utilize solver signals for training the proposer π θ \pi_{\theta} , we design a specialized reward function to encourage both verifiability (the task must be solvable) and difficulty (the task must not be trivial). We leverage the solver’s pass rate on the generated questions as a proxy for these properties. Let k k denote the number of correct solutions out of n n sampled attempts; we penalize instances where the solver either fails completely ( k = 0 k=0 ) or succeeds trivially ( k = n k=n ), incentivizing the generation of questions that maximize: r ⁡ ( y , { y ^ i } i = 1 n ) = 𝕀 ⁡ ( 0 < k < n ) ​ n − k n − 1 + r f , with ​ k = ∑ i = 1 n 𝕀 ⁡ ( y = y ^ i ) , r(y,\{\hat{y}_{i}\}_{i=1}^{n})=\mathbb{I}(0<k<n)\frac{n-k}{n-1}+r^{f},\;\text{with}\;k=\sum_{i=1}^{n}\mathbb{I}(y=\hat{y}_{i}), (4) here, the reward is maximized when exactly one solution is correct and decays linearly as the number of correct predictions increases. We additionally impose a format reward r f r^{f} to mitigate structural degradation during complex generation. This enables the proposer to effectively learn to interleave reasoning with search, yielding QA pairs that are both well-formed and challenging. The proposer prompt also requires each hop and final answer to be supported by the seed or retrieved passages (see Figure 5 ). These grounding requirements and the absence of difficulty reward for zero-pass cases reduce, but do not eliminate, incorrect or ambiguous synthetic answers; richer signals such as retrieval uncertainty or answer ambiguity could be incorporated for less-verifiable settings.

### 3.3 Solver Training

For solver training, we sample data pairs ( x , y ) (x,y) from the proposer π θ \pi_{\theta} and optimize π ϕ \pi_{\phi} via group relative policy optimization (GRPO) ( Shao et al., 2024 ) . By computing advantages from the empirical group statistics, GRPO reinforces valid trajectories and refines the model’s search and reasoning capabilities without requiring a separate value function: 𝒥 ⁡ ( ϕ ) \displaystyle\mathcal{J}(\phi) = 𝔼 ( x , y ) ∼ π θ ( ⋅ | ℛ ) , { y ^ i } i = 1 n ∼ π ϕ old ( ⋅ | x , ℛ ) \displaystyle=\mathbb{E}_{\begin{subarray}{c}(x,y)\sim\pi_{\theta}(\cdot|\mathcal{R}),\{\hat{y}_{i}\}_{i=1}^{n}\sim\pi_{\phi_{\mathrm{old}}}(\cdot|x,\mathcal{R})\end{subarray}} (5) [ 1 n ​ ∑ i = 1 n min ⁡ ( π ϕ ​ ( y ^ i | x , ℛ ) π ϕ old ​ ( y ^ i | x , ℛ ) ​ A i , clip ​ ( π ϕ ​ ( y ^ i | x , ℛ ) π ϕ old ​ ( y ^ i | x , ℛ ) , 1 − ϵ , 1 + ϵ ) ​ A i ) ] − β ​ 𝔻 K ​ L , \displaystyle[\frac{1}{n}\sum_{i=1}^{n}\min\left(\frac{\pi_{\phi}(\hat{y}_{i}|x,\mathcal{R})}{\pi_{\phi_{\mathrm{old}}}(\hat{y}_{i}|x,\mathcal{R})}A_{i},\text{clip}(\frac{\pi_{\phi}(\hat{y}_{i}|x,\mathcal{R})}{\pi_{\phi_{\mathrm{old}}}(\hat{y}_{i}|x,\mathcal{R})},1-\epsilon,1+\epsilon)A_{i}\right)]-\beta\mathbb{D}_{KL}, where the advantages are computed via reward standardization (i.e., A i = 𝕀 ⁡ ( y = y ^ i ) − mean ​ ( { 𝕀 ⁡ ( y = y ^ i ) } i = 1 n ) std ​ ( { 𝕀 ⁡ ( y = y ^ i ) } i = 1 n ) + δ A_{i}=\frac{\mathbb{I}(y=\hat{y}_{i})-\texttt{mean}(\{\mathbb{I}(y=\hat{y}_{i})\}_{i=1}^{n})}{\texttt{std}(\{\mathbb{I}(y=\hat{y}_{i})\}_{i=1}^{n})+\delta} ). The optimization is driven by an outcome-based reward that solely evaluates the correctness of final predictions against the synthesized ground truth y y . With increasingly complex queries from the proposer, the solver is motivated to refine its search and reasoning capabilities. Such interactions create a dynamic curriculum that supports solver improvement across diverse problem domains without curated QA training data. The solver’s progress, in turn, encourages the proposer to synthesize more complex cases, establishing a feedback loop that can expand both agents’ capabilities.

### 3.4 The Self-Evolving Dr. Zero

In summary, we introduce Dr. Zero, a scalable and effective framework that leverages self-evolution without curated QA training data to iteratively enhance both the proposer and solver (see Figure 2 ). In each iteration, the proposer π θ \pi_{\theta} synthesizes a batch of QA pairs with heterogeneous hop structures. Utilizing solver feedback, the proposer is optimized via HRPO to produce verifiable, diverse and challenging queries. Meanwhile, the solver leverages the generated data through GRPO to refine its search and reasoning capabilities. This alternating optimization loop creates a symbiotic feedback mechanism: as the solver improves, simple queries yield diminishing rewards, encouraging the proposer to explore more complex reasoning paths to maximize its returns. Conversely, increasingly difficult questions help sustain useful training signals, allowing the solver to continue refining its reasoning skills. Both models are initialized from the same base LLM and evolve without human-written training questions or annotated answers, relying on the external search environment to drive their performance improvements.

## 4 Experiments

### 4.1 Experiment Settings

Datasets & Models. For evaluation, we experiment on multiple open-domain question answering benchmarks, including three one-hop datasets Natural Questions (NQ) ( Kwiatkowski et al., 2019 ) , TriviaQA ( Joshi et al., 2017 ) , PopQA ( Mallen et al., 2022 ) ; and four multi-hop datasets HotpotQA ( Yang et al., 2018 ) , 2WikiMultihopQA (2WikiMQA) ( Ho et al., 2020 ) , MuSiQue ( Trivedi et al., 2022 ) and Bamboogle ( Press et al., 2023 ) . These datasets cover diverse search and reasoning challenges, ensuring a comprehensive evaluation of Dr. Zero across both single-turn and multi-hop scenarios. In our experiments, we use Qwen2.5 3B/7B Instruct as base LLMs for both baseline methods and Dr. Zero.

Baseline & Evaluation. To demonstrate the efficacy of Dr. Zero, we evaluate it against several few-shot and supervised baseline search agents. Few-shot baselines include standard prompting, IRCoT ( Trivedi et al., 2023 ) , Search-o1 ( Li et al., 2025 ) and retrieval augmented generation (RAG) ( Lewis et al., 2020 ) . Supervised baselines consist of supervised fine-tuning (SFT), RL-based fine-tuning without search (R1) ( Guo et al., 2025 ) and the RL-based search agent Search-R1 ( Jin et al., 2025 ) . All models are evaluated using exact match with identical search engine (E5 base) and corpus settings (English Wikipedia dump). Unlike the baselines, Dr. Zero uses neither human-annotated QA training data nor demonstrations; the shared Wikipedia corpus and search engine serve as its external knowledge environment. Further implementation details are provided in Section .1 .

### 4.2 Experiment Results

Overall Performance. We first discuss the main evaluation results as reported in Table 1 . Based on the presented results, we draw several key observations: (1) Overall Performance : Dr. Zero is competitive with the strongest supervised baselines across both single-hop and multi-hop benchmarks, and exceeds them on several individual tasks. These results demonstrate that Dr. Zero effectively extends autonomous self-evolution to search agents without curated QA training data. (2) Superiority over Few-Shot Baselines : Dr. Zero substantially outperforms few-shot methods across nearly every benchmark. For instance, using Qwen2.5-3B on NQ, Dr. Zero achieves 0.397 EM, significantly outperforming few-shot prompting (0.106), IRCoT (0.111) and Search-o1 (0.238). Unlike such baselines, our iterative training loop dynamically refines search capabilities across all tasks. (3) Parity with Supervised Search Agents : Remarkably, Dr. Zero excels without human-annotated QA training data . On single-hop tasks (i.e., NQ, TriviaQA and PopQA), our 3B model outperforms the supervised Search-R1 by 22.9%, 6.5% and 18.4% respectively. In complex multi-hop scenarios, the 7B variant achieves roughly 90% of Search-R1’s performance and even outperforms it on the challenging 2WikiMQA. (4) Scaling and Self-Evolution : As the base model scales from 3B to 7B, Dr. Zero exhibits robust gains, particularly in multi-hop reasoning. By utilizing HRPO and a difficulty-guided reward, our framework generates a high-quality curriculum that supports gains across the initial self-evolution iterations. Overall, our findings demonstrate that Dr. Zero effectively leverages the proposer-solver dynamics to enhance agentic search and evidence-grounded answering across diverse benchmarks; our evaluation does not separately isolate these gains from reasoning over retrieved passages.

Comparison to Data-Free Baselines. We further compare Dr. Zero against existing data-free methods, specifically self-questioning language models (SQLM) and self-evolving reasoning LLMs (R-Zero) ( Chen et al., 2025 ; Huang et al., 2025a ) . To ensure a fair comparison, we augment baselines with multi-turn reasoning and search capabilities, denoting them SQLM* and R-Zero*. Experiment results on the 3B backbone are reported in Table 2 . From these results, we observe the following: (1) Dr. Zero consistently performs the best across all tasks, confirming that our proposer and training pipeline are highly effective for data-free search agents. It exceeds SQLM* and R-Zero* by an average of 39.9% and 27.3% respectively. (2) Unlike the zero-variance filtering used by SQLM*, our pass-rate reward explicitly adapts to the current solver: questions for which all solver attempts are correct or incorrect receive no difficulty reward, while partially solved questions are favored. This design provides a direct signal for producing challenging yet solvable questions. (3) Dr. Zero achieves superior performance and higher efficiency through the proposed HRPO and reward formulation. Specifically, hop-based clustering significantly reduces proposer training costs without compromising performance. Correspondingly, Dr. Zero yields an average relative gain of 83.3% over R-Zero* across the four multi-hop benchmarks. Overall, by integrating a multi-turn reasoning-search framework with difficulty-guided rewards and the proposed HRPO, Dr. Zero demonstrates clear advantages over existing data-free methods.

Training Dynamics. To better understand the self-evolving dynamics of Dr. Zero, we investigate the performance and rewards across iterations, with detailed reward curves and performance metrics summarized in Figure 3 and Table 3 . These results provide several key observations: (1) We observe a steady upward trend in performance and rewards at the start of training. Using exclusively synthetic data from the proposer, both the 3B and 7B solvers rapidly reach a performance peak within approximately 50 steps, suggesting that the most significant gains are realized during this initial phase. (2) Following the first training phase, both the proposer and solver show substantial gains in search and reasoning. The second iteration yields consistent improvements across both solver sizes, with average gains of 4.93% and 1.64% across benchmarks. This confirms the continuous self-evolutionary trend of Dr. Zero. (3) After the second iteration, results vary by model size. The 3B model shows a modest 2.2% increase, but the 7B model drops slightly from 0.372 to 0.360, indicating a performance plateau. Beyond this point, additional iterations for either model size yielded only marginal or no further improvements. (4) We identify several training failure modes, with the most common of which stemming from inconsistent token IDs across multi-turn search and reasoning steps. Interestingly, the 7B model exhibits this trend more frequently (e.g., 7B proposer iteration 3 in Figure 3 ), leading to increased training instability compared to the 3B variant. We also present a comparison with GRPO in Section .2 . In summary, the results highlight the effectiveness of the proposer-solver interplay while identifying the technical constraints that limit indefinite self-evolution. These dynamics validate the design of our Dr. Zero framework, illustrating that autonomous self-evolution successfully replaces supervision for developing multi-turn reasoning and search capabilities.

Response Length and Entropy. Aside from performance, we investigate response entropy and length throughout the training process. Here, we focus on Dr. Zero 3B variant and visualize the average entropy values and response length in Figure 4 , leading to several noteworthy observations: (1) For the solver, both entropy and response length decrease gradually before stabilizing toward the end of the first iteration. This trend indicates that the solver acquires its core search and reasoning capabilities early on, followed by steady, incremental refinements in subsequent iterations. (2) While the proposer follows an initial trend similar to the solver, its entropy and length fluctuate in later iterations. These variations indicate that the proposer maintain generation diversity, continuously exploring different paths to present the solver with increasingly complex challenges. (3) Compared to trajectory lengths and the proposer entropy, the solver’s entropy values are more stable; they decrease drastically and stabilize at a low level. This trend suggests that the model learns quickly and becomes more confident as its reasoning capabilities improve. Together, these trends confirm that the proposer-solver interplay creates a dynamic learning environment: the solver achieves rapid and stable convergence, while the proposer preserves the generation diversity to ensure a broader spectrum of challenging, high-quality questions.

Different Hop Ratios. Finally, we inspect how question composition affects the overall performance of Dr. Zero. Specifically, we vary the distribution of synthetic questions from the proposer, testing ratios of 1/2/3/4-hop questions at 1:1:1:1, 2:1:1:1 and our default 4:3:2:1. These comparative results are detailed in Table 4 . Interestingly, the 3B model does not exhibit performance gains as the proportion of multi-hop questions increases. The optimal multi-hop performance is achieved using the 4:3:2:1 ratio, yielding an average 0.220 EM. This suggests that for smaller base models, strengthening fundamental search capabilities (even via single-hop queries) can more effectively improve performance than focusing exclusively on complex tasks. In contrast, the 7B variant of Dr. Zero achieves its highest overall average with the default 4:3:2:1 composition, while the other compositions remain slightly better on some complex benchmarks. This indicates that performance depends on balancing questions across hop levels rather than simply maximizing the proportion of multi-hop questions. In summary, both model sizes benefit from a mixed curriculum, while the most effective balance varies by model scale and benchmark.

## 5 Conclusion

We introduced Dr. Zero, a self-evolution framework that enhances the reasoning and search capabilities of language agents without curated QA training data. By utilizing an iterative proposer-solver training paradigm, Dr. Zero autonomously generates diverse and increasingly challenging short-form, verifiable open-domain questions without human-written questions or annotated answers. In addition, the proposed HRPO effectively addresses the computational bottlenecks of multi-turn tool use, enabling efficient training by clustering structurally similar queries to estimate advantages. Experimental results demonstrate that Dr. Zero is competitive with supervised search agents and surpasses them on several question answering benchmarks. These findings validate the potential of the self-evolving Dr. Zero as a powerful paradigm for developing advanced search agents in data-scarce environments. Future work will focus on extending the stability of self-evolution to overcome performance plateaus and prevent entropy collapse in larger models. Furthermore, we plan to safeguard the self-evolution process against reward hacking and bias amplification, aiming to develop robust learning frameworks that maintain the integrity and reliability of the feedback loop even in the absence of human supervision.

## References

Ahmadian et al. (2024) Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740 , 2024.

Chen et al. (2025) Lili Chen, Mihir Prabhudesai, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Self-questioning language models. arXiv preprint arXiv:2508.03682 , 2025.

Chen et al. (2024) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335 , 2024.

Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025.

Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning , pages 3929–3938. PMLR, 2020.

Ho et al. (2020) Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics , pages 6609–6625, 2020.

Hu (2025) Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262 , 2025.

Huang et al. (2025a) Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning llm from zero data. arXiv preprint arXiv:2508.05004 , 2025a.

Huang et al. (2025b) Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749 , 2025b.

Jiang et al. (2025) Pengcheng Jiang, Xueqiang Xu, Jiacheng Lin, Jinfeng Xiao, Zifeng Wang, Jimeng Sun, and Jiawei Han. s3: You don’t need that much data to train a search agent via rl. arXiv preprint arXiv:2505.14146 , 2025.

Jin et al. (2025) Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 , 2025.

Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1601–1611, 2017.

Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:453–466, 2019.

Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems , 33:9459–9474, 2020.

Li et al. (2025) Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366 , 2025.

Liu et al. (2025a) Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, et al. Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. arXiv preprint arXiv:2506.24119 , 2025a.

Liu et al. (2025b) Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, and Jason Weston. Spice: Self-play in corpus environments improves reasoning. arXiv preprint arXiv:2510.24684 , 2025b.

Lu et al. (2025) Hongliang Lu, Yuhang Wen, Pengyu Cheng, Ruijin Ding, Haotian Xu, Jiaqi Guo, Chutian Wang, Haonan Chen, Xiaoxi Jiang, and Guanjun Jiang. Search self-play: Pushing the frontier of agent capability without supervision. arXiv preprint arXiv:2510.18821 , 2025.

Mallen et al. (2022) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511 , 7, 2022.

Mnih et al. (2016) Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928–1937. PmLR, 2016.

OpenAI et al. (2021) OpenAI OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D’Sa, Arthur Petron, Henrique P d O Pinto, et al. Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint arXiv:2101.04882 , 2021.

Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730–27744, 2022.

Pang et al. (2024) Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems , 37:116617–116637, 2024.

Press et al. (2023) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 5687–5711, 2023.

Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36:53728–53741, 2023.

Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024.

Sun et al. (2025) Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of llms without searching. arXiv preprint arXiv:2505.04588 , 2025.

Sutton et al. (1998) Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction . MIT press Cambridge, 1998.

Sutton et al. (1999) Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

Trivedi et al. (2022) Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics , 10:539–554, 2022.

Trivedi et al. (2023) Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 10014–10037, 2023.

Wang et al. (2022) Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533 , 2022.

Wang et al. (2025a) Shaobo Wang, Zhengbo Jiao, Zifan Zhang, Yilang Peng, Xu Ze, Boyu Yang, Wei Wang, Hu Wei, and Linfeng Zhang. Socratic-zero: Bootstrapping reasoning via data-free agent co-evolution. arXiv preprint arXiv:2509.24726 , 2025a.

Wang et al. (2025b) Shu Wang, Yixiang Fang, Yingli Zhou, Xilin Liu, and Yuchi Ma. Archrag: Attributed community-based hierarchical retrieval-augmented generation. arXiv preprint arXiv:2502.09891 , 2025b.

Wang et al. (2025c) Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571 , 2025c.

Wu et al. (2024) Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675 , 2024.

Xia et al. (2025) Peng Xia, Kaide Zeng, Jiaqi Liu, Can Qin, Fang Wu, Yiyang Zhou, Caiming Xiong, and Huaxiu Yao. Agent0: Unleashing self-evolving agents from zero data via tool-integrated reasoning. arXiv preprint arXiv:2511.16043 , 2025.

Yang et al. (2024) An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 , 2024.

Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2369–2380, 2018.

Yoran et al. (2024) Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context. In The Twelfth International Conference on Learning Representations , 2024.

Yu et al. (2025) Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 , 2025.

Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning , 2024.

Yue et al. (2024) Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. Inference scaling for long-context retrieval augmented generation. arXiv preprint arXiv:2410.04343 , 2024.

Zhang et al. (2025a) Ding-Chu Zhang, Yida Zhao, Jialong Wu, Liwen Zhang, Baixuan Li, Wenbiao Yin, Yong Jiang, Yu-Feng Li, Kewei Tu, Pengjun Xie, et al. Evolvesearch: An iterative self-evolving search agent. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages 13134–13147, 2025a.

Zhang et al. (2025b) Weizhi Zhang, Yangning Li, Yuanchen Bei, Junyu Luo, Guancheng Wan, Liangwei Yang, Chenxuan Xie, Yuyao Yang, Wei-Chieh Huang, Chunyu Miao, et al. From web search towards agentic deep research: Incentivizing search with reasoning agents. arXiv preprint arXiv:2506.18959 , 2025b.

Zhao et al. (2025) Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.03335 , 2025.

Zheng et al. (2025a) Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071 , 2025a.

Zheng et al. (2025b) Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160 , 2025b.

Zuo et al. (2025) Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084 , 2025.

### .1 Implementation

In our experiments, we implement Dr. Zero as an alternating optimization of the proposer and solver, as illustrated in Figure 1 . We start by training the proposer and utilizing the base model as the generative reward Yang et al. (2024) . Specifically, we rollout one response per prompt and extract the corresponding question and answer. These serve as inputs to the solver to compute the reward defined in Equation 4 . Finally, this reward is employed by HRPO to update the proposer model, as described in Equation 2 . We observe that the reward saturates within approximately 50 steps due to our efficient design. Therefore, we train the proposer for 50 steps, generate QA data on the corresponding prompts, and subsequently utilize this data to train the solver for 50 steps via GRPO Shao et al. (2024) . In our experiments, the proposer is configured with a default generation ratio of 4:3:2:1 for 1-, 2-, 3-, and 4-hop questions respectively. Under this setting, the solver performance typically peaks after 2 to 3 self-evolution iterations, with further training yielding only marginal improvements. Therefore, we limit training to 3 iterations (150 steps per model), which is significantly fewer than baselines like R1 and Search-R1. For further hyperparameters, we conduct a minimal search over the maximum gradient norm and KL divergence coefficient to maintain training stability. The hyperparameters are held constant in our experiments, with full details reported in Table 5 and Table 6 .

For the format reward in HRPO, we define four requirements: (1) adherence to the <think>...</think> structure; (2) valid tool usage, including correct tool call and arguments; (3) an extractable question enclosed in <question>...</question> tags; and (4) an extractable answer within <answer>...</answer> tags. These components are computed individually with a lower bound of 0 and sum to a maximum total of 0.5. This stands in contrast to the difficulty score, which ranges from 0 to 1 (see Equation 4 ). We detail the tool instructions and prompts for the proposer and solver in Figure 5 and Figure 6 , respectively, with qualitative examples available in Section .3 . For the external search engine, we configure it by indexing the corpus and embedding queries using the E5 model Wang et al. (2022) following Yue et al. (2024) ; Jin et al. (2025) . During inference, we perform an approximate nearest neighbor (ANN) search to retrieve the top-3 documents. These passages are subsequently formatted and returned as the tool response. Finally, for both training and evaluation, we utilize exact match to calculate the instance-level score.

### .2 Additional Results

We first provide a detailed quantitative breakdown of the computational efficiency of HRPO compared to standard GRPO. As noted in the main text, standard GRPO relies on asynchronous, nested rollouts, which do not scale efficiently to larger LLMs such as our 7B model. To highlight the efficiency gains, we measured the training time for the 3B model (referenced in Table 9 ) using a single 8xH100 node. HRPO requires significantly fewer rollouts per iteration compared to standard GRPO with nested sampling. This reduction translates directly to a substantial decrease in both wall-clock time and total GPU-hours per iteration, alongside an improvement in average GPU utilization 1 1 1 The solver training remains identical for both methods, completing in approximately 3 hours. .

We provide additional results validating the design choices behind Dr. Zero and HRPO. Specifically, we assess the effectiveness of the proposer within Dr. Zero by ablating key components, such as the format reward and the difficulty-based reward. To accommodate computational constraints, we limit the training of the 3B backbone to a single iteration while selectively removing components from the full framework. We examine the following variations: (1) extending training to 100 steps; (2) removing the format reward; (3) employing a parabolic reward in the proposer (which peaks when solver accuracy is approximately 50%); and (4) omitting the initial document. The ablation results are reported in Table 7 , based on which we observe: (1) increasing the training duration from 50 to 100 steps results in comparable or slightly degraded average performance (0.304 vs. 0.302), suggesting that Dr. Zero converges efficiently and does not benefit from prolonged training in each self-evolution iteration. (2) Removing the format reward results in an average performance drop from 0.304 to 0.289, validating its critical role in guiding the proposer to conduct effective search and reasoning while maintaining structural validity. (3) The default difficulty-based reward outperforms the parabolic design. This indicates that our proposed reward shaping provides more effective signals for generating challenging queries compared to a simple parabolic objective. (4) The most significant performance drop occurs when the initial document is removed (average score falls to 0.245). This indicates that the initial context is crucial for the model to generate diverse synthetic data. Overall, the results demonstrate that each component of Dr. Zero plays a critical role in guiding the proposer to generate structurally valid and challenging queries, validating our integrated design as an effective framework for self-evolution without curated QA training data.

Furthermore, we compare the efficiency of HRPO against the standard GRPO algorithm. For GRPO, we generate 4 questions per prompt and perform 4 rollouts per question, totaling 16 solver rollouts per prompt. In contrast, HRPO generates a single question per prompt with 5 solver predictions, reducing the total rollout count to less than one-third of that required by GRPO. We conduct this comparison using the 3B backbone and present the corresponding results in Table 8 . The results show that HRPO achieves an average score of 0.326, even surpassing the 0.320 baseline of GRPO. Notably, HRPO demonstrates superior performance on one-hop datasets (e.g., NQ), while GRPO retains an advantage on multi-hop benchmarks (e.g., HotpotQA). This suggests that for complex multi-step reasoning tasks, higher computational resources are still essential to ensure accurate baseline estimation and maximize the learning signals from synthetic data. In summary, HRPO achieves higher aggregate performance while utilizing significantly fewer computational resources than GRPO, highlighting its effectiveness for efficient self-evolution.

Finally, we provide additional significance testing of Dr. Zero against supervised baselines. In Table 1 , we follow the verl implementation and adopt greedy decoding for evaluation. Here, we perform additional experiments to provide average results with standard deviations. The results are reported in Table 10 and we mark results in bold if the performance gains are statistically significant . From the results we observe: (1) for the 3B backbone, Dr. Zero shows consistent gains on knowledge-intensive tasks like NQ. This indicates that self-evolution effectively optimizes search policies for direct retrieval, significantly outperforming the strong Search-R1 baseline. (2) On complex multi-hop datasets, the 7B backbone demonstrates more prominent gains, achieving a significant 7.67% relative improvement on 2WikiMQA. This suggests that larger models are more effective for handling complex interleaved search and reasoning. Overall, we find that our gains are consistent, demonstrating that Dr. Zero can indeed match or exceed supervised baselines.

#### Generated Question Analysis.

To directly examine the structural diversity of questions produced by existing data-free baselines, we sample and analyze 500 generated questions from SQLM* and R-Zero* using gemini-3-flash-preview to identify their hop structure. One-hop questions account for 93.4% and 95.6% of their respective outputs, confirming a strong bias toward structurally simple questions. In contrast, Dr. Zero explicitly controls structural coverage through its generation schedule: the default 4:3:2:1 ratio allocates 60% of seed prompts to questions requiring two to four hops. We provide representative Dr. Zero generations at each hop level in Figures 7 , 8 , 9 and 10 . This diagnostic uses diversity specifically to refer to coverage across question hop structures.

### .3 Qualitative Examples

In this section, we present qualitative examples to illustrate the details of the Dr. Zero framework. We begin by providing the full prompts used to guide the proposer and solver in Figure 5 and Figure 6 , respectively. To demonstrate the proposer’s ability to synthesize training data of varying complexity, we showcase generation trajectories across different reasoning depths from Figures 7 , 8 , 9 and 10 . These examples highlight the proposer’s capacity to ground questions in initial documents and extend them to more challenging ones via iterative search and reasoning. Finally, Figures 11 , 12 , 13 and 14 depict the solver’s inference process, verifying its ability to successfully resolve these multi-step queries through structured reasoning and external tool utilization. Note that the last examples for the proposer and solver did not yield satisfactory reasoning or outputs. These instances were specifically selected to illustrate failure modes, such as instruction deviation or truncation due to length constraints. From these qualitative examples, we make the following observations: (1) the proposer effectively scales from single-hop extraction to complex multi-hop synthesis. It demonstrates a sophisticated ability to identify ”bridge entities” that link disparate documents, transforming a simple starting point into a challenging multi-step retrieval task. (2) The solver consistently utilizes the thinking blocks to decompose questions into manageable sub-queries. This internal pattern allows the model to verify intermediate facts before proceeding to the next reasoning step, as seen in Figure 13 . (3) Both models exhibit high proficiency in using the search tool. The solver, in particular, demonstrates ‘adaptive retrieval’, knowing when its internal knowledge is insufficient and formulating targeted search queries to fill such gaps. (4) While the framework is generally robust, the final examples highlight current limitations in long-context generation. Specifically, as the number of hops increases, the models can occasionally deviate from strict formatting constraints or reach maximum token limits, resulting in incomplete trajectories. Overall, these qualitative examples demonstrate that while the Dr. Zero models are capable of sophisticated multi-hop reasoning, enhancing foundational model capabilities and instruction-following robustness remains essential for extending the frontier of self-evolving LLMs toward increasingly high-complexity tasks.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .