Dr. Zero: Self-Evolving Search Agents without Training Data β€” part 2/3

Cleaned-text code points [14584, 29147). Bibliography omitted here; source.md includes it.

Motivated by these limitations, we propose hop-grouped relative policy optimization (HRPO) to train the proposer model. Instead of sampling multiple responses for a single seed prompt, HRPO calculates advantages by grouping structurally similar questions across prompts. Specifically, we cluster the generated QA pairs by their cross-hop complexity, denoted by the number of hops h ∈ β„‹ h\in\mathcal{H} . Questions with fewer hops are typically simpler, whereas higher-hop questions demand extensive search and multi-step reasoning. Here, hop count is used only as a structural grouping variable; empirical question difficulty is determined by the solver pass rate in Equation 4 . This hop-specific normalization of returns produces low-variance advantage estimates while avoiding the computational cost of sampling multiple candidate questions per seed prompt. With five solver predictions, HRPO requires one proposer rollout and five solver rollouts (6 total). HRPO can be formulated with: π’₯ ⁑ ( ΞΈ ) \displaystyle\mathcal{J}(\theta) = 𝔼 { ( x i , y i ) ∼ Ο€ ΞΈ ( β‹… | β„› ) , { y ^ i , k } k = 1 n ∼ Ο€ Ο• ( β‹… | x i , β„› ) } i = 1 N \displaystyle=\mathbb{E}_{\begin{subarray}{c}\{(x_{i},y_{i})\sim\pi_{\theta}(\cdot|\mathcal{R}),\{\hat{y}_{i,k}\}_{k=1}^{n}\sim\pi_{\phi}(\cdot|x_{i},\mathcal{R})\}_{i=1}^{N}\end{subarray}} (2) [ 1 N ​ βˆ‘ h ∈ β„‹ βˆ‘ i ∈ ℐ h log ⁑ Ο€ ΞΈ ​ ( x i , y i | β„› ) ​ A i , h ] βˆ’ Ξ² ​ 𝔻 K ​ L , \displaystyle[\frac{1}{N}\sum_{h\in\mathcal{H}}\sum_{i\in\mathcal{I}_{h}}\log\pi_{\theta}(x_{i},y_{i}|\mathcal{R})A_{i,h}]-\beta\mathbb{D}_{KL}, where N N denotes the size of the sampled batch, and Ξ² \beta is the hyperparameter controlling the KL regularizer. A i , h A_{i,h} denotes the advantage of the generated QA pair ( x i , y i ) (x_{i},y_{i}) , computed by standardizing the solver’s reward scores over all h h -hop questions: A i , h = r i βˆ’ 𝔼 j ∈ ℐ h ​ [ r j ] 𝕍 ​ ar j ∈ ℐ h ​ [ r j ] + Ξ΄ . A_{i,h}=\frac{r_{i}-\mathbb{E}_{j\in\mathcal{I}_{h}}[r_{j}]}{\sqrt{\mathbb{V}\text{ar}_{j\in\mathcal{I}_{h}}[r_{j}]}+\delta}. (3) For optimal proposer performance and training efficiency, we adopt a strictly on-policy framework and omit ratio clipping. While single-response methods like REINFORCE++ reduce sampling costs, we find that a global baseline becomes unstable when processing diverse query structures. This mismatch induces high variance in policy gradients, frequently leading to training failures. In contrast, HRPO mitigates this issue by computing relative advantages among structurally aligned trajectories.

To utilize solver signals for training the proposer Ο€ ΞΈ \pi_{\theta} , we design a specialized reward function to encourage both verifiability (the task must be solvable) and difficulty (the task must not be trivial). We leverage the solver’s pass rate on the generated questions as a proxy for these properties. Let k k denote the number of correct solutions out of n n sampled attempts; we penalize instances where the solver either fails completely ( k = 0 k=0 ) or succeeds trivially ( k = n k=n ), incentivizing the generation of questions that maximize: r ⁑ ( y , { y ^ i } i = 1 n ) = 𝕀 ⁑ ( 0 < k < n ) ​ n βˆ’ k n βˆ’ 1 + r f , with ​ k = βˆ‘ i = 1 n 𝕀 ⁑ ( y = y ^ i ) , r(y,\{\hat{y}_{i}\}_{i=1}^{n})=\mathbb{I}(0<k<n)\frac{n-k}{n-1}+r^{f},\;\text{with}\;k=\sum_{i=1}^{n}\mathbb{I}(y=\hat{y}_{i}), (4) here, the reward is maximized when exactly one solution is correct and decays linearly as the number of correct predictions increases. We additionally impose a format reward r f r^{f} to mitigate structural degradation during complex generation. This enables the proposer to effectively learn to interleave reasoning with search, yielding QA pairs that are both well-formed and challenging. The proposer prompt also requires each hop and final answer to be supported by the seed or retrieved passages (see Figure 5 ). These grounding requirements and the absence of difficulty reward for zero-pass cases reduce, but do not eliminate, incorrect or ambiguous synthetic answers; richer signals such as retrieval uncertainty or answer ambiguity could be incorporated for less-verifiable settings.

### 3.3 Solver Training

For solver training, we sample data pairs ( x , y ) (x,y) from the proposer Ο€ ΞΈ \pi_{\theta} and optimize Ο€ Ο• \pi_{\phi} via group relative policy optimization (GRPO) ( Shao et al., 2024 ) . By computing advantages from the empirical group statistics, GRPO reinforces valid trajectories and refines the model’s search and reasoning capabilities without requiring a separate value function: π’₯ ⁑ ( Ο• ) \displaystyle\mathcal{J}(\phi) = 𝔼 ( x , y ) ∼ Ο€ ΞΈ ( β‹… | β„› ) , { y ^ i } i = 1 n ∼ Ο€ Ο• old ( β‹… | x , β„› ) \displaystyle=\mathbb{E}_{\begin{subarray}{c}(x,y)\sim\pi_{\theta}(\cdot|\mathcal{R}),\{\hat{y}_{i}\}_{i=1}^{n}\sim\pi_{\phi_{\mathrm{old}}}(\cdot|x,\mathcal{R})\end{subarray}} (5) [ 1 n ​ βˆ‘ i = 1 n min ⁑ ( Ο€ Ο• ​ ( y ^ i | x , β„› ) Ο€ Ο• old ​ ( y ^ i | x , β„› ) ​ A i , clip ​ ( Ο€ Ο• ​ ( y ^ i | x , β„› ) Ο€ Ο• old ​ ( y ^ i | x , β„› ) , 1 βˆ’ Ο΅ , 1 + Ο΅ ) ​ A i ) ] βˆ’ Ξ² ​ 𝔻 K ​ L , \displaystyle[\frac{1}{n}\sum_{i=1}^{n}\min\left(\frac{\pi_{\phi}(\hat{y}_{i}|x,\mathcal{R})}{\pi_{\phi_{\mathrm{old}}}(\hat{y}_{i}|x,\mathcal{R})}A_{i},\text{clip}(\frac{\pi_{\phi}(\hat{y}_{i}|x,\mathcal{R})}{\pi_{\phi_{\mathrm{old}}}(\hat{y}_{i}|x,\mathcal{R})},1-\epsilon,1+\epsilon)A_{i}\right)]-\beta\mathbb{D}_{KL}, where the advantages are computed via reward standardization (i.e., A i = 𝕀 ⁑ ( y = y ^ i ) βˆ’ mean ​ ( { 𝕀 ⁑ ( y = y ^ i ) } i = 1 n ) std ​ ( { 𝕀 ⁑ ( y = y ^ i ) } i = 1 n ) + Ξ΄ A_{i}=\frac{\mathbb{I}(y=\hat{y}_{i})-\texttt{mean}(\{\mathbb{I}(y=\hat{y}_{i})\}_{i=1}^{n})}{\texttt{std}(\{\mathbb{I}(y=\hat{y}_{i})\}_{i=1}^{n})+\delta} ). The optimization is driven by an outcome-based reward that solely evaluates the correctness of final predictions against the synthesized ground truth y y . With increasingly complex queries from the proposer, the solver is motivated to refine its search and reasoning capabilities. Such interactions create a dynamic curriculum that supports solver improvement across diverse problem domains without curated QA training data. The solver’s progress, in turn, encourages the proposer to synthesize more complex cases, establishing a feedback loop that can expand both agents’ capabilities.

### 3.4 The Self-Evolving Dr. Zero

In summary, we introduce Dr. Zero, a scalable and effective framework that leverages self-evolution without curated QA training data to iteratively enhance both the proposer and solver (see Figure 2 ). In each iteration, the proposer Ο€ ΞΈ \pi_{\theta} synthesizes a batch of QA pairs with heterogeneous hop structures. Utilizing solver feedback, the proposer is optimized via HRPO to produce verifiable, diverse and challenging queries. Meanwhile, the solver leverages the generated data through GRPO to refine its search and reasoning capabilities. This alternating optimization loop creates a symbiotic feedback mechanism: as the solver improves, simple queries yield diminishing rewards, encouraging the proposer to explore more complex reasoning paths to maximize its returns. Conversely, increasingly difficult questions help sustain useful training signals, allowing the solver to continue refining its reasoning skills. Both models are initialized from the same base LLM and evolve without human-written training questions or annotated answers, relying on the external search environment to drive their performance improvements.

## 4 Experiments

### 4.1 Experiment Settings

Datasets & Models. For evaluation, we experiment on multiple open-domain question answering benchmarks, including three one-hop datasets Natural Questions (NQ) ( Kwiatkowski et al., 2019 ) , TriviaQA ( Joshi et al., 2017 ) , PopQA ( Mallen et al., 2022 ) ; and four multi-hop datasets HotpotQA ( Yang et al., 2018 ) , 2WikiMultihopQA (2WikiMQA) ( Ho et al., 2020 ) , MuSiQue ( Trivedi et al., 2022 ) and Bamboogle ( Press et al., 2023 ) . These datasets cover diverse search and reasoning challenges, ensuring a comprehensive evaluation of Dr. Zero across both single-turn and multi-hop scenarios. In our experiments, we use Qwen2.5 3B/7B Instruct as base LLMs for both baseline methods and Dr. Zero.

Baseline & Evaluation. To demonstrate the efficacy of Dr. Zero, we evaluate it against several few-shot and supervised baseline search agents. Few-shot baselines include standard prompting, IRCoT ( Trivedi et al., 2023 ) , Search-o1 ( Li et al., 2025 ) and retrieval augmented generation (RAG) ( Lewis et al., 2020 ) . Supervised baselines consist of supervised fine-tuning (SFT), RL-based fine-tuning without search (R1) ( Guo et al., 2025 ) and the RL-based search agent Search-R1 ( Jin et al., 2025 ) . All models are evaluated using exact match with identical search engine (E5 base) and corpus settings (English Wikipedia dump). Unlike the baselines, Dr. Zero uses neither human-annotated QA training data nor demonstrations; the shared Wikipedia corpus and search engine serve as its external knowledge environment. Further implementation details are provided in Section .1 .

### 4.2 Experiment Results

Overall Performance. We first discuss the main evaluation results as reported in Table 1 . Based on the presented results, we draw several key observations: (1) Overall Performance : Dr. Zero is competitive with the strongest supervised baselines across both single-hop and multi-hop benchmarks, and exceeds them on several individual tasks. These results demonstrate that Dr. Zero effectively extends autonomous self-evolution to search agents without curated QA training data. (2) Superiority over Few-Shot Baselines : Dr. Zero substantially outperforms few-shot methods across nearly every benchmark. For instance, using Qwen2.5-3B on NQ, Dr. Zero achieves 0.397 EM, significantly outperforming few-shot prompting (0.106), IRCoT (0.111) and Search-o1 (0.238). Unlike such baselines, our iterative training loop dynamically refines search capabilities across all tasks. (3) Parity with Supervised Search Agents : Remarkably, Dr. Zero excels without human-annotated QA training data . On single-hop tasks (i.e., NQ, TriviaQA and PopQA), our 3B model outperforms the supervised Search-R1 by 22.9%, 6.5% and 18.4% respectively. In complex multi-hop scenarios, the 7B variant achieves roughly 90% of Search-R1’s performance and even outperforms it on the challenging 2WikiMQA. (4) Scaling and Self-Evolution : As the base model scales from 3B to 7B, Dr. Zero exhibits robust gains, particularly in multi-hop reasoning. By utilizing HRPO and a difficulty-guided reward, our framework generates a high-quality curriculum that supports gains across the initial self-evolution iterations. Overall, our findings demonstrate that Dr. Zero effectively leverages the proposer-solver dynamics to enhance agentic search and evidence-grounded answering across diverse benchmarks; our evaluation does not separately isolate these gains from reasoning over retrieved passages.

Comparison to Data-Free Baselines. We further compare Dr. Zero against existing data-free methods, specifically self-questioning language models (SQLM) and self-evolving reasoning LLMs (R-Zero) ( Chen et al., 2025 ; Huang et al., 2025a ) . To ensure a fair comparison, we augment baselines with multi-turn reasoning and search capabilities, denoting them SQLM* and R-Zero*. Experiment results on the 3B backbone are reported in Table 2 . From these results, we observe the following: (1) Dr. Zero consistently performs the best across all tasks, confirming that our proposer and training pipeline are highly effective for data-free search agents. It exceeds SQLM* and R-Zero* by an average of 39.9% and 27.3% respectively. (2) Unlike the zero-variance filtering used by SQLM*, our pass-rate reward explicitly adapts to the current solver: questions for which all solver attempts are correct or incorrect receive no difficulty reward, while partially solved questions are favored. This design provides a direct signal for producing challenging yet solvable questions. (3) Dr. Zero achieves superior performance and higher efficiency through the proposed HRPO and reward formulation. Specifically, hop-based clustering significantly reduces proposer training costs without compromising performance. Correspondingly, Dr. Zero yields an average relative gain of 83.3% over R-Zero* across the four multi-hop benchmarks. Overall, by integrating a multi-turn reasoning-search framework with difficulty-guided rewards and the proposed HRPO, Dr. Zero demonstrates clear advantages over existing data-free methods.

Training Dynamics. To better understand the self-evolving dynamics of Dr. Zero, we investigate the performance and rewards across iterations, with detailed reward curves and performance metrics summarized in Figure 3 and Table 3 . These results provide several key observations: (1) We observe a steady upward trend in performance and rewards at the start of training. Using exclusively synthetic data from the proposer, both the 3B and 7B solvers rapidly reach a performance peak within approximately 50 steps, suggesting that the most significant gains are realized during this initial phase. (2) Following the first training phase, both the proposer and solver show substantial gains in search and reasoning. The second iteration yields consistent improvements across both solver sizes, with average gains of 4.93% and 1.64% across benchmarks. This confirms the continuous self-evolutionary trend of Dr. Zero. (3) After the second iteration, results vary by model size. The 3B model shows a modest 2.2% increase, but the 7B model drops slightly from 0.372 to 0.360, indicating a performance plateau. Beyond this point, additional iterations for either model size yielded only marginal or no further improvements. (4) We identify several training failure modes, with the most common of which stemming from inconsistent token IDs across multi-turn search and reasoning steps. Interestingly, the 7B model exhibits this trend more frequently (e.g., 7B proposer iteration 3 in Figure 3 ), leading to increased training instability compared to the 3B variant. We also present a comparison with GRPO in Section .2 . In summary, the results highlight the effectiveness of the proposer-solver interplay while identifying the technical constraints that limit indefinite self-evolution. These dynamics validate the design of our Dr. Zero framework, illustrating that autonomous self-evolution successfully replaces supervision for developing multi-turn reasoning and search capabilities.