Dr. Zero: Self-Evolving Search Agents without Training Data — part 3/3

Cleaned-text code points [29147, 33142). Bibliography omitted here; source.md includes it.

Response Length and Entropy. Aside from performance, we investigate response entropy and length throughout the training process. Here, we focus on Dr. Zero 3B variant and visualize the average entropy values and response length in Figure 4 , leading to several noteworthy observations: (1) For the solver, both entropy and response length decrease gradually before stabilizing toward the end of the first iteration. This trend indicates that the solver acquires its core search and reasoning capabilities early on, followed by steady, incremental refinements in subsequent iterations. (2) While the proposer follows an initial trend similar to the solver, its entropy and length fluctuate in later iterations. These variations indicate that the proposer maintain generation diversity, continuously exploring different paths to present the solver with increasingly complex challenges. (3) Compared to trajectory lengths and the proposer entropy, the solver’s entropy values are more stable; they decrease drastically and stabilize at a low level. This trend suggests that the model learns quickly and becomes more confident as its reasoning capabilities improve. Together, these trends confirm that the proposer-solver interplay creates a dynamic learning environment: the solver achieves rapid and stable convergence, while the proposer preserves the generation diversity to ensure a broader spectrum of challenging, high-quality questions.

Different Hop Ratios. Finally, we inspect how question composition affects the overall performance of Dr. Zero. Specifically, we vary the distribution of synthetic questions from the proposer, testing ratios of 1/2/3/4-hop questions at 1:1:1:1, 2:1:1:1 and our default 4:3:2:1. These comparative results are detailed in Table 4 . Interestingly, the 3B model does not exhibit performance gains as the proportion of multi-hop questions increases. The optimal multi-hop performance is achieved using the 4:3:2:1 ratio, yielding an average 0.220 EM. This suggests that for smaller base models, strengthening fundamental search capabilities (even via single-hop queries) can more effectively improve performance than focusing exclusively on complex tasks. In contrast, the 7B variant of Dr. Zero achieves its highest overall average with the default 4:3:2:1 composition, while the other compositions remain slightly better on some complex benchmarks. This indicates that performance depends on balancing questions across hop levels rather than simply maximizing the proportion of multi-hop questions. In summary, both model sizes benefit from a mixed curriculum, while the most effective balance varies by model scale and benchmark.

## 5 Conclusion

We introduced Dr. Zero, a self-evolution framework that enhances the reasoning and search capabilities of language agents without curated QA training data. By utilizing an iterative proposer-solver training paradigm, Dr. Zero autonomously generates diverse and increasingly challenging short-form, verifiable open-domain questions without human-written questions or annotated answers. In addition, the proposed HRPO effectively addresses the computational bottlenecks of multi-turn tool use, enabling efficient training by clustering structurally similar queries to estimate advantages. Experimental results demonstrate that Dr. Zero is competitive with supervised search agents and surpasses them on several question answering benchmarks. These findings validate the potential of the self-evolving Dr. Zero as a powerful paradigm for developing advanced search agents in data-scarce environments. Future work will focus on extending the stability of self-evolution to overcome performance plateaus and prevent entropy collapse in larger models. Furthermore, we plan to safeguard the self-evolution process against reward hacking and bias amplification, aiming to develop robust learning frameworks that maintain the integrity and reliability of the feedback loop even in the absence of human supervision.