Reinforcing General Reasoning without Verifiers β part 4/4
Cleaned-text code points [46578, 55849). Bibliography omitted here; source.md includes it.
The law of total variance states that for a random variable W W and conditioning variables S S , Var β ( W ) = πΌ S β [ Var β [ W | S ] ] + Var S β [ πΌ β‘ [ W | S ] ] \text{Var}(W)=\mathbb{E}_{S}[\text{Var}[W|S]]+\text{Var}_{S}[\mathbb{E}[W|S]] . Let W = G ^ Verifier β ( π , π β , π , π ) W=\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}}) . Given π {\bm{x}} and π β {\bm{y}}^{\star} , the sources of randomness for G ^ Verifier \hat{G}_{\text{Verifier}} are π {\bm{z}} and π {\bm{y}} . Let S = π S={\bm{z}} be the conditioning variables. The randomness in G ^ Verifier \hat{G}_{\text{Verifier}} given S S comes from π βΌ Ο ΞΈ ( β
| π , π ) {\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}}) . Applying the law: Var π , π β [ G ^ Verifier β ( π , π β , π , π ) ] = πΌ π β [ Var π | π β [ G ^ Verifier β ( π , π β , π , π ) ] ] + Var π β [ πΌ π | π β [ G ^ Verifier β ( π , π β , π , π ) ] ] \displaystyle\text{Var}_{{\bm{z}},{\bm{y}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]=\mathbb{E}_{{\bm{z}}}\!\Big[\text{Var}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]\Big]\!+\!\text{Var}_{{\bm{z}}}\!\Big[\mathbb{E}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]\Big] The expectation πΌ π \mathbb{E}_{{\bm{z}}} is over π βΌ Ο ΞΈ ( β
| π ) {\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}}) . The conditional variance Var π | π \text{Var}_{{\bm{y}}|{\bm{z}}} and expectation πΌ π | π \mathbb{E}_{{\bm{y}}|{\bm{z}}} are over π βΌ Ο ΞΈ ( β
| π , π ) {\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}}) for fixed π , π β , π {\bm{x}},{\bm{y}}^{\star},{\bm{z}} .
Substituting the result from Eq. 8 into the law of total variance: Var π , π β [ G ^ Verifier β ( π , π β , π , π ) ] = πΌ π β [ Var π | π β [ G ^ Verifier β ( π , π β , π , π ) ] ] + Var π β [ G ^ VeriFree β ( π , π β , π ) ] \displaystyle\text{Var}_{{\bm{z}},{\bm{y}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]=\mathbb{E}_{{\bm{z}}}\!\Big[\text{Var}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]\Big]+\text{Var}_{{\bm{z}}}\big[\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big] The second term, Var π β [ G ^ VeriFree β ( π , π β , π ) ] \text{Var}_{{\bm{z}}}\big[\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big] , is the definition of the variance of the estimator G ^ VeriFree \hat{G}_{\text{{VeriFree}{}}} . The first term, πΌ π β [ Var π | π β [ G ^ Verifier β ( π , π β , π ) ] ] \mathbb{E}_{{\bm{z}}}\Big[\text{Var}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big]\Big] , is an expectation of a variance. Since variance is always non-negative, Var π | π β [ G ^ Verifier β ( π , π β , π ) ] β₯ 0 \text{Var}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big]\geq 0 . Therefore, its expectation is also non-negative: πΌ π β [ Var π | π β [ G ^ Verifier β ( π , π β , π ) ] ] β₯ 0 \displaystyle\mathbb{E}_{{\bm{z}}}\Big[\text{Var}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big]\Big]\geq 0 Thus, we have: Var π , π β [ G ^ Verifier β ( π , π β , π , π ) ] = ( a non-negative term ) + Var π β [ G ^ VeriFree β ( π , π β , π ) ] \displaystyle\text{Var}_{{\bm{z}},{\bm{y}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]=(\text{a non-negative term})+\text{Var}_{{\bm{z}}}\big[\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big] This implies: Var π , π β [ G ^ Verifier β ( π , π β , π , π ) ] β₯ Var π β [ G ^ VeriFree β ( π , π β , π ) ] \displaystyle\text{Var}_{{\bm{z}},{\bm{y}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]\geq\text{Var}_{{\bm{z}}}\big[\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big] The variance of the policy gradient estimator G ^ VeriFree \hat{G}_{\text{{VeriFree}{}}} is less than or equal to that of G ^ Verifier \hat{G}_{\text{Verifier}} . This is an instance of Rao-Blackwellization, where analytically integrating out a source of randomness (the sampling of π {\bm{y}} ) by using its conditional expectation reduces variance. β
## Appendix B Off-policy Gradient Estimators
In the main paper we provide an expression for the gradient estimator when the data is fully on-policy. VeriFree is also fully compatible with PPO-style gradient clipping for the case when data is reused to improve sample efficiency. In this case the gradient estimator is:
β ΞΈ J VeriFree β ( ΞΈ ) \displaystyle\nabla_{\theta}J_{\textrm{VeriFree}{}}(\theta) = 1 G β β i = 1 G [ β t = 1 | π i | Clip β { Ο ΞΈ β ( π i , t β£ π , π i , < t ) Ο ΞΈ old β ( π i , t β£ π , π i , < t ) } β A i + β t β² = 1 | π β | Clip β { Ο ΞΈ β ( π t β β£ π , π i ) Ο ΞΈ old β ( π t β β£ π , π i ) } β R i ] , \displaystyle=\frac{1}{G}\sum_{i=1}^{G}\Bigg[\sum_{t=1}^{\left|{\bm{z}}_{i}\right|}\text{Clip}\bigg\{\frac{\pi_{\theta}\left({\bm{z}}_{i,t}\mid{\bm{x}},{\bm{z}}_{i,<t}\right)}{\pi_{\theta_{\text{old}}}\left({\bm{z}}_{i,t}\mid{\bm{x}},{\bm{z}}_{i,<t}\right)}\bigg\}A_{i}+\sum_{t^{\prime}=1}^{\left|{\bm{y}}^{\star}\right|}\text{Clip}\bigg\{\frac{\pi_{\theta}\left({\bm{y}}^{\star}_{t}\mid{\bm{x}},{\bm{z}}_{i}\right)}{\pi_{\theta_{\text{old}}}\left({\bm{y}}^{\star}_{t}\mid{\bm{x}},{\bm{z}}_{i}\right)}\bigg\}R_{i}\Bigg], where Ο ΞΈ old \pi_{\theta_{\text{old}}} is the sampling policy, A i = Ο ΞΈ old β ( π β | π , π i ) β 1 G β 1 β β j β i Ο ΞΈ old β ( π β | π , π j ) A_{i}=\pi_{\theta_{\text{old}}}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{i})-\frac{1}{G-1}\sum_{j\neq i}\pi_{\theta_{\text{old}}}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{j}) , R i = Ο ΞΈ old β ( π β | π , π i ) R_{i}=\pi_{\theta_{\text{old}}}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{i}) , and Clip β { β
} \text{Clip}\{\cdot\} denotes the PPO clipping operation.
## Appendix C Dataset Details
The category distribution in WebData (our training data) is visualized in Figure 7 .
## Appendix D Extended Empirical Results
In Tables 1 and 2 , we provide detailed benchmark results for MMLU-Pro and SuperGPQA using domain name abbreviations. The full nomenclature is as follows:
MMLU-Pro ( Table 1 ): CS (Computer Science), Math (Mathematics), Chem (Chemistry), Eng (Engineering), Law (Law), Bio (Biology), Health (Health), Phys (Physics), Bus (Business), Phil (Philosophy), Econ (Economics), Other (Other), Psy (Psychology), Hist (History).
SuperGPQA ( Table 2 ): Eng. (Engineering), Med. (Medicine), Sci. (Science), Phil. (Philosophy), M.S. (Military Science), Econ. (Economics), Mgmt. (Management), Socio. (Sociology), L/A (Literature and Arts), Hist. (History), Agron. (Agronomy), Law (Law), Edu (Education).
In Table 3 , we present an accuracy comparison across six math evaluation benchmarks and GPQA. Models trained with VeriFree demonstrate consistent and significant improvements over the base models, further validating the effectiveness of our approach.
## Appendix E Case Study
In this section, we provide some examples of reasoning traces produced by the model trained by our method on training questions. And we also provide an example with both our modelβs and base modelβs responses on the test dataset.
## Appendix F Broader Impact
Improving the general reasoning capabilities of large language models (LLMs) has the potential to significantly expand their applicability across domains such as education, scientific discovery, legal analysis, and decision support. By removing reliance on explicit verifiersβoften expensive or domain-specificβour approach democratizes access to advanced reasoning systems, enabling broader use in resource-constrained settings.
However, improved reasoning can amplify both beneficial and harmful behaviors, depending on the intent and oversight of the user. As such, responsible usage, transparent reporting of model limitations, and value alignment remain critical.
We encourage future work that complements our method with safeguards for misuse prevention and performance guarantees under distributional shifts, particularly as general-purpose reasoning models become more widely integrated into real-world workflows.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .