# Reinforcing General Reasoning without Verifiers

###### Abstract

The recent paradigm shift towards training large language models (LLMs) using DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has led to impressive advancements in code and mathematical reasoning. However, this methodology is limited to tasks where rule-based answer verification is possible and does not naturally extend to real-world domains such as chemistry, healthcare, engineering, law, biology, business, and economics. Current practical workarounds use an additional LLM as a model-based verifier; however, this introduces issues such as reliance on a strong verifier LLM, susceptibility to reward hacking, and the practical burden of maintaining the verifier model in memory during training. To address this and extend DeepSeek-R1-Zero-style training to general reasoning domains, we propose a verifier-free method ( VeriFree ) that bypasses answer verification and instead uses RL to directly maximize the probability of generating the reference answer. We compare VeriFree with verifier-based methods and demonstrate that, in addition to its significant practical benefits and reduced compute requirements, VeriFree matches and even surpasses verifier-based methods on extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related benchmarks. Moreover, we provide insights into this method from multiple perspectives: as an elegant integration of training both the policy and implicit verifier in a unified model, and as a variational optimization approach. Code is available at https://github.com/sail-sg/VeriFree .

## 1 Introduction

DeepSeek-R1-Zero [ 10 ] recently demonstrated that training large language models (LLMs) using reinforcement learning (RL) with verifiable rewards can be extremely effective in improving reasoning capabilities. In this RL with verifiable rewards (RLVR) framework [ 17 ] , the LLM generates a reasoning trace (i.e., chain of thoughts, CoT) followed by a final answer. A rule-based program then extracts and evaluates the final answer, assigning a reward of 1 to the response if the final answer is correct and 0 otherwise. The model is trained with RL using GRPO [ 37 ] —a simplified variant of PPO [ 36 ] .

The simplicity of this approach, coupled with impressive performance improvements in mathematical reasoning tasks, has sparked a wave of follow-up works in this paradigm of RL with rule-based verifiable rewards [ 24 , 26 , 45 ] , which we will refer to as the R1-Zero-style training in the following. However, these methods remain limited to domains such as mathematics and code, where rule-based verification is feasible. Reasoning is critical far beyond math and coding; however, the difficulty of answer verification in general reasoning tasks poses a major obstacle to applying this training paradigm to broader domains. To address this limitation, we investigate how to extend R1-Zero-style training to tasks where rule-based answer verification is not possible.

A natural extension, as explored in recent general reasoning works [ 38 , 27 ] , is to introduce a specialized LLM as a verifier, similar to the reward model used in RL from human feedback (RLHF) [ 51 , 30 ] . In these methods, the model-based verifier is queried to determine whether the generated answer is equivalent to the reference answer. Although this approach bypasses the need for rule-based evaluation, it introduces several potential drawbacks (as in standard RLHF): it depends on the availability of a strong verifier LLM, it converts the R1-Zero-style paradigm into optimizing a model-based reward, which makes it vulnerable to reward hacking [ 7 ] , and it adds significant computational overhead by requiring an additional model to be held in memory and queried during training.

In this work, we propose an alternative: a verifier-free approach that preserves the benefits of the RL paradigm while eliminating the reliance on explicit verification, either performed by rules or by models. Our method proceeds as follows. Given a question, we only generate the reasoning trace and concatenate it with the reference answer from the dataset. We then evaluate the likelihood of the reference answer conditioned on the question and the generated reasoning trace. This likelihood serves both as a reward signal for policy gradients on the reasoning trace and as a weighting term for supervised training of the reference answer. We term our method VeriFree since it does not rely on rule- or model-based verifiers, and give an illustration in Figure 2 .

This approach has several appealing properties. First, when there is a unique correct answer string, our method is equivalent in expectation to the objective in RLVR, but with lower variance, which can be viewed as a form of reward shaping [ 28 , 34 ] . Even when multiple valid answers exist, we show empirically that using just one as a reference provides a sufficient learning signal to elicit strong reasoning behavior. Additionally, this framework can be viewed through a variational lens as a neat way of optimizing over latent reasoning traces.

To make this work in practice, we identify and address several subtle challenges, including effective variance reduction and precise handling of tokenization at the reasoning-answer patching point. We conduct comprehensive ablations to understand the impact of each design choice. We benchmark our method across a diverse set of general reasoning tasks, and the results are striking: as shown in Figure 1 , VeriFree not only matches but often outperforms verifier-based alternatives, while being simpler, faster, less memory-intensive, and more robust.

## 2 Methodology

### 2.1 Preliminaries: Verifier-Based Reinforcement Learning

In RL applied to LLMs, the language model is treated as a policy π θ \pi_{\theta} that generates an output 𝒐 {\bm{o}} autoregressively in response to an input question 𝒙 {\bm{x}} . The goal is typically to optimize π θ \pi_{\theta} to maximize a given reward function R ⁡ ( 𝒙 , 𝒐 ) R({\bm{x}},{\bm{o}}) : θ ∈ arg ⁡ max 𝜃 𝔼 𝒐 ∼ π θ ( ⋅ | 𝒙 ) [ R ( 𝒙 , 𝒐 ) ] . \theta\in\underset{\theta}{\arg\max}\,\mathbb{E}_{{\bm{o}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\left[R({\bm{x}},{\bm{o}})\right]. (1) In R1-Zero-style RL, the reward is computed by first parsing the response 𝒐 {\bm{o}} into a reasoning trace 𝒛 {\bm{z}} and a final answer 𝒚 {\bm{y}} . A verifier then checks 𝒚 {\bm{y}} against the ground-truth reference answer 𝒚 ⋆ {\bm{y}}^{\star} and assigns a binary reward based on correctness, namely R Verifier ( 𝒚 ; 𝒚 ⋆ ) = 𝟙 { 𝒚 ≡ 𝒚 ⋆ } R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})=\mathds{1}_{\{{\bm{y}}\equiv{\bm{y}}^{\star}\}} . 1 1 1 We use ‘ ≡ \equiv ’ to denote semantic equivalence, where multiple expressions can be judged correct. For example, if 𝒚 ⋆ = {\bm{y}}^{\star}= ‘ 8/5 ’, then ‘ 8/5 ’, ‘ 1.6 ’, and ‘ \frac{8}{5} ’ are all considered correct. Decomposing the model output as 𝒐 = ( 𝒛 , 𝒚 ) {\bm{o}}=({\bm{z}},{\bm{y}}) , we can rewrite the objective in Equation 1 as: J Verifier ( θ ; 𝒙 , 𝒚 ⋆ ) = 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ R Verifier ( 𝒚 ; 𝒚 ⋆ ) ] , \displaystyle J_{\text{Verifier}}(\theta;{\bm{x}},{\bm{y}}^{\star})=\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}{\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}[R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})]}, (2) which separates the sampling of the reasoning trace and the final answer. To maximize this objective, the model π θ \pi_{\theta} is typically updated using the policy gradient estimator [ 39 ] : ∇ θ J Verifier ​ ( θ , 𝒙 , 𝒚 ⋆ ) = 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) ​ 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) ​ [ R Verifier ​ ( 𝒚 , 𝒚 ⋆ ) ​ [ ∇ θ ​ log ​ π θ ​ ( 𝒛 | 𝒙 ) + ∇ θ ​ log ​ π θ ​ ( 𝒚 | 𝒙 , 𝒛 ) ] ] . \small\nabla_{\theta}J_{\text{Verifier}}(\theta;{\bm{x}},{\bm{y}}^{\star})=\underset{{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}}{\mathbb{E}}~\underset{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}{\mathbb{E}}\Big[R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\big]\Big]. (3) However, this approach requires evaluating answer correctness via R Verifier ( 𝒚 ; 𝒚 ⋆ ) = 𝟙 { 𝒚 ≡ 𝒚 ⋆ } R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})=\mathds{1}_{\{{\bm{y}}\equiv{\bm{y}}^{\star}\}} , which is often nontrivial. While in domains such as math and code, this evaluation is feasible via rules [ 10 , 24 ] or test cases [ 8 ] , accurate verification in general reasoning tasks is substantially more difficult. As a result, recent advances in R1-Zero-style training have largely been restricted to verifiable domains, leaving reasoning tasks in general domains underexplored. In light of this, we present an alternative verifier-free approach which naturally extends this training paradigm to broader reasoning domains.

### 2.2 VeriFree Policy Optimization

We begin with the standard objective in Equation 2 and show that, in the case of a single correct answer, we can derive an equivalent objective that does not require a verifier. Moreover, we compare the gradient estimators of this new objective and the verifier-based objective ( Equation 3 ), and demonstrate that our verifier-free gradient estimator has the additional benefit of lower variance.

Starting from Equation 2 and assuming a unique correct answer such that R Verifier ( 𝒚 ; 𝒚 ∗ ) = 𝟙 { 𝒚 = 𝒚 ⋆ } R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{*})=\mathds{1}_{\{{\bm{y}}={\bm{y}}^{\star}\}} (i.e., exact match rather than semantic equivalence 𝟙 { 𝒚 ≡ 𝒚 ⋆ } \mathds{1}_{\{{\bm{y}}\equiv{\bm{y}}^{\star}\}} ), the VeriFree objective is derived as follows: J Verifier ​ ( θ , 𝒙 , 𝒚 ⋆ ) \displaystyle J_{\text{Verifier}}(\theta;{\bm{x}},{\bm{y}}^{\star}) = 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ 𝟙 { 𝒚 = 𝒚 ⋆ } ⏞ R Verifier ] ] \displaystyle=\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[{\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}[\,\overbrace{\mathds{1}_{\{{\bm{y}}={\bm{y}}^{\star}\}}}^{R_{\text{Verifier}}}\,]}\Big] = 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ ∑ 𝒚 π θ ( 𝒚 | 𝒙 , 𝒛 ) 𝟙 { 𝒚 = 𝒚 ⋆ } ] \displaystyle=\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[\sum_{{\bm{y}}}\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\mathds{1}_{\{{\bm{y}}={\bm{y}}^{\star}\}}\Big] = 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ⏟ R VeriFree ] ≜ J VeriFree ( θ ; 𝒙 , 𝒚 ⋆ ) . \displaystyle=\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[{\qquad\>\,\underbrace{\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})}_{R_{\text{{VeriFree} }}}\qquad\>\,}\Big]\triangleq J_{\text{{VeriFree} }}(\theta;{\bm{x}},{\bm{y}}^{\star}). (4) This can be interpreted as follows: if only one answer 𝒚 ⋆ {\bm{y}}^{\star} is correct and receives a reward of 1 (while all others receive 0), then the expected reward given a reasoning trace 𝒛 {\bm{z}} can be computed directly as the probability assigned to 𝒚 ⋆ {\bm{y}}^{\star} , effectively marginalizing out 𝒚 {\bm{y}} . The corresponding gradient estimator is given by (see Section A.1 for a full derivation): ∇ θ J VeriFree ( θ ; 𝒙 , 𝒚 ⋆ ) = 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ R VeriFree ( 𝒙 , 𝒚 ⋆ , 𝒛 ) [ ∇ θ ​ log ​ π θ ​ ( 𝒛 | 𝒙 ) ⏟ reasoning term + ∇ θ ​ log ​ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ⏟ reference answer term ] ] . \footnotesize\nabla_{\theta}J_{\text{{VeriFree} }}(\theta;{\bm{x}},{\bm{y}}^{\star})=\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\bigg[R_{\text{{VeriFree} }}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big[\underbrace{\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})}_{\text{reasoning term}}+\underbrace{\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})}_{\text{reference answer term}}\big]\bigg]. (5) Both the objective and its gradient estimator ( Equations 4 and 5 ) are equivalent in expectation to their verifier-based counterparts ( Equations 2 and 3 ). Intuitively, the “reasoning term” in Equation 5 can be interpreted as a policy gradient where the reward for a reasoning trace 𝒛 {\bm{z}} is the probability that the policy will generate the correct answer 𝒚 ⋆ {\bm{y}}^{\star} given 𝒛 {\bm{z}} , while “reference answer term” can be viewed as a reward-weighted supervised learning term for 𝒚 ⋆ {\bm{y}}^{\star} given 𝒛 {\bm{z}} . We will further elaborate on this interpretation in Section 2.3 . In addition to bypassing the need for a verifier, our VeriFree gradient estimator also benefits from reduced variance:

###### Theorem 1 .

( Variance Reduction ) Let G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) \hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}}) and G ^ VeriFree ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) \hat{G}_{\text{{VeriFree} }}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}}) denote the single-sample Monte Carlo estimators of ∇ θ J Verifier \nabla_{\theta}J_{\text{Verifier}} and ∇ θ J VeriFree \nabla_{\theta}J_{\text{{VeriFree} }} given 𝒙 {\bm{x}} and 𝒚 ⋆ {\bm{y}}^{\star} , respectively. Then we have Var 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ G ^ VeriFree ( 𝒙 , 𝒚 ⋆ , 𝒛 ) ] ≤ Var 𝒛 ∼ π θ ( ⋅ | 𝒙 ) , 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ G ^ Verifier ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) ] . \text{Var}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\big[\hat{G}_{\text{{VeriFree} }}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big]\leq\text{Var}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}}),{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}})\big]. (6)

This reduction in variance arises from Rao-Blackwellization [ 2 ] . For intuition, the variance in the Monte Carlo estimate of ∇ θ J Verifier \nabla_{\theta}J_{\text{Verifier}} stems from the randomness in sampling 𝒛 ∼ π θ ( ⋅ | 𝒙 ) , 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) {\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}}),{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}}) , while for estimating ∇ θ J VeriFree \nabla_{\theta}J_{\text{{VeriFree} }} we analytically marginalizes out 𝒚 {\bm{y}} , thereby removing this source of randomness. We provide a full proof in Section A.2 .

Our gradient estimator ∇ θ J VeriFree ​ ( θ , 𝒙 , 𝒚 ⋆ ) \nabla_{\theta}J_{\text{{VeriFree} }}(\theta;{\bm{x}},{\bm{y}}^{\star}) is fully compatible with other variance reduction techniques, including RLOO [ 1 ] , GRPO [ 37 ] reward normalizations, and the PPO [ 36 ] clipping operation. As such, we sample multiple responses for each prompt and apply the RLOO baseline to the reasoning term in Equation 5 . We also adopt the corrected response-length normalization from Liu et al. [24] . The final on-policy gradient estimator is as follows: ∇ θ J VeriFree ​ ( θ ) = 1 G ​ ∑ i = 1 G [ A i ⋅ ∇ θ ​ log ​ π θ ​ ( 𝒛 i | 𝒙 ) + R i ⋅ ∇ θ ​ log ​ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 i ) ] , \nabla_{\theta}J_{\text{{VeriFree} }}(\theta)=\frac{1}{G}\sum_{i=1}^{G}\left[A_{i}\cdot\nabla_{\theta}\log\pi_{\theta}({\bm{z}}_{i}|{\bm{x}})+R_{i}\cdot\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{i})\right], (7) where 𝒛 i ∼ π θ ( ⋅ | 𝒙 ) {\bm{z}}_{i}\sim\pi_{\theta}(\cdot|{\bm{x}}) , R i = π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 i ) R_{i}=\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{i}) , and A i = π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 i ) − 1 G − 1 ​ ∑ j ≠ i π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 j ) A_{i}=\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{i})-\frac{1}{G-1}\sum_{j\neq i}\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{j}) . We also provide the PPO-based off-policy variant in Appendix B .

### 2.3 Comparison to Existing Approaches

There have been two main prior works that, although derived from a different perspective, arrive at related alternative gradient estimators: Tang et al. [40] (JLB), and Chen et al. [4] (LaTRO). ∇ θ J Verifier \displaystyle\nabla_{\theta}J_{\text{Verifier}} = 𝔼 𝒛 , 𝒚 ​ [ 𝟙 { 𝒚 ≡ 𝒚 ⋆ } ​ ∇ θ ​ log ​ π θ ​ ( 𝒛 | 𝒙 ) ⏞ reasoning term + 𝟙 { 𝒚 ≡ 𝒚 ⋆ } ​ ∇ θ ​ log ​ π θ ​ ( 𝒚 | 𝒙 , 𝒛 ) ⏞ answer term ] \displaystyle=\mathbb{E}_{{\bm{z}},{\bm{y}}}\big[\hbox{\pagecolor{myred}$\mathds{1}_{\{{\bm{y}}\equiv{\bm{y}}^{\star}\}}$}\overbrace{\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})}^{\text{reasoning term}}+\hbox{\pagecolor{myblue}$\mathds{1}_{\{{\bm{y}}\equiv{\bm{y}}^{\star}\}}$}\overbrace{\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}^{\text{answer term}}\big] (R1-Zero) ∇ θ J VeriFree \displaystyle\nabla_{\theta}J_{\text{{VeriFree} }} = 𝔼 𝒛 ​ [ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ​ ∇ θ ​ log ⁡ π θ ​ ( 𝒛 | 𝒙 ) + π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ​ ∇ θ ​ log ​ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ⏞ reference answer term ] \displaystyle=\mathbb{E}_{\bm{z}}\big[\hbox{\pagecolor{myred}$\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})$}\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\hbox{\pagecolor{myblue}$\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})$}\overbrace{\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})}^{\text{reference answer term}}\big]\qquad (Ours) ∇ θ J JLB \displaystyle\nabla_{\theta}J_{\text{JLB}} = 𝔼 𝒛 ​ [ log ⁡ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ​ ∇ θ ​ log ​ π θ ​ ( 𝒛 | 𝒙 ) + 1 ⋅ ∇ θ ​ log ​ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] \displaystyle=\mathbb{E}_{\bm{z}}\big[\hbox{\pagecolor{myred}$\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})$}\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\hbox{\pagecolor{myblue}$1$}\cdot\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big] ( Tang et al. [40] ) ∇ θ J LaTRO \displaystyle\nabla_{\theta}J_{\text{LaTRO}} = 𝔼 𝒛 ​ [ ( log ⁡ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) − log ⁡ π θ ​ ( 𝒛 | 𝒙 ) π ref ​ ( 𝒛 | 𝒙 ) ) ​ ∇ θ ​ log ​ π θ ​ ( 𝒛 | 𝒙 ) + 1 ⋅ ∇ θ ​ log ​ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] \displaystyle=\mathbb{E}_{\bm{z}}\big[\hbox{\pagecolor{myred}$(\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})-\log\frac{\pi_{\theta}({\bm{z}}|{\bm{x}})}{\pi_{\text{ref}}({\bm{z}}|{\bm{x}})})$}\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\hbox{\pagecolor{myblue}$1$}\cdot\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big] ( Chen et al. [4] )

Both JLB and LaTRO treat the reasoning trace 𝒛 {\bm{z}} as a latent variable and extend the standard supervised learning objective (log-likelihood) to optimize lower bounds on log ( E 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] ) \log(E_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\left[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\right]) and log ( E 𝒛 ∼ π ref ( ⋅ | 𝒙 ) [ π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] ) \log(E_{{\bm{z}}\sim\pi_{\text{ref}}(\cdot|{\bm{x}})}\left[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\right]) , respectively. The primary difference is that JLB samples 𝒛 {\bm{z}} from the learned policy π θ \pi_{\theta} , while LaTRO uses a fixed reference policy π ref \pi_{\text{ref}} . Despite originating from different perspectives, these approaches arrive at alternative gradient estimators that can be used similarly to ours, as shown in the above comparisons. However, as reported in Tang et al. [40] , these verifier-free, variational-inference-based methods consistently underperform the standard verifier-based R1-Zero approach. In contrast, our formulation matches or outperforms the verifier-based baseline.

One possible explanation is that our method exactly recovers the original verifier-based objective under the single-correct-answer assumption, whereas JLB and LaTRO optimize subtly different objectives. For example, JLB effectively uses a reward of R = log ⁡ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) R=\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}) , as highlighted in the gradient expressions above. Another distinction lies in the weighting of the “reference answer term” ∇ θ ​ log ​ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) \nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}) . In our method, this term is weighted by the probability π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) \pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}) , which is the likelihood of the reference answer given the sampled reasoning trace. In contrast, both JLB and LaTRO use a fixed weight of 1, thereby increasing the probability of 𝒚 ⋆ {\bm{y}}^{\star} regardless of the quality of the reasoning trace 𝒛 {\bm{z}} . We hypothesize that this behavior could promote poor reasoning. For instance, suppose the model generates the reasoning trace “ ... minus 2 apples, finally resulting in a total of 7 apples ” when the correct answer is “ 6 ”. The JLB and LaTRO objectives would still push the model to output “ 6 ” from that flawed trace, reinforcing a mismatch between reasoning and answer. Our method avoids this by down-weighting contributions from low-quality traces.

### 2.4 How to handle the tokenization at patching point?

A critical consideration when extracting reasoning traces 𝒛 {\bm{z}} from model responses ( 𝒛 , 𝒚 ) ({\bm{z}},{\bm{y}}) and subsequently replacing 𝒚 {\bm{y}} with 𝒚 ⋆ {\bm{y}}^{\star} stems from the fact that LLMs operate on token sequences, not raw text strings. While human-readable outputs (e.g., “ ...<answer> \\boxed{...} </answer> ” as in Template 1 ) suggest splitting reasoning traces 𝒛 {\bm{z}} at specific text patterns like “ <answer> ”, such text-based splitting strategy may cause tokenization inconsistencies. For example, the “ > ” character might be tokenized differently depending on its surrounding context in 𝒚 {\bm{y}} versus 𝒚 ⋆ {\bm{y}}^{\star} . While one potential solution is to introduce special tokens to enforce consistent tokenization boundaries, these novel tokens could harm model performance due to their absence from the base model’s pretraining vocabulary.

Instead, we resolve this by defining the end of 𝒛 {\bm{z}} at the token corresponding to <answer ” (i.e., without “ > ”), leveraging the fact that the pattern “ r> ” does not appear in standard tokenizer vocabularies. This ensures consistent token-space alignment between sampling and optimization in RL, avoiding instability due to off-policy mismatches. Notably, this approach is operationally equivalent to setting “ <answer ” (instead of “ <answer> ”) as the stop word during sampling, a mechanism natively supported by modern inference engines like vLLM [ 16 ] . In this case, we can sample reasoning traces 𝒛 {\bm{z}} directly, instead of first generating the full response ( 𝒛 {\bm{z}} , 𝒚 {\bm{y}} ) and then extracting 𝒛 {\bm{z}} post hoc.

## 3 Experiments

### 3.1 Setup

Training. Following the “zero” setting widely adopted in recent work [ 10 , 14 , 24 , 27 ] , we directly fine-tune the base LLM, skipping the intermediate stage of supervised fine-tuning (SFT). We implement our RL training pipeline using Oat [ 22 ] by instantiating their base modules and incorporate our algorithmic changes. Our experiments are conducted using Qwen3 [ 41 ] base models across multiple scales, including 1.7B, 4B, and 8B parameters. We adopt the prompt template shown in Template 1 . We do not employ KL regularization losses or KL penalties in rewards, as recent studies suggest that removing KL terms does not have a significant impact [ 24 , 14 ] . As a result, our method does not require maintaining a reference model in memory.

For the 1.7B and 4B models, we conduct fine-tuning for approximately 4,000 policy gradient steps; for the 8B models, we fine-tune for around 3,000 policy gradient steps. During each step, the policy model (i.e., the LLM) generates 8 responses for each question (i.e., group_size=8 ), with 16 questions processed per step. We use the sampling configurations temperature=1.0 , top_p=1 , and max_tokens=3000 for the rollout process. The responses are then parsed into reasoning traces and model-predicted answers. We replace the model-predicted answers with the reference answers from the training dataset. Subsequently, a single forward pass is executed to compute the conditional probability of the reference answer, conditioned on all preceding tokens including the prompt and the reasoning trace. This procedure introduces only a minimal additional computational cost, as the forward pass of the LLM does not require autoregressive decoding and does not require storing intermediate states for backpropagation. All collected samples from each step are used for one optimization step. The training is conducted on a single node with 8 × \times H100 GPUs.

Dataset. To support general reasoning, we begin with the dataset curated by Ma et al. [27] , sourced from WebInstruct [ 49 ] . To improve data quality and reliability and reduce size, we retain samples with answers that consist of fewer than seven tokens, and use Qwen2.5-72B-Instruct [ 43 ] to filter out low-quality and noisy data. This process results in approximately 61,000 data samples spanning diverse domains, which we refer to as WebData . The category distribution is visualized in Figure 7 .

Evaluation. In line with prior work [ 27 ] , we employ multiple-choice questions for evaluation to facilitate verification. To assess general reasoning abilities, we utilize the following benchmarks: MMLU-Pro [ 42 ] , a challenging multi-task understanding benchmark designed to evaluate the capabilities of LLMs across various domains; SuperGPQA [ 6 ] , a large-scale benchmark consisting of graduate-level reasoning questions spanning 285 diverse disciplines; and GPQA [ 35 ] , which focuses on graduate-level question-answering and is designed to resist shallow pattern-matching and memorization. While our primary focus is not on enhancing mathematical abilities, we also evaluate math reasoning using a suite of standard math reasoning benchmarks. This suite includes MATH-500 [ 13 ] , OlympiadBench [ 11 ] , Minerva Math [ 18 ] , GSM8K [ 5 ] , AMC and AIME24 [ 19 ] . We utilize Math-Verify 2 2 2 https://github.com/huggingface/Math-Verify to check for answer equivalence. Except for AIME24, where we employ a temperature=1.0 and repeat each question 32 times, all other benchmarks are evaluated using temperature=0.0 . We use max_tokens=8192 for all evaluations.

Baselines. Our primary baseline, denoted Verifier , is a verifier-based approach using the verifier from Ma et al. [27] . The verifier is initialized from Qwen2.5-Math-1.5B [ 44 ] and fine-tuned on data generated by Gemini 2.0 Flash to assess equivalence between the reference and predicted answers, conditioned on the question. We apply Dr. GRPO [ 24 ] as the optimization algorithm for the baseline, ensuring that all other settings are consistent with our approach. Following Ma et al. [27] , the reward definition incorporates additional factors beyond verifier correctness, including format compliance and the length of generated answers. If the format is incorrect (e.g., missing \\boxed{} in the model response), a negative reward of -0.5 is applied. Moreover, a length penalty of -0.05 × \times min(10, abs(length_of_correct_answer - length_of_answer)) is added.

We also report the results for the base and the instruct models of Qwen-3-1.7B/4B/8B [ 41 ] and Qwen2.5-7B [ 43 ] , as well as the checkpoints released by Qwen2.5-7B-SimpleRL-Zoo [ 50 ] , Qwen2.5-Math-7B-Oat-Zero [ 24 ] , and General-Reasoner-7B [ 27 ] . 3 3 3 Qwen2.5 and Qwen3 use different naming conventions in their official release. For base models: Qwen2.5 has no suffix (e.g., Qwen2.5-7B), whereas Qwen3 adds “-Base” (e.g., Qwen3-8B-Base). For instruct models: Qwen2.5 uses “-Instruct” (e.g., Qwen2.5-7B-Instruct), while Qwen3 omits the suffix (e.g., Qwen3-8B). We follow these conventions consistently in the paper. Notably, Qwen3 integrates both a thinking mode (for complex, multi-step reasoning) and a non-thinking mode (for rapid, context-driven responses) within a unified framework. We report results of both modes on Qwen3 instruct models.

### 3.2 Main Results

VeriFree improves general reasoning capabilities. We begin by evaluating the effectiveness of VeriFree in enhancing the general reasoning capabilities of LLMs using the MMLU-Pro and SuperGPQA benchmarks. Table 1 presents a detailed comparison across model scales and domains in the MMLU-Pro benchmark. Starting from base models, applying RL with VeriFree yields substantial gains in average accuracy—ranging from 12% to 40%—demonstrating that VeriFree effectively fine-tunes LLMs to improve general reasoning performance.

Similar improvements are observed on the SuperGPQA benchmark, as shown in Table 2 , where VeriFree consistently enhances the performance of base models by a significant margin. Notably, VeriFree achieves performance comparable to, or even surpassing, that of the instruct model in thinking mode and the Verifier baseline (i.e., the RL-tuned model learned with an additional model-based verifier)—without relying on any explicit verification signals.

In addition to accuracy gains, we also observe an increase in response length after tuned by VeriFree, suggesting that the model explores longer reasoning traces to arrive at more accurate answers—a behavior reminiscent of DeepSeek-R1-Zero [ 10 ] . Results on the GPQA benchmark are provided in Appendix D due to space constraints.

VeriFree leads to better learning efficiency. We compare VeriFree with the baseline that learns from a model-based verifier reward (i.e., Verifier). As shown in Figure 4 (Left), VeriFree consistently outperforms the baseline, achieving higher accuracy with fewer training steps. We attribute this improved learning efficiency to reduced gradient variance, enabled by VeriFree’s continuous reward signals and the RLOO objective. While both approaches optimize the same reward signal in expectation, VeriFree provides more stable and informative policy gradients, which accelerate convergence and leads to better final performance.

Model confidence is a good reasoning capability proxy. Our analysis based on Qwen3-8B base model reveals a strong positive correlation ( ρ = 0.82 \rho=0.82 ) between MMLU-Pro accuracy and the average model confidence π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) \pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}) during training ( Figure 4 , Right). This empirically demonstrates that the model’s self-estimated confidence in the correct answer, i.e., π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) \pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}) , serves as an effective metric for quantifying emergent reasoning capabilities in LLMs.

VeriFree learns transferable reasoning skills. To evaluate the transferability of reasoning acquired through VeriFree, we train a model on a dataset with all math-related examples removed, and assess its performance on both general and math-specific benchmarks. As shown in Figure 5 , VeriFree not only improves reasoning performance on general tasks, as expected, but also demonstrates strong transfer to math benchmarks—despite the absence of math supervision during training. This highlights VeriFree’s ability to induce general reasoning capabilities that extend across domains.

### 3.3 Ablation Study

To systematically evaluate method components and offer a comprehensive understanding of VeriFree, we conduct ablation studies based on Qwen3-1.7B base models as follows.

Effects of extraction strategy for reasoning trace z {\bm{z}} . Our method requires precise separation between reasoning path 𝒛 {\bm{z}} and answer 𝒚 {\bm{y}} to enable answer replacement. While human-readable splits using <answer> seem intuitive, we instead define 𝒛 {\bm{z}} to end at “ <answer ” (omitting “ > ”), ensuring consistent tokenization boundaries (see Section 2.4 ). We compare with a variant using text-based splitting (denoted as “VeriFree w/o token split”) on Qwen3-1.7B via MMLU-Pro ( Figure 6 ). Our tokenization-aware approach achieves superior convergence, while the variant suffers optimization instability due to effectively introducing off-policy data.

Effects of RLOO. As observed in Figure 6 (Left), removing RLOO leads to a consistent drop in performance throughout training, with final accuracy more than 3 % 3\% lower than that of the full method. This highlights the importance of RLOO in stabilizing learning and guiding the model toward better generalization. Without RLOO, the model converges prematurely and fails to reach the same peak accuracy.

Effects of equivalence class. As mentioned in Section 2.1 , verifier-based RL typically assesses answer correctness as rewards. Correct answers within a specific class often form an equivalence class. Our method, however, utilizes model confidence by focusing on a single reference answer for a given question and the model’s reasoning trace. To explore the potential advantages of integrating an “equivalence class” into our approach, we conducted ablation studies as follows. We employed a model fine-tuned on the MATH-12k dataset [ 12 , 20 ] from Qwen3-8B base model through Dr. GRPO [ 24 ] with rule-based verification to sample answers on MATH-12k, subsequently verifying answer correctness using Math-Verify. This approach enabled us to create an extended dataset with a set of equivalent correct answers for each question. We then fine-tuned Qwen3-1.7B base models using our method on both the original and the extended MATH-12k datasets incorporating equivalence classes. These models are evaluated on GSM8K, MATH-500, Minerva Math, and OlympiadBench to assess the impact of including equivalence classes. The results, shown in Figure 6 (Right), indicate that considering equivalence classes in our method offers slight performance improvements, aligning with our expectations. This highlights a minor limitation of our current formulation and motivates future work on algorithms that can better leverage answer equivalence.

## 4 Related Works

DeepSeek-R1-Zero-like reinforcement learning. DeepSeek-R1-Zero [ 10 ] , Tülu [ 17 ] and OpenAI’s o1 [ 29 ] recently demonstrated that applying RL to learn directly from binary verification-based rewards can be extremely powerful in enhancing the reasoning capabilities of base LLMs. Since then several works have reproduced R1-Zero-like training on smaller scales [ 50 , 31 , 23 , 14 , 26 , 25 ] . The aforementioned works all focus on math and coding, domains where verifiable rewards are readily available. By contrast, our work aims to extend the R1-Zero-like training paradigm to general domains where verifiable rewards are not available.

Reasoning beyond verifiable domains. Previous work on reasoning without access to verifiable rewards has been based around employing an additional LLM to act as a proxy verifier or reward model. NaturalReasoning [ 47 ] introduces a large, multi-domain dataset and presents baselines trained using RFT [ 48 ] and DPO [ 33 ] , leveraging a second LLM as a reward model, while Su et al. [38] and General-Reasoner [ 27 ] similarly incorporate a separate LLM to serve as a verifier. X-Reasoner [ 21 ] also investigates general reasoning but circumvents the lack of rule-based verification by dropping the R1-Zero-style training paradigm, instead training via SFT [ 9 ] on responses sampled from more capable models.

Self-improving language models. Several works have explored training LLMs using signals based on the model’s own outputs. Yuan et al. [46] propose to prompt the model to judge and rank different responses and select the best and worst for preference learning. Chen et al. [3] leverage the DPO implicit rewards for more efficient and robust self-alignment via iterative DPO [ 33 ] . Zuo et al. [52] use majority voting to construct self-labeled rewards for RL to further improve well-trained models during test time, which can be understood as a form of sharpening [ 15 ] . Another line of research [ 32 , 4 , 40 ] approaches LLM reasoning from the direction of variational optimization, treating the reasoning trace as a latent variable. Despite starting from a different viewpoint our method has interesting and close connections to this perspective which we discuss in detail in Section 2.3 .

## 5 Conclusions

In this paper, we rethink reinforcement learning with verifiable rewards (RLVR) for LLMs from a novel perspective. By leveraging the gradient equivalence under the unique answer assumption, we derive a new optimization objective that eliminates the need for explicit verification. Our proposed method, VeriFree, is particularly well-suited for general reasoning tasks, where rule-based verifiers are infeasible and model-based verifiers are both expensive and vulnerable to reward hacking. Through extensive experiments and ablations, we demonstrate the effectiveness and robustness of VeriFree on a wide range of general reasoning benchmarks. We hope our work offers a fresh viewpoint for the LLM RL community and provides a practical approach for building future general-purpose reasoners.

## Appendix

## Table of Contents

## Appendix A Theoretical Analysis

### A.1 Derivation of Gradient Estimators

Here we provide derivations for Equation 5 —the gradient estimator of VeriFree. We also include the corresponding derivation for the standard verifier-based gradient estimator for completeness.

###### Proof.

The gradient estimator for J Verifier J_{\text{Verifier}} is derived as follows: ∇ θ J Verifier ​ ( θ , 𝒙 , 𝒚 ⋆ ) \displaystyle\nabla_{\theta}J_{\text{Verifier}}(\theta;{\bm{x}},{\bm{y}}^{\star}) = \displaystyle=\, ∇ θ 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ R Verifier ( 𝒚 ; 𝒚 ⋆ ) ] ] \displaystyle\nabla_{\theta}\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\big[R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})\big]\Big] = \displaystyle=\, ∇ θ ∑ 𝒛 , 𝒚 R Verifier ( 𝒚 ; 𝒚 ⋆ ) π θ ( 𝒚 | 𝒙 , 𝒛 ) π θ ( 𝒛 | 𝒙 ) \displaystyle\nabla_{\theta}\sum_{{\bm{z}},{\bm{y}}}R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}}) = \displaystyle=\, ∑ 𝒛 , 𝒚 R Verifier ​ ( 𝒚 , 𝒚 ⋆ ) ​ [ π θ ​ ( 𝒚 | 𝒙 , 𝒛 ) ​ ∇ θ π θ ​ ( 𝒛 | 𝒙 ) + π θ ​ ( 𝒛 | 𝒙 ) ​ ∇ θ π θ ​ ( 𝒚 | 𝒙 , 𝒛 ) ] \displaystyle\sum_{{\bm{z}},{\bm{y}}}R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})\big[\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\nabla_{\theta}\pi_{\theta}({\bm{z}}|{\bm{x}})+\pi_{\theta}({\bm{z}}|{\bm{x}})\nabla_{\theta}\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\big] = \displaystyle=\, ∑ 𝒛 , 𝒚 R Verifier ​ ( 𝒚 , 𝒚 ⋆ ) ​ [ π θ ​ ( 𝒚 | 𝒙 , 𝒛 ) ​ π θ ​ ( 𝒛 | 𝒙 ) ​ ∇ θ ​ log ⁡ π θ ​ ( 𝒛 | 𝒙 ) + π θ ​ ( 𝒛 | 𝒙 ) ​ π θ ​ ( 𝒚 | 𝒙 , 𝒛 ) ​ ∇ θ ​ log ⁡ π θ ​ ( 𝒚 | 𝒙 , 𝒛 ) ] \displaystyle\sum_{{\bm{z}},{\bm{y}}}R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})\big[\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}})\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\pi_{\theta}({\bm{z}}|{\bm{x}})\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\big] = \displaystyle=\, 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ R Verifier ( 𝒚 ; 𝒚 ⋆ ) [ ∇ θ log π θ ( 𝒛 | 𝒙 ) + ∇ θ log π θ ( 𝒚 | 𝒙 , 𝒛 ) ] ] ] \displaystyle\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\bigg[\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\Big[R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\big]\Big]\bigg] The gradient estimator for J VeriFree J_{\text{{VeriFree} }} is derived as follows: ∇ θ J VeriFree ​ ( θ , 𝒙 , 𝒚 ⋆ ) \displaystyle\nabla_{\theta}J_{\text{{VeriFree}{}}}(\theta;{\bm{x}},{\bm{y}}^{\star}) = \displaystyle=\, ∇ θ 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ R VeriFree ( 𝒛 ; 𝒙 , 𝒚 ⋆ ) ] \displaystyle\nabla_{\theta}\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}[R_{\text{{VeriFree}{}}}({\bm{z}};{\bm{x}},{\bm{y}}^{\star})] = \displaystyle=\, ∇ θ 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] \displaystyle\nabla_{\theta}\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})] = \displaystyle=\, ∇ θ ∑ 𝒛 π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) π θ ( 𝒛 | 𝒙 ) \displaystyle\nabla_{\theta}\sum_{{\bm{z}}}\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}}) = \displaystyle=\, ∑ 𝒛 [ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ​ ∇ θ π θ ​ ( 𝒛 | 𝒙 ) + π θ ​ ( 𝒛 | 𝒙 ) ​ ∇ θ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] \displaystyle\sum_{{\bm{z}}}\big[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\nabla_{\theta}\pi_{\theta}({\bm{z}}|{\bm{x}})+\pi_{\theta}({\bm{z}}|{\bm{x}})\nabla_{\theta}\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big] = \displaystyle=\, ∑ 𝒛 [ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ​ π θ ​ ( 𝒛 | 𝒙 ) ​ ∇ θ ​ log ⁡ π θ ​ ( 𝒛 | 𝒙 ) + π θ ​ ( 𝒛 | 𝒙 ) ​ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ​ ∇ θ ​ log ⁡ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] \displaystyle\sum_{{\bm{z}}}\big[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}})\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\pi_{\theta}({\bm{z}}|{\bm{x}})\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big] = \displaystyle=\, 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) [ ∇ θ log π θ ( 𝒛 | 𝒙 ) + ∇ θ log π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] ] \displaystyle\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big]\Big] = \displaystyle=\, 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ R VeriFree ( 𝒛 ; 𝒙 , 𝒚 ⋆ ) [ ∇ θ log π θ ( 𝒛 | 𝒙 ) + ∇ θ log π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] ] \displaystyle\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[R_{\text{{VeriFree}{}}}({\bm{z}};{\bm{x}},{\bm{y}}^{\star})\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big]\Big] ∎

### A.2 Proof of Lower Variance

Here we provide a full proof of Equation 6 , the reduced variance property of VeriFree. We show that the policy gradient estimator derived from J VeriFree ​ ( θ , 𝒙 , 𝒚 ⋆ ) J_{\text{{VeriFree}{}}}(\theta;{\bm{x}},{\bm{y}}^{\star}) has variance less than or equal to that of the estimator derived from J Verifier ​ ( θ , 𝒙 , 𝒚 ⋆ ) J_{\text{Verifier}}(\theta;{\bm{x}},{\bm{y}}^{\star}) for any given 𝒙 , 𝒚 ⋆ {\bm{x}},{\bm{y}}^{\star} . The same relationship will hold for the global objectives averaged over ( 𝒙 , 𝒚 ⋆ ) ∼ 𝒟 ({\bm{x}},{\bm{y}}^{\star})\sim\mathcal{D} .

###### Proof.

The global objective functions are: J Verifier ​ ( θ ) \displaystyle J_{\text{Verifier}}(\theta) = 𝔼 ( 𝒙 , 𝒚 ⋆ ) ∼ 𝒟 [ 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ 𝟙 { 𝒚 = 𝒚 ⋆ } ] ] ] \displaystyle=\mathbb{E}_{({\bm{x}},{\bm{y}}^{\star})\sim\mathcal{D}}\left[\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\big[\mathbbm{1}\{{\bm{y}}={\bm{y}}^{\star}\}\big]\Big]\right] J VeriFree ​ ( θ ) \displaystyle J_{\text{{VeriFree}{}}}(\theta) = 𝔼 ( 𝒙 , 𝒚 ⋆ ) ∼ 𝒟 [ 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] ] \displaystyle=\mathbb{E}_{({\bm{x}},{\bm{y}}^{\star})\sim\mathcal{D}}\Big[\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\big[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big]\Big] For a given ( 𝒙 , 𝒚 ⋆ ) ∼ 𝒟 ({\bm{x}},{\bm{y}}^{\star})\sim\mathcal{D} , the single-sample Monte Carlo gradient estimators are: G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) \displaystyle\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}}) = 𝟙 { 𝒚 = 𝒚 ⋆ } [ ∇ θ log π θ ( 𝒛 | 𝒙 ) + ∇ θ log π θ ( 𝒚 | 𝒙 , 𝒛 ) ] \displaystyle=\mathbbm{1}\{{\bm{y}}={\bm{y}}^{\star}\}\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\big] where 𝒛 ∼ π θ ( ⋅ | 𝒙 ) {\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}}) , 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) {\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}}) , and G ^ VeriFree ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) \displaystyle\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}}) = π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ​ [ ∇ θ ​ log ​ π θ ​ ( 𝒛 | 𝒙 ) + ∇ θ ​ log ​ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] \displaystyle=\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big] where 𝒛 ∼ π θ ( ⋅ | 𝒙 ) {\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}}) . The proof relies on the law of total variance and the relationship between G ^ Verifier \hat{G}_{\text{Verifier}} and G ^ VeriFree \hat{G}_{\text{{VeriFree}{}}} .

First, we show that G ^ VeriFree ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) \hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}}) is the conditional expectation of G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) \hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}}) given 𝒙 , 𝒚 ⋆ , 𝒛 {\bm{x}},{\bm{y}}^{\star},{\bm{z}} . The expectation is taken over 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) {\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}}) : 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ G ^ Verifier ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) | 𝒙 , 𝒚 ⋆ , 𝒛 ] \displaystyle\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}})|{\bm{x}},{\bm{y}}^{\star},{\bm{z}}\big] = \displaystyle=\, 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ 𝟙 { 𝒚 = 𝒚 ⋆ } [ ∇ θ log π θ ( 𝒛 | 𝒙 ) + ∇ θ log π θ ( 𝒚 | 𝒙 , 𝒛 ) ] | 𝒙 , 𝒚 ⋆ , 𝒛 ] \displaystyle\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\Big[\mathbbm{1}\{{\bm{y}}={\bm{y}}^{\star}\}\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\big]\Big|{\bm{x}},{\bm{y}}^{\star},{\bm{z}}\Big] = \displaystyle=\, ∑ 𝒚 ′ π θ ( 𝒚 ′ | 𝒙 , 𝒛 ) [ 𝟙 { 𝒚 ′ = 𝒚 ⋆ } [ ∇ θ log π θ ( 𝒛 | 𝒙 ) + ∇ θ log π θ ( 𝒚 ′ | 𝒙 , 𝒛 ) ] ] \displaystyle\sum_{{\bm{y}}^{\prime}}\pi_{\theta}({\bm{y}}^{\prime}|{\bm{x}},{\bm{z}})\Big[\mathbbm{1}\{{\bm{y}}^{\prime}={\bm{y}}^{\star}\}\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\prime}|{\bm{x}},{\bm{z}})\big]\Big] = \displaystyle=\, π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ​ [ ∇ θ ​ log ​ π θ ​ ( 𝒛 | 𝒙 ) + ∇ θ ​ log ​ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] \displaystyle\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big] = \displaystyle=\, G ^ VeriFree ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) \displaystyle\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}}) We denote 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ G ^ Verifier ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) | 𝒙 , 𝒚 ⋆ , 𝒛 ] \mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}})\big|{\bm{x}},{\bm{y}}^{\star},{\bm{z}}\big] as 𝔼 𝒚 | 𝒛 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) ] \mathbb{E}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}})\big] for brevity in the following, since 𝒙 {\bm{x}} and 𝒚 ⋆ {\bm{y}}^{\star} are already given and fixed. Thus, we have 𝔼 𝒚 | 𝒛 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) ] = G ^ VeriFree ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) \displaystyle\mathbb{E}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}})\big]=\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}}) (8)

The law of total variance states that for a random variable W W and conditioning variables S S , Var ​ ( W ) = 𝔼 S ​ [ Var ​ [ W | S ] ] + Var S ​ [ 𝔼 ⁡ [ W | S ] ] \text{Var}(W)=\mathbb{E}_{S}[\text{Var}[W|S]]+\text{Var}_{S}[\mathbb{E}[W|S]] . Let W = G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) W=\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}}) . Given 𝒙 {\bm{x}} and 𝒚 ⋆ {\bm{y}}^{\star} , the sources of randomness for G ^ Verifier \hat{G}_{\text{Verifier}} are 𝒛 {\bm{z}} and 𝒚 {\bm{y}} . Let S = 𝒛 S={\bm{z}} be the conditioning variables. The randomness in G ^ Verifier \hat{G}_{\text{Verifier}} given S S comes from 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) {\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}}) . Applying the law: Var 𝒛 , 𝒚 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) ] = 𝔼 𝒛 ​ [ Var 𝒚 | 𝒛 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) ] ] + Var 𝒛 ​ [ 𝔼 𝒚 | 𝒛 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) ] ] \displaystyle\text{Var}_{{\bm{z}},{\bm{y}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]=\mathbb{E}_{{\bm{z}}}\!\Big[\text{Var}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]\Big]\!+\!\text{Var}_{{\bm{z}}}\!\Big[\mathbb{E}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]\Big] The expectation 𝔼 𝒛 \mathbb{E}_{{\bm{z}}} is over 𝒛 ∼ π θ ( ⋅ | 𝒙 ) {\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}}) . The conditional variance Var 𝒚 | 𝒛 \text{Var}_{{\bm{y}}|{\bm{z}}} and expectation 𝔼 𝒚 | 𝒛 \mathbb{E}_{{\bm{y}}|{\bm{z}}} are over 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) {\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}}) for fixed 𝒙 , 𝒚 ⋆ , 𝒛 {\bm{x}},{\bm{y}}^{\star},{\bm{z}} .

Substituting the result from Eq. 8 into the law of total variance: Var 𝒛 , 𝒚 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) ] = 𝔼 𝒛 ​ [ Var 𝒚 | 𝒛 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) ] ] + Var 𝒛 ​ [ G ^ VeriFree ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) ] \displaystyle\text{Var}_{{\bm{z}},{\bm{y}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]=\mathbb{E}_{{\bm{z}}}\!\Big[\text{Var}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]\Big]+\text{Var}_{{\bm{z}}}\big[\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big] The second term, Var 𝒛 ​ [ G ^ VeriFree ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) ] \text{Var}_{{\bm{z}}}\big[\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big] , is the definition of the variance of the estimator G ^ VeriFree \hat{G}_{\text{{VeriFree}{}}} . The first term, 𝔼 𝒛 ​ [ Var 𝒚 | 𝒛 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) ] ] \mathbb{E}_{{\bm{z}}}\Big[\text{Var}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big]\Big] , is an expectation of a variance. Since variance is always non-negative, Var 𝒚 | 𝒛 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) ] ≥ 0 \text{Var}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big]\geq 0 . Therefore, its expectation is also non-negative: 𝔼 𝒛 ​ [ Var 𝒚 | 𝒛 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) ] ] ≥ 0 \displaystyle\mathbb{E}_{{\bm{z}}}\Big[\text{Var}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big]\Big]\geq 0 Thus, we have: Var 𝒛 , 𝒚 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) ] = ( a non-negative term ) + Var 𝒛 ​ [ G ^ VeriFree ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) ] \displaystyle\text{Var}_{{\bm{z}},{\bm{y}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]=(\text{a non-negative term})+\text{Var}_{{\bm{z}}}\big[\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big] This implies: Var 𝒛 , 𝒚 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) ] ≥ Var 𝒛 ​ [ G ^ VeriFree ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) ] \displaystyle\text{Var}_{{\bm{z}},{\bm{y}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star}\!\!,{\bm{z}},{\bm{y}})\big]\geq\text{Var}_{{\bm{z}}}\big[\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big] The variance of the policy gradient estimator G ^ VeriFree \hat{G}_{\text{{VeriFree}{}}} is less than or equal to that of G ^ Verifier \hat{G}_{\text{Verifier}} . This is an instance of Rao-Blackwellization, where analytically integrating out a source of randomness (the sampling of 𝒚 {\bm{y}} ) by using its conditional expectation reduces variance. ∎

## Appendix B Off-policy Gradient Estimators

In the main paper we provide an expression for the gradient estimator when the data is fully on-policy. VeriFree is also fully compatible with PPO-style gradient clipping for the case when data is reused to improve sample efficiency. In this case the gradient estimator is:

∇ θ J VeriFree ​ ( θ ) \displaystyle\nabla_{\theta}J_{\textrm{VeriFree}{}}(\theta) = 1 G ​ ∑ i = 1 G [ ∑ t = 1 | 𝒛 i | Clip ​ { π θ ​ ( 𝒛 i , t ∣ 𝒙 , 𝒛 i , < t ) π θ old ​ ( 𝒛 i , t ∣ 𝒙 , 𝒛 i , < t ) } ​ A i + ∑ t ′ = 1 | 𝒚 ⋆ | Clip ​ { π θ ​ ( 𝒚 t ⋆ ∣ 𝒙 , 𝒛 i ) π θ old ​ ( 𝒚 t ⋆ ∣ 𝒙 , 𝒛 i ) } ​ R i ] , \displaystyle=\frac{1}{G}\sum_{i=1}^{G}\Bigg[\sum_{t=1}^{\left|{\bm{z}}_{i}\right|}\text{Clip}\bigg\{\frac{\pi_{\theta}\left({\bm{z}}_{i,t}\mid{\bm{x}},{\bm{z}}_{i,<t}\right)}{\pi_{\theta_{\text{old}}}\left({\bm{z}}_{i,t}\mid{\bm{x}},{\bm{z}}_{i,<t}\right)}\bigg\}A_{i}+\sum_{t^{\prime}=1}^{\left|{\bm{y}}^{\star}\right|}\text{Clip}\bigg\{\frac{\pi_{\theta}\left({\bm{y}}^{\star}_{t}\mid{\bm{x}},{\bm{z}}_{i}\right)}{\pi_{\theta_{\text{old}}}\left({\bm{y}}^{\star}_{t}\mid{\bm{x}},{\bm{z}}_{i}\right)}\bigg\}R_{i}\Bigg], where π θ old \pi_{\theta_{\text{old}}} is the sampling policy, A i = π θ old ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 i ) − 1 G − 1 ​ ∑ j ≠ i π θ old ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 j ) A_{i}=\pi_{\theta_{\text{old}}}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{i})-\frac{1}{G-1}\sum_{j\neq i}\pi_{\theta_{\text{old}}}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{j}) , R i = π θ old ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 i ) R_{i}=\pi_{\theta_{\text{old}}}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{i}) , and Clip ​ { ⋅ } \text{Clip}\{\cdot\} denotes the PPO clipping operation.

## Appendix C Dataset Details

The category distribution in WebData (our training data) is visualized in Figure 7 .

## Appendix D Extended Empirical Results

In Tables 1 and 2 , we provide detailed benchmark results for MMLU-Pro and SuperGPQA using domain name abbreviations. The full nomenclature is as follows:

MMLU-Pro ( Table 1 ): CS (Computer Science), Math (Mathematics), Chem (Chemistry), Eng (Engineering), Law (Law), Bio (Biology), Health (Health), Phys (Physics), Bus (Business), Phil (Philosophy), Econ (Economics), Other (Other), Psy (Psychology), Hist (History).

SuperGPQA ( Table 2 ): Eng. (Engineering), Med. (Medicine), Sci. (Science), Phil. (Philosophy), M.S. (Military Science), Econ. (Economics), Mgmt. (Management), Socio. (Sociology), L/A (Literature and Arts), Hist. (History), Agron. (Agronomy), Law (Law), Edu (Education).

In Table 3 , we present an accuracy comparison across six math evaluation benchmarks and GPQA. Models trained with VeriFree demonstrate consistent and significant improvements over the base models, further validating the effectiveness of our approach.

## Appendix E Case Study

In this section, we provide some examples of reasoning traces produced by the model trained by our method on training questions. And we also provide an example with both our model’s and base model’s responses on the test dataset.

## Appendix F Broader Impact

Improving the general reasoning capabilities of large language models (LLMs) has the potential to significantly expand their applicability across domains such as education, scientific discovery, legal analysis, and decision support. By removing reliance on explicit verifiers—often expensive or domain-specific—our approach democratizes access to advanced reasoning systems, enabling broader use in resource-constrained settings.

However, improved reasoning can amplify both beneficial and harmful behaviors, depending on the intent and oversight of the user. As such, responsible usage, transparent reporting of model limitations, and value alignment remain critical.

We encourage future work that complements our method with safeguards for misuse prevention and performance guarantees under distributional shifts, particularly as general-purpose reasoning models become more widely integrated into real-world workflows.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .