Reinforcing General Reasoning without Verifiers — part 1/4

Cleaned-text code points [0, 15350). Bibliography omitted here; source.md includes it.
# Reinforcing General Reasoning without Verifiers

###### Abstract

The recent paradigm shift towards training large language models (LLMs) using DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has led to impressive advancements in code and mathematical reasoning. However, this methodology is limited to tasks where rule-based answer verification is possible and does not naturally extend to real-world domains such as chemistry, healthcare, engineering, law, biology, business, and economics. Current practical workarounds use an additional LLM as a model-based verifier; however, this introduces issues such as reliance on a strong verifier LLM, susceptibility to reward hacking, and the practical burden of maintaining the verifier model in memory during training. To address this and extend DeepSeek-R1-Zero-style training to general reasoning domains, we propose a verifier-free method ( VeriFree ) that bypasses answer verification and instead uses RL to directly maximize the probability of generating the reference answer. We compare VeriFree with verifier-based methods and demonstrate that, in addition to its significant practical benefits and reduced compute requirements, VeriFree matches and even surpasses verifier-based methods on extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related benchmarks. Moreover, we provide insights into this method from multiple perspectives: as an elegant integration of training both the policy and implicit verifier in a unified model, and as a variational optimization approach. Code is available at https://github.com/sail-sg/VeriFree .

## 1 Introduction

DeepSeek-R1-Zero [ 10 ] recently demonstrated that training large language models (LLMs) using reinforcement learning (RL) with verifiable rewards can be extremely effective in improving reasoning capabilities. In this RL with verifiable rewards (RLVR) framework [ 17 ] , the LLM generates a reasoning trace (i.e., chain of thoughts, CoT) followed by a final answer. A rule-based program then extracts and evaluates the final answer, assigning a reward of 1 to the response if the final answer is correct and 0 otherwise. The model is trained with RL using GRPO [ 37 ] —a simplified variant of PPO [ 36 ] .

The simplicity of this approach, coupled with impressive performance improvements in mathematical reasoning tasks, has sparked a wave of follow-up works in this paradigm of RL with rule-based verifiable rewards [ 24 , 26 , 45 ] , which we will refer to as the R1-Zero-style training in the following. However, these methods remain limited to domains such as mathematics and code, where rule-based verification is feasible. Reasoning is critical far beyond math and coding; however, the difficulty of answer verification in general reasoning tasks poses a major obstacle to applying this training paradigm to broader domains. To address this limitation, we investigate how to extend R1-Zero-style training to tasks where rule-based answer verification is not possible.

A natural extension, as explored in recent general reasoning works [ 38 , 27 ] , is to introduce a specialized LLM as a verifier, similar to the reward model used in RL from human feedback (RLHF) [ 51 , 30 ] . In these methods, the model-based verifier is queried to determine whether the generated answer is equivalent to the reference answer. Although this approach bypasses the need for rule-based evaluation, it introduces several potential drawbacks (as in standard RLHF): it depends on the availability of a strong verifier LLM, it converts the R1-Zero-style paradigm into optimizing a model-based reward, which makes it vulnerable to reward hacking [ 7 ] , and it adds significant computational overhead by requiring an additional model to be held in memory and queried during training.

In this work, we propose an alternative: a verifier-free approach that preserves the benefits of the RL paradigm while eliminating the reliance on explicit verification, either performed by rules or by models. Our method proceeds as follows. Given a question, we only generate the reasoning trace and concatenate it with the reference answer from the dataset. We then evaluate the likelihood of the reference answer conditioned on the question and the generated reasoning trace. This likelihood serves both as a reward signal for policy gradients on the reasoning trace and as a weighting term for supervised training of the reference answer. We term our method VeriFree since it does not rely on rule- or model-based verifiers, and give an illustration in Figure 2 .

This approach has several appealing properties. First, when there is a unique correct answer string, our method is equivalent in expectation to the objective in RLVR, but with lower variance, which can be viewed as a form of reward shaping [ 28 , 34 ] . Even when multiple valid answers exist, we show empirically that using just one as a reference provides a sufficient learning signal to elicit strong reasoning behavior. Additionally, this framework can be viewed through a variational lens as a neat way of optimizing over latent reasoning traces.

To make this work in practice, we identify and address several subtle challenges, including effective variance reduction and precise handling of tokenization at the reasoning-answer patching point. We conduct comprehensive ablations to understand the impact of each design choice. We benchmark our method across a diverse set of general reasoning tasks, and the results are striking: as shown in Figure 1 , VeriFree not only matches but often outperforms verifier-based alternatives, while being simpler, faster, less memory-intensive, and more robust.

## 2 Methodology

### 2.1 Preliminaries: Verifier-Based Reinforcement Learning

In RL applied to LLMs, the language model is treated as a policy π θ \pi_{\theta} that generates an output 𝒐 {\bm{o}} autoregressively in response to an input question 𝒙 {\bm{x}} . The goal is typically to optimize π θ \pi_{\theta} to maximize a given reward function R ⁡ ( 𝒙 , 𝒐 ) R({\bm{x}},{\bm{o}}) : θ ∈ arg ⁡ max 𝜃 𝔼 𝒐 ∼ π θ ( ⋅ | 𝒙 ) [ R ( 𝒙 , 𝒐 ) ] . \theta\in\underset{\theta}{\arg\max}\,\mathbb{E}_{{\bm{o}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\left[R({\bm{x}},{\bm{o}})\right]. (1) In R1-Zero-style RL, the reward is computed by first parsing the response 𝒐 {\bm{o}} into a reasoning trace 𝒛 {\bm{z}} and a final answer 𝒚 {\bm{y}} . A verifier then checks 𝒚 {\bm{y}} against the ground-truth reference answer 𝒚 ⋆ {\bm{y}}^{\star} and assigns a binary reward based on correctness, namely R Verifier ( 𝒚 ; 𝒚 ⋆ ) = 𝟙 { 𝒚 ≡ 𝒚 ⋆ } R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})=\mathds{1}_{\{{\bm{y}}\equiv{\bm{y}}^{\star}\}} . 1 1 1 We use ‘ ≡ \equiv ’ to denote semantic equivalence, where multiple expressions can be judged correct. For example, if 𝒚 ⋆ = {\bm{y}}^{\star}= ‘ 8/5 ’, then ‘ 8/5 ’, ‘ 1.6 ’, and ‘ \frac{8}{5} ’ are all considered correct. Decomposing the model output as 𝒐 = ( 𝒛 , 𝒚 ) {\bm{o}}=({\bm{z}},{\bm{y}}) , we can rewrite the objective in Equation 1 as: J Verifier ( θ ; 𝒙 , 𝒚 ⋆ ) = 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ R Verifier ( 𝒚 ; 𝒚 ⋆ ) ] , \displaystyle J_{\text{Verifier}}(\theta;{\bm{x}},{\bm{y}}^{\star})=\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}{\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}[R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})]}, (2) which separates the sampling of the reasoning trace and the final answer. To maximize this objective, the model π θ \pi_{\theta} is typically updated using the policy gradient estimator [ 39 ] : ∇ θ J Verifier  ( θ , 𝒙 , 𝒚 ⋆ ) = 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 )  𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 )  [ R Verifier  ( 𝒚 , 𝒚 ⋆ )  [ ∇ θ  log  π θ  ( 𝒛 | 𝒙 ) + ∇ θ  log  π θ  ( 𝒚 | 𝒙 , 𝒛 ) ] ] . \small\nabla_{\theta}J_{\text{Verifier}}(\theta;{\bm{x}},{\bm{y}}^{\star})=\underset{{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}}{\mathbb{E}}~\underset{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}{\mathbb{E}}\Big[R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\big]\Big]. (3) However, this approach requires evaluating answer correctness via R Verifier ( 𝒚 ; 𝒚 ⋆ ) = 𝟙 { 𝒚 ≡ 𝒚 ⋆ } R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})=\mathds{1}_{\{{\bm{y}}\equiv{\bm{y}}^{\star}\}} , which is often nontrivial. While in domains such as math and code, this evaluation is feasible via rules [ 10 , 24 ] or test cases [ 8 ] , accurate verification in general reasoning tasks is substantially more difficult. As a result, recent advances in R1-Zero-style training have largely been restricted to verifiable domains, leaving reasoning tasks in general domains underexplored. In light of this, we present an alternative verifier-free approach which naturally extends this training paradigm to broader reasoning domains.

### 2.2 VeriFree Policy Optimization

We begin with the standard objective in Equation 2 and show that, in the case of a single correct answer, we can derive an equivalent objective that does not require a verifier. Moreover, we compare the gradient estimators of this new objective and the verifier-based objective ( Equation 3 ), and demonstrate that our verifier-free gradient estimator has the additional benefit of lower variance.

Starting from Equation 2 and assuming a unique correct answer such that R Verifier ( 𝒚 ; 𝒚 ∗ ) = 𝟙 { 𝒚 = 𝒚 ⋆ } R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{*})=\mathds{1}_{\{{\bm{y}}={\bm{y}}^{\star}\}} (i.e., exact match rather than semantic equivalence 𝟙 { 𝒚 ≡ 𝒚 ⋆ } \mathds{1}_{\{{\bm{y}}\equiv{\bm{y}}^{\star}\}} ), the VeriFree objective is derived as follows: J Verifier  ( θ , 𝒙 , 𝒚 ⋆ ) \displaystyle J_{\text{Verifier}}(\theta;{\bm{x}},{\bm{y}}^{\star}) = 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ 𝟙 { 𝒚 = 𝒚 ⋆ } ⏞ R Verifier ] ] \displaystyle=\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[{\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}[\,\overbrace{\mathds{1}_{\{{\bm{y}}={\bm{y}}^{\star}\}}}^{R_{\text{Verifier}}}\,]}\Big] = 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ ∑ 𝒚 π θ ( 𝒚 | 𝒙 , 𝒛 ) 𝟙 { 𝒚 = 𝒚 ⋆ } ] \displaystyle=\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[\sum_{{\bm{y}}}\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\mathds{1}_{\{{\bm{y}}={\bm{y}}^{\star}\}}\Big] = 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ π θ  ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ⏟ R VeriFree ] ≜ J VeriFree ( θ ; 𝒙 , 𝒚 ⋆ ) . \displaystyle=\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[{\qquad\>\,\underbrace{\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})}_{R_{\text{{VeriFree} }}}\qquad\>\,}\Big]\triangleq J_{\text{{VeriFree} }}(\theta;{\bm{x}},{\bm{y}}^{\star}). (4) This can be interpreted as follows: if only one answer 𝒚 ⋆ {\bm{y}}^{\star} is correct and receives a reward of 1 (while all others receive 0), then the expected reward given a reasoning trace 𝒛 {\bm{z}} can be computed directly as the probability assigned to 𝒚 ⋆ {\bm{y}}^{\star} , effectively marginalizing out 𝒚 {\bm{y}} . The corresponding gradient estimator is given by (see Section A.1 for a full derivation): ∇ θ J VeriFree ( θ ; 𝒙 , 𝒚 ⋆ ) = 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ R VeriFree ( 𝒙 , 𝒚 ⋆ , 𝒛 ) [ ∇ θ  log  π θ  ( 𝒛 | 𝒙 ) ⏟ reasoning term + ∇ θ  log  π θ  ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ⏟ reference answer term ] ] . \footnotesize\nabla_{\theta}J_{\text{{VeriFree} }}(\theta;{\bm{x}},{\bm{y}}^{\star})=\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\bigg[R_{\text{{VeriFree} }}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big[\underbrace{\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})}_{\text{reasoning term}}+\underbrace{\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})}_{\text{reference answer term}}\big]\bigg]. (5) Both the objective and its gradient estimator ( Equations 4 and 5 ) are equivalent in expectation to their verifier-based counterparts ( Equations 2 and 3 ). Intuitively, the “reasoning term” in Equation 5 can be interpreted as a policy gradient where the reward for a reasoning trace 𝒛 {\bm{z}} is the probability that the policy will generate the correct answer 𝒚 ⋆ {\bm{y}}^{\star} given 𝒛 {\bm{z}} , while “reference answer term” can be viewed as a reward-weighted supervised learning term for 𝒚 ⋆ {\bm{y}}^{\star} given 𝒛 {\bm{z}} . We will further elaborate on this interpretation in Section 2.3 . In addition to bypassing the need for a verifier, our VeriFree gradient estimator also benefits from reduced variance:

###### Theorem 1 .

( Variance Reduction ) Let G ^ Verifier  ( 𝒙 , 𝒚 ⋆ , 𝒛 ) \hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}}) and G ^ VeriFree  ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) \hat{G}_{\text{{VeriFree} }}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}}) denote the single-sample Monte Carlo estimators of ∇ θ J Verifier \nabla_{\theta}J_{\text{Verifier}} and ∇ θ J VeriFree \nabla_{\theta}J_{\text{{VeriFree} }} given 𝒙 {\bm{x}} and 𝒚 ⋆ {\bm{y}}^{\star} , respectively. Then we have Var 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ G ^ VeriFree ( 𝒙 , 𝒚 ⋆ , 𝒛 ) ] ≤ Var 𝒛 ∼ π θ ( ⋅ | 𝒙 ) , 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ G ^ Verifier ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) ] . \text{Var}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\big[\hat{G}_{\text{{VeriFree} }}({\bm{x}},{\bm{y}}^{\star},{\bm{z}})\big]\leq\text{Var}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}}),{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}})\big]. (6)

This reduction in variance arises from Rao-Blackwellization [ 2 ] . For intuition, the variance in the Monte Carlo estimate of ∇ θ J Verifier \nabla_{\theta}J_{\text{Verifier}} stems from the randomness in sampling 𝒛 ∼ π θ ( ⋅ | 𝒙 ) , 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) {\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}}),{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}}) , while for estimating ∇ θ J VeriFree \nabla_{\theta}J_{\text{{VeriFree} }} we analytically marginalizes out 𝒚 {\bm{y}} , thereby removing this source of randomness. We provide a full proof in Section A.2 .

Our gradient estimator ∇ θ J VeriFree  ( θ , 𝒙 , 𝒚 ⋆ ) \nabla_{\theta}J_{\text{{VeriFree} }}(\theta;{\bm{x}},{\bm{y}}^{\star}) is fully compatible with other variance reduction techniques, including RLOO [ 1 ] , GRPO [ 37 ] reward normalizations, and the PPO [ 36 ] clipping operation. As such, we sample multiple responses for each prompt and apply the RLOO baseline to the reasoning term in Equation 5 . We also adopt the corrected response-length normalization from Liu et al. [24] . The final on-policy gradient estimator is as follows: ∇ θ J VeriFree  ( θ ) = 1 G  ∑ i = 1 G [ A i ⋅ ∇ θ  log  π θ  ( 𝒛 i | 𝒙 ) + R i ⋅ ∇ θ  log  π θ  ( 𝒚 ⋆ | 𝒙 , 𝒛 i ) ] , \nabla_{\theta}J_{\text{{VeriFree} }}(\theta)=\frac{1}{G}\sum_{i=1}^{G}\left[A_{i}\cdot\nabla_{\theta}\log\pi_{\theta}({\bm{z}}_{i}|{\bm{x}})+R_{i}\cdot\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{i})\right], (7) where 𝒛 i ∼ π θ ( ⋅ | 𝒙 ) {\bm{z}}_{i}\sim\pi_{\theta}(\cdot|{\bm{x}}) , R i = π θ  ( 𝒚 ⋆ | 𝒙 , 𝒛 i ) R_{i}=\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{i}) , and A i = π θ  ( 𝒚 ⋆ | 𝒙 , 𝒛 i ) − 1 G − 1  ∑ j ≠ i π θ  ( 𝒚 ⋆ | 𝒙 , 𝒛 j ) A_{i}=\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{i})-\frac{1}{G-1}\sum_{j\neq i}\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}_{j}) . We also provide the PPO-based off-policy variant in Appendix B .

### 2.3 Comparison to Existing Approaches