Reinforcing General Reasoning without Verifiers β part 2/4
Cleaned-text code points [15350, 30676). Bibliography omitted here; source.md includes it.
There have been two main prior works that, although derived from a different perspective, arrive at related alternative gradient estimators: Tang et al. [40] (JLB), and Chen et al. [4] (LaTRO). β ΞΈ J Verifier \displaystyle\nabla_{\theta}J_{\text{Verifier}} = πΌ π , π β [ π { π β‘ π β } β β ΞΈ β log β Ο ΞΈ β ( π | π ) β reasoning term + π { π β‘ π β } β β ΞΈ β log β Ο ΞΈ β ( π | π , π ) β answer term ] \displaystyle=\mathbb{E}_{{\bm{z}},{\bm{y}}}\big[\hbox{\pagecolor{myred}$\mathds{1}_{\{{\bm{y}}\equiv{\bm{y}}^{\star}\}}$}\overbrace{\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})}^{\text{reasoning term}}+\hbox{\pagecolor{myblue}$\mathds{1}_{\{{\bm{y}}\equiv{\bm{y}}^{\star}\}}$}\overbrace{\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}^{\text{answer term}}\big] (R1-Zero) β ΞΈ J VeriFree \displaystyle\nabla_{\theta}J_{\text{{VeriFree} }} = πΌ π β [ Ο ΞΈ β ( π β | π , π ) β β ΞΈ β log β‘ Ο ΞΈ β ( π | π ) + Ο ΞΈ β ( π β | π , π ) β β ΞΈ β log β Ο ΞΈ β ( π β | π , π ) β reference answer term ] \displaystyle=\mathbb{E}_{\bm{z}}\big[\hbox{\pagecolor{myred}$\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})$}\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\hbox{\pagecolor{myblue}$\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})$}\overbrace{\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})}^{\text{reference answer term}}\big]\qquad (Ours) β ΞΈ J JLB \displaystyle\nabla_{\theta}J_{\text{JLB}} = πΌ π β [ log β‘ Ο ΞΈ β ( π β | π , π ) β β ΞΈ β log β Ο ΞΈ β ( π | π ) + 1 β
β ΞΈ β log β Ο ΞΈ β ( π β | π , π ) ] \displaystyle=\mathbb{E}_{\bm{z}}\big[\hbox{\pagecolor{myred}$\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})$}\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\hbox{\pagecolor{myblue}$1$}\cdot\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big] ( Tang et al. [40] ) β ΞΈ J LaTRO \displaystyle\nabla_{\theta}J_{\text{LaTRO}} = πΌ π β [ ( log β‘ Ο ΞΈ β ( π β | π , π ) β log β‘ Ο ΞΈ β ( π | π ) Ο ref β ( π | π ) ) β β ΞΈ β log β Ο ΞΈ β ( π | π ) + 1 β
β ΞΈ β log β Ο ΞΈ β ( π β | π , π ) ] \displaystyle=\mathbb{E}_{\bm{z}}\big[\hbox{\pagecolor{myred}$(\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})-\log\frac{\pi_{\theta}({\bm{z}}|{\bm{x}})}{\pi_{\text{ref}}({\bm{z}}|{\bm{x}})})$}\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\hbox{\pagecolor{myblue}$1$}\cdot\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big] ( Chen et al. [4] )
Both JLB and LaTRO treat the reasoning trace π {\bm{z}} as a latent variable and extend the standard supervised learning objective (log-likelihood) to optimize lower bounds on log ( E π βΌ Ο ΞΈ ( β
| π ) [ Ο ΞΈ ( π β | π , π ) ] ) \log(E_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\left[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\right]) and log ( E π βΌ Ο ref ( β
| π ) [ Ο ΞΈ ( π β | π , π ) ] ) \log(E_{{\bm{z}}\sim\pi_{\text{ref}}(\cdot|{\bm{x}})}\left[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\right]) , respectively. The primary difference is that JLB samples π {\bm{z}} from the learned policy Ο ΞΈ \pi_{\theta} , while LaTRO uses a fixed reference policy Ο ref \pi_{\text{ref}} . Despite originating from different perspectives, these approaches arrive at alternative gradient estimators that can be used similarly to ours, as shown in the above comparisons. However, as reported in Tang et al. [40] , these verifier-free, variational-inference-based methods consistently underperform the standard verifier-based R1-Zero approach. In contrast, our formulation matches or outperforms the verifier-based baseline.
One possible explanation is that our method exactly recovers the original verifier-based objective under the single-correct-answer assumption, whereas JLB and LaTRO optimize subtly different objectives. For example, JLB effectively uses a reward of R = log β‘ Ο ΞΈ β ( π β | π , π ) R=\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}) , as highlighted in the gradient expressions above. Another distinction lies in the weighting of the βreference answer termβ β ΞΈ β log β Ο ΞΈ β ( π β | π , π ) \nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}) . In our method, this term is weighted by the probability Ο ΞΈ β ( π β | π , π ) \pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}) , which is the likelihood of the reference answer given the sampled reasoning trace. In contrast, both JLB and LaTRO use a fixed weight of 1, thereby increasing the probability of π β {\bm{y}}^{\star} regardless of the quality of the reasoning trace π {\bm{z}} . We hypothesize that this behavior could promote poor reasoning. For instance, suppose the model generates the reasoning trace β ... minus 2 apples, finally resulting in a total of 7 apples β when the correct answer is β 6 β. The JLB and LaTRO objectives would still push the model to output β 6 β from that flawed trace, reinforcing a mismatch between reasoning and answer. Our method avoids this by down-weighting contributions from low-quality traces.
### 2.4 How to handle the tokenization at patching point?
A critical consideration when extracting reasoning traces π {\bm{z}} from model responses ( π , π ) ({\bm{z}},{\bm{y}}) and subsequently replacing π {\bm{y}} with π β {\bm{y}}^{\star} stems from the fact that LLMs operate on token sequences, not raw text strings. While human-readable outputs (e.g., β ...<answer> \\boxed{...} </answer> β as in Template 1 ) suggest splitting reasoning traces π {\bm{z}} at specific text patterns like β <answer> β, such text-based splitting strategy may cause tokenization inconsistencies. For example, the β > β character might be tokenized differently depending on its surrounding context in π {\bm{y}} versus π β {\bm{y}}^{\star} . While one potential solution is to introduce special tokens to enforce consistent tokenization boundaries, these novel tokens could harm model performance due to their absence from the base modelβs pretraining vocabulary.
Instead, we resolve this by defining the end of π {\bm{z}} at the token corresponding to <answer β (i.e., without β > β), leveraging the fact that the pattern β r> β does not appear in standard tokenizer vocabularies. This ensures consistent token-space alignment between sampling and optimization in RL, avoiding instability due to off-policy mismatches. Notably, this approach is operationally equivalent to setting β <answer β (instead of β <answer> β) as the stop word during sampling, a mechanism natively supported by modern inference engines like vLLM [ 16 ] . In this case, we can sample reasoning traces π {\bm{z}} directly, instead of first generating the full response ( π {\bm{z}} , π {\bm{y}} ) and then extracting π {\bm{z}} post hoc.
## 3 Experiments
### 3.1 Setup
Training. Following the βzeroβ setting widely adopted in recent work [ 10 , 14 , 24 , 27 ] , we directly fine-tune the base LLM, skipping the intermediate stage of supervised fine-tuning (SFT). We implement our RL training pipeline using Oat [ 22 ] by instantiating their base modules and incorporate our algorithmic changes. Our experiments are conducted using Qwen3 [ 41 ] base models across multiple scales, including 1.7B, 4B, and 8B parameters. We adopt the prompt template shown in Template 1 . We do not employ KL regularization losses or KL penalties in rewards, as recent studies suggest that removing KL terms does not have a significant impact [ 24 , 14 ] . As a result, our method does not require maintaining a reference model in memory.
For the 1.7B and 4B models, we conduct fine-tuning for approximately 4,000 policy gradient steps; for the 8B models, we fine-tune for around 3,000 policy gradient steps. During each step, the policy model (i.e., the LLM) generates 8 responses for each question (i.e., group_size=8 ), with 16 questions processed per step. We use the sampling configurations temperature=1.0 , top_p=1 , and max_tokens=3000 for the rollout process. The responses are then parsed into reasoning traces and model-predicted answers. We replace the model-predicted answers with the reference answers from the training dataset. Subsequently, a single forward pass is executed to compute the conditional probability of the reference answer, conditioned on all preceding tokens including the prompt and the reasoning trace. This procedure introduces only a minimal additional computational cost, as the forward pass of the LLM does not require autoregressive decoding and does not require storing intermediate states for backpropagation. All collected samples from each step are used for one optimization step. The training is conducted on a single node with 8 Γ \times H100 GPUs.
Dataset. To support general reasoning, we begin with the dataset curated by Ma et al. [27] , sourced from WebInstruct [ 49 ] . To improve data quality and reliability and reduce size, we retain samples with answers that consist of fewer than seven tokens, and use Qwen2.5-72B-Instruct [ 43 ] to filter out low-quality and noisy data. This process results in approximately 61,000 data samples spanning diverse domains, which we refer to as WebData . The category distribution is visualized in Figure 7 .
Evaluation. In line with prior work [ 27 ] , we employ multiple-choice questions for evaluation to facilitate verification. To assess general reasoning abilities, we utilize the following benchmarks: MMLU-Pro [ 42 ] , a challenging multi-task understanding benchmark designed to evaluate the capabilities of LLMs across various domains; SuperGPQA [ 6 ] , a large-scale benchmark consisting of graduate-level reasoning questions spanning 285 diverse disciplines; and GPQA [ 35 ] , which focuses on graduate-level question-answering and is designed to resist shallow pattern-matching and memorization. While our primary focus is not on enhancing mathematical abilities, we also evaluate math reasoning using a suite of standard math reasoning benchmarks. This suite includes MATH-500 [ 13 ] , OlympiadBench [ 11 ] , Minerva Math [ 18 ] , GSM8K [ 5 ] , AMC and AIME24 [ 19 ] . We utilize Math-Verify 2 2 2 https://github.com/huggingface/Math-Verify to check for answer equivalence. Except for AIME24, where we employ a temperature=1.0 and repeat each question 32 times, all other benchmarks are evaluated using temperature=0.0 . We use max_tokens=8192 for all evaluations.
Baselines. Our primary baseline, denoted Verifier , is a verifier-based approach using the verifier from Ma et al. [27] . The verifier is initialized from Qwen2.5-Math-1.5B [ 44 ] and fine-tuned on data generated by Gemini 2.0 Flash to assess equivalence between the reference and predicted answers, conditioned on the question. We apply Dr. GRPO [ 24 ] as the optimization algorithm for the baseline, ensuring that all other settings are consistent with our approach. Following Ma et al. [27] , the reward definition incorporates additional factors beyond verifier correctness, including format compliance and the length of generated answers. If the format is incorrect (e.g., missing \\boxed{} in the model response), a negative reward of -0.5 is applied. Moreover, a length penalty of -0.05 Γ \times min(10, abs(length_of_correct_answer - length_of_answer)) is added.
We also report the results for the base and the instruct models of Qwen-3-1.7B/4B/8B [ 41 ] and Qwen2.5-7B [ 43 ] , as well as the checkpoints released by Qwen2.5-7B-SimpleRL-Zoo [ 50 ] , Qwen2.5-Math-7B-Oat-Zero [ 24 ] , and General-Reasoner-7B [ 27 ] . 3 3 3 Qwen2.5 and Qwen3 use different naming conventions in their official release. For base models: Qwen2.5 has no suffix (e.g., Qwen2.5-7B), whereas Qwen3 adds β-Baseβ (e.g., Qwen3-8B-Base). For instruct models: Qwen2.5 uses β-Instructβ (e.g., Qwen2.5-7B-Instruct), while Qwen3 omits the suffix (e.g., Qwen3-8B). We follow these conventions consistently in the paper. Notably, Qwen3 integrates both a thinking mode (for complex, multi-step reasoning) and a non-thinking mode (for rapid, context-driven responses) within a unified framework. We report results of both modes on Qwen3 instruct models.
### 3.2 Main Results
VeriFree improves general reasoning capabilities. We begin by evaluating the effectiveness of VeriFree in enhancing the general reasoning capabilities of LLMs using the MMLU-Pro and SuperGPQA benchmarks. Table 1 presents a detailed comparison across model scales and domains in the MMLU-Pro benchmark. Starting from base models, applying RL with VeriFree yields substantial gains in average accuracyβranging from 12% to 40%βdemonstrating that VeriFree effectively fine-tunes LLMs to improve general reasoning performance.
Similar improvements are observed on the SuperGPQA benchmark, as shown in Table 2 , where VeriFree consistently enhances the performance of base models by a significant margin. Notably, VeriFree achieves performance comparable to, or even surpassing, that of the instruct model in thinking mode and the Verifier baseline (i.e., the RL-tuned model learned with an additional model-based verifier)βwithout relying on any explicit verification signals.
In addition to accuracy gains, we also observe an increase in response length after tuned by VeriFree, suggesting that the model explores longer reasoning traces to arrive at more accurate answersβa behavior reminiscent of DeepSeek-R1-Zero [ 10 ] . Results on the GPQA benchmark are provided in Appendix D due to space constraints.
VeriFree leads to better learning efficiency. We compare VeriFree with the baseline that learns from a model-based verifier reward (i.e., Verifier). As shown in Figure 4 (Left), VeriFree consistently outperforms the baseline, achieving higher accuracy with fewer training steps. We attribute this improved learning efficiency to reduced gradient variance, enabled by VeriFreeβs continuous reward signals and the RLOO objective. While both approaches optimize the same reward signal in expectation, VeriFree provides more stable and informative policy gradients, which accelerate convergence and leads to better final performance.
Model confidence is a good reasoning capability proxy. Our analysis based on Qwen3-8B base model reveals a strong positive correlation ( Ο = 0.82 \rho=0.82 ) between MMLU-Pro accuracy and the average model confidence Ο ΞΈ β ( π β | π , π ) \pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}) during training ( Figure 4 , Right). This empirically demonstrates that the modelβs self-estimated confidence in the correct answer, i.e., Ο ΞΈ β ( π β | π , π ) \pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}}) , serves as an effective metric for quantifying emergent reasoning capabilities in LLMs.
VeriFree learns transferable reasoning skills. To evaluate the transferability of reasoning acquired through VeriFree, we train a model on a dataset with all math-related examples removed, and assess its performance on both general and math-specific benchmarks. As shown in Figure 5 , VeriFree not only improves reasoning performance on general tasks, as expected, but also demonstrates strong transfer to math benchmarksβdespite the absence of math supervision during training. This highlights VeriFreeβs ability to induce general reasoning capabilities that extend across domains.
### 3.3 Ablation Study
To systematically evaluate method components and offer a comprehensive understanding of VeriFree, we conduct ablation studies based on Qwen3-1.7B base models as follows.