Reinforcing General Reasoning without Verifiers — part 3/4

Cleaned-text code points [30676, 46578). Bibliography omitted here; source.md includes it.

Effects of extraction strategy for reasoning trace z {\bm{z}} . Our method requires precise separation between reasoning path 𝒛 {\bm{z}} and answer 𝒚 {\bm{y}} to enable answer replacement. While human-readable splits using <answer> seem intuitive, we instead define 𝒛 {\bm{z}} to end at “ <answer ” (omitting “ > ”), ensuring consistent tokenization boundaries (see Section 2.4 ). We compare with a variant using text-based splitting (denoted as “VeriFree w/o token split”) on Qwen3-1.7B via MMLU-Pro ( Figure 6 ). Our tokenization-aware approach achieves superior convergence, while the variant suffers optimization instability due to effectively introducing off-policy data.

Effects of RLOO. As observed in Figure 6 (Left), removing RLOO leads to a consistent drop in performance throughout training, with final accuracy more than 3 % 3\% lower than that of the full method. This highlights the importance of RLOO in stabilizing learning and guiding the model toward better generalization. Without RLOO, the model converges prematurely and fails to reach the same peak accuracy.

Effects of equivalence class. As mentioned in Section 2.1 , verifier-based RL typically assesses answer correctness as rewards. Correct answers within a specific class often form an equivalence class. Our method, however, utilizes model confidence by focusing on a single reference answer for a given question and the model’s reasoning trace. To explore the potential advantages of integrating an “equivalence class” into our approach, we conducted ablation studies as follows. We employed a model fine-tuned on the MATH-12k dataset [ 12 , 20 ] from Qwen3-8B base model through Dr. GRPO [ 24 ] with rule-based verification to sample answers on MATH-12k, subsequently verifying answer correctness using Math-Verify. This approach enabled us to create an extended dataset with a set of equivalent correct answers for each question. We then fine-tuned Qwen3-1.7B base models using our method on both the original and the extended MATH-12k datasets incorporating equivalence classes. These models are evaluated on GSM8K, MATH-500, Minerva Math, and OlympiadBench to assess the impact of including equivalence classes. The results, shown in Figure 6 (Right), indicate that considering equivalence classes in our method offers slight performance improvements, aligning with our expectations. This highlights a minor limitation of our current formulation and motivates future work on algorithms that can better leverage answer equivalence.

## 4 Related Works

DeepSeek-R1-Zero-like reinforcement learning. DeepSeek-R1-Zero [ 10 ] , Tülu [ 17 ] and OpenAI’s o1 [ 29 ] recently demonstrated that applying RL to learn directly from binary verification-based rewards can be extremely powerful in enhancing the reasoning capabilities of base LLMs. Since then several works have reproduced R1-Zero-like training on smaller scales [ 50 , 31 , 23 , 14 , 26 , 25 ] . The aforementioned works all focus on math and coding, domains where verifiable rewards are readily available. By contrast, our work aims to extend the R1-Zero-like training paradigm to general domains where verifiable rewards are not available.

Reasoning beyond verifiable domains. Previous work on reasoning without access to verifiable rewards has been based around employing an additional LLM to act as a proxy verifier or reward model. NaturalReasoning [ 47 ] introduces a large, multi-domain dataset and presents baselines trained using RFT [ 48 ] and DPO [ 33 ] , leveraging a second LLM as a reward model, while Su et al. [38] and General-Reasoner [ 27 ] similarly incorporate a separate LLM to serve as a verifier. X-Reasoner [ 21 ] also investigates general reasoning but circumvents the lack of rule-based verification by dropping the R1-Zero-style training paradigm, instead training via SFT [ 9 ] on responses sampled from more capable models.

Self-improving language models. Several works have explored training LLMs using signals based on the model’s own outputs. Yuan et al. [46] propose to prompt the model to judge and rank different responses and select the best and worst for preference learning. Chen et al. [3] leverage the DPO implicit rewards for more efficient and robust self-alignment via iterative DPO [ 33 ] . Zuo et al. [52] use majority voting to construct self-labeled rewards for RL to further improve well-trained models during test time, which can be understood as a form of sharpening [ 15 ] . Another line of research [ 32 , 4 , 40 ] approaches LLM reasoning from the direction of variational optimization, treating the reasoning trace as a latent variable. Despite starting from a different viewpoint our method has interesting and close connections to this perspective which we discuss in detail in Section 2.3 .

## 5 Conclusions

In this paper, we rethink reinforcement learning with verifiable rewards (RLVR) for LLMs from a novel perspective. By leveraging the gradient equivalence under the unique answer assumption, we derive a new optimization objective that eliminates the need for explicit verification. Our proposed method, VeriFree, is particularly well-suited for general reasoning tasks, where rule-based verifiers are infeasible and model-based verifiers are both expensive and vulnerable to reward hacking. Through extensive experiments and ablations, we demonstrate the effectiveness and robustness of VeriFree on a wide range of general reasoning benchmarks. We hope our work offers a fresh viewpoint for the LLM RL community and provides a practical approach for building future general-purpose reasoners.

## Appendix

## Table of Contents

## Appendix A Theoretical Analysis

### A.1 Derivation of Gradient Estimators

Here we provide derivations for Equation 5 —the gradient estimator of VeriFree. We also include the corresponding derivation for the standard verifier-based gradient estimator for completeness.

###### Proof.

The gradient estimator for J Verifier J_{\text{Verifier}} is derived as follows: ∇ θ J Verifier ​ ( θ , 𝒙 , 𝒚 ⋆ ) \displaystyle\nabla_{\theta}J_{\text{Verifier}}(\theta;{\bm{x}},{\bm{y}}^{\star}) = \displaystyle=\, ∇ θ 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ R Verifier ( 𝒚 ; 𝒚 ⋆ ) ] ] \displaystyle\nabla_{\theta}\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\big[R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})\big]\Big] = \displaystyle=\, ∇ θ ∑ 𝒛 , 𝒚 R Verifier ( 𝒚 ; 𝒚 ⋆ ) π θ ( 𝒚 | 𝒙 , 𝒛 ) π θ ( 𝒛 | 𝒙 ) \displaystyle\nabla_{\theta}\sum_{{\bm{z}},{\bm{y}}}R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}}) = \displaystyle=\, ∑ 𝒛 , 𝒚 R Verifier ​ ( 𝒚 , 𝒚 ⋆ ) ​ [ π θ ​ ( 𝒚 | 𝒙 , 𝒛 ) ​ ∇ θ π θ ​ ( 𝒛 | 𝒙 ) + π θ ​ ( 𝒛 | 𝒙 ) ​ ∇ θ π θ ​ ( 𝒚 | 𝒙 , 𝒛 ) ] \displaystyle\sum_{{\bm{z}},{\bm{y}}}R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})\big[\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\nabla_{\theta}\pi_{\theta}({\bm{z}}|{\bm{x}})+\pi_{\theta}({\bm{z}}|{\bm{x}})\nabla_{\theta}\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\big] = \displaystyle=\, ∑ 𝒛 , 𝒚 R Verifier ​ ( 𝒚 , 𝒚 ⋆ ) ​ [ π θ ​ ( 𝒚 | 𝒙 , 𝒛 ) ​ π θ ​ ( 𝒛 | 𝒙 ) ​ ∇ θ ​ log ⁡ π θ ​ ( 𝒛 | 𝒙 ) + π θ ​ ( 𝒛 | 𝒙 ) ​ π θ ​ ( 𝒚 | 𝒙 , 𝒛 ) ​ ∇ θ ​ log ⁡ π θ ​ ( 𝒚 | 𝒙 , 𝒛 ) ] \displaystyle\sum_{{\bm{z}},{\bm{y}}}R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})\big[\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}})\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\pi_{\theta}({\bm{z}}|{\bm{x}})\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\big] = \displaystyle=\, 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ R Verifier ( 𝒚 ; 𝒚 ⋆ ) [ ∇ θ log π θ ( 𝒛 | 𝒙 ) + ∇ θ log π θ ( 𝒚 | 𝒙 , 𝒛 ) ] ] ] \displaystyle\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\bigg[\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\Big[R_{\text{Verifier}}({\bm{y}};{\bm{y}}^{\star})\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\big]\Big]\bigg] The gradient estimator for J VeriFree J_{\text{{VeriFree} }} is derived as follows: ∇ θ J VeriFree ​ ( θ , 𝒙 , 𝒚 ⋆ ) \displaystyle\nabla_{\theta}J_{\text{{VeriFree}{}}}(\theta;{\bm{x}},{\bm{y}}^{\star}) = \displaystyle=\, ∇ θ 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ R VeriFree ( 𝒛 ; 𝒙 , 𝒚 ⋆ ) ] \displaystyle\nabla_{\theta}\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}[R_{\text{{VeriFree}{}}}({\bm{z}};{\bm{x}},{\bm{y}}^{\star})] = \displaystyle=\, ∇ θ 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] \displaystyle\nabla_{\theta}\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})] = \displaystyle=\, ∇ θ ∑ 𝒛 π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) π θ ( 𝒛 | 𝒙 ) \displaystyle\nabla_{\theta}\sum_{{\bm{z}}}\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}}) = \displaystyle=\, ∑ 𝒛 [ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ​ ∇ θ π θ ​ ( 𝒛 | 𝒙 ) + π θ ​ ( 𝒛 | 𝒙 ) ​ ∇ θ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] \displaystyle\sum_{{\bm{z}}}\big[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\nabla_{\theta}\pi_{\theta}({\bm{z}}|{\bm{x}})+\pi_{\theta}({\bm{z}}|{\bm{x}})\nabla_{\theta}\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big] = \displaystyle=\, ∑ 𝒛 [ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ​ π θ ​ ( 𝒛 | 𝒙 ) ​ ∇ θ ​ log ⁡ π θ ​ ( 𝒛 | 𝒙 ) + π θ ​ ( 𝒛 | 𝒙 ) ​ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ​ ∇ θ ​ log ⁡ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] \displaystyle\sum_{{\bm{z}}}\big[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}})\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\pi_{\theta}({\bm{z}}|{\bm{x}})\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big] = \displaystyle=\, 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) [ ∇ θ log π θ ( 𝒛 | 𝒙 ) + ∇ θ log π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] ] \displaystyle\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big]\Big] = \displaystyle=\, 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ R VeriFree ( 𝒛 ; 𝒙 , 𝒚 ⋆ ) [ ∇ θ log π θ ( 𝒛 | 𝒙 ) + ∇ θ log π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] ] \displaystyle\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[R_{\text{{VeriFree}{}}}({\bm{z}};{\bm{x}},{\bm{y}}^{\star})\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big]\Big] ∎

### A.2 Proof of Lower Variance

Here we provide a full proof of Equation 6 , the reduced variance property of VeriFree. We show that the policy gradient estimator derived from J VeriFree ​ ( θ , 𝒙 , 𝒚 ⋆ ) J_{\text{{VeriFree}{}}}(\theta;{\bm{x}},{\bm{y}}^{\star}) has variance less than or equal to that of the estimator derived from J Verifier ​ ( θ , 𝒙 , 𝒚 ⋆ ) J_{\text{Verifier}}(\theta;{\bm{x}},{\bm{y}}^{\star}) for any given 𝒙 , 𝒚 ⋆ {\bm{x}},{\bm{y}}^{\star} . The same relationship will hold for the global objectives averaged over ( 𝒙 , 𝒚 ⋆ ) ∼ 𝒟 ({\bm{x}},{\bm{y}}^{\star})\sim\mathcal{D} .

###### Proof.

The global objective functions are: J Verifier ​ ( θ ) \displaystyle J_{\text{Verifier}}(\theta) = 𝔼 ( 𝒙 , 𝒚 ⋆ ) ∼ 𝒟 [ 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ 𝟙 { 𝒚 = 𝒚 ⋆ } ] ] ] \displaystyle=\mathbb{E}_{({\bm{x}},{\bm{y}}^{\star})\sim\mathcal{D}}\left[\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\Big[\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\big[\mathbbm{1}\{{\bm{y}}={\bm{y}}^{\star}\}\big]\Big]\right] J VeriFree ​ ( θ ) \displaystyle J_{\text{{VeriFree}{}}}(\theta) = 𝔼 ( 𝒙 , 𝒚 ⋆ ) ∼ 𝒟 [ 𝔼 𝒛 ∼ π θ ( ⋅ | 𝒙 ) [ π θ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] ] \displaystyle=\mathbb{E}_{({\bm{x}},{\bm{y}}^{\star})\sim\mathcal{D}}\Big[\mathbb{E}_{{\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}})}\big[\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big]\Big] For a given ( 𝒙 , 𝒚 ⋆ ) ∼ 𝒟 ({\bm{x}},{\bm{y}}^{\star})\sim\mathcal{D} , the single-sample Monte Carlo gradient estimators are: G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) \displaystyle\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}}) = 𝟙 { 𝒚 = 𝒚 ⋆ } [ ∇ θ log π θ ( 𝒛 | 𝒙 ) + ∇ θ log π θ ( 𝒚 | 𝒙 , 𝒛 ) ] \displaystyle=\mathbbm{1}\{{\bm{y}}={\bm{y}}^{\star}\}\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\big] where 𝒛 ∼ π θ ( ⋅ | 𝒙 ) {\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}}) , 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) {\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}}) , and G ^ VeriFree ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) \displaystyle\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}}) = π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ​ [ ∇ θ ​ log ​ π θ ​ ( 𝒛 | 𝒙 ) + ∇ θ ​ log ​ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] \displaystyle=\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big] where 𝒛 ∼ π θ ( ⋅ | 𝒙 ) {\bm{z}}\sim\pi_{\theta}(\cdot|{\bm{x}}) . The proof relies on the law of total variance and the relationship between G ^ Verifier \hat{G}_{\text{Verifier}} and G ^ VeriFree \hat{G}_{\text{{VeriFree}{}}} .

First, we show that G ^ VeriFree ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) \hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}}) is the conditional expectation of G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) \hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}}) given 𝒙 , 𝒚 ⋆ , 𝒛 {\bm{x}},{\bm{y}}^{\star},{\bm{z}} . The expectation is taken over 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) {\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}}) : 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ G ^ Verifier ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) | 𝒙 , 𝒚 ⋆ , 𝒛 ] \displaystyle\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}})|{\bm{x}},{\bm{y}}^{\star},{\bm{z}}\big] = \displaystyle=\, 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ 𝟙 { 𝒚 = 𝒚 ⋆ } [ ∇ θ log π θ ( 𝒛 | 𝒙 ) + ∇ θ log π θ ( 𝒚 | 𝒙 , 𝒛 ) ] | 𝒙 , 𝒚 ⋆ , 𝒛 ] \displaystyle\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\Big[\mathbbm{1}\{{\bm{y}}={\bm{y}}^{\star}\}\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\big]\Big|{\bm{x}},{\bm{y}}^{\star},{\bm{z}}\Big] = \displaystyle=\, ∑ 𝒚 ′ π θ ( 𝒚 ′ | 𝒙 , 𝒛 ) [ 𝟙 { 𝒚 ′ = 𝒚 ⋆ } [ ∇ θ log π θ ( 𝒛 | 𝒙 ) + ∇ θ log π θ ( 𝒚 ′ | 𝒙 , 𝒛 ) ] ] \displaystyle\sum_{{\bm{y}}^{\prime}}\pi_{\theta}({\bm{y}}^{\prime}|{\bm{x}},{\bm{z}})\Big[\mathbbm{1}\{{\bm{y}}^{\prime}={\bm{y}}^{\star}\}\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\prime}|{\bm{x}},{\bm{z}})\big]\Big] = \displaystyle=\, π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ​ [ ∇ θ ​ log ​ π θ ​ ( 𝒛 | 𝒙 ) + ∇ θ ​ log ​ π θ ​ ( 𝒚 ⋆ | 𝒙 , 𝒛 ) ] \displaystyle\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big[\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{\star}|{\bm{x}},{\bm{z}})\big] = \displaystyle=\, G ^ VeriFree ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) \displaystyle\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}}) We denote 𝔼 𝒚 ∼ π θ ( ⋅ | 𝒙 , 𝒛 ) [ G ^ Verifier ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) | 𝒙 , 𝒚 ⋆ , 𝒛 ] \mathbb{E}_{{\bm{y}}\sim\pi_{\theta}(\cdot|{\bm{x}},{\bm{z}})}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}})\big|{\bm{x}},{\bm{y}}^{\star},{\bm{z}}\big] as 𝔼 𝒚 | 𝒛 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) ] \mathbb{E}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}})\big] for brevity in the following, since 𝒙 {\bm{x}} and 𝒚 ⋆ {\bm{y}}^{\star} are already given and fixed. Thus, we have 𝔼 𝒚 | 𝒛 ​ [ G ^ Verifier ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 , 𝒚 ) ] = G ^ VeriFree ​ ( 𝒙 , 𝒚 ⋆ , 𝒛 ) \displaystyle\mathbb{E}_{{\bm{y}}|{\bm{z}}}\big[\hat{G}_{\text{Verifier}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}},{\bm{y}})\big]=\hat{G}_{\text{{VeriFree}{}}}({\bm{x}},{\bm{y}}^{\star},{\bm{z}}) (8)