Understanding R1-Zero-Like Training: A Critical Perspective — part 3/4

Cleaned-text code points [27934, 43928). Bibliography omitted here; source.md includes it.

The Monte Carlo policy gradient ( Sutton & Barto, 2018 ) of Eq. 4 is ∇ θ 𝒥 ​ ( π θ ) = 𝔼 𝐪 ∼ p 𝒬 ​ [ 𝔼 𝐨 ∼ π θ ( ⋅ | 𝐪 ) ​ [ ∇ θ ​ log ​ π θ ​ ( 𝐨 | 𝐪 ) ​ R ​ ( 𝐪 , 𝐨 ) ] ] = 𝔼 𝐪 ∼ p 𝒬 [ 𝔼 𝐨 ∼ π θ ( ⋅ | 𝐪 ) [ ∇ θ ∑ t = 1 | 𝐨 | log π θ ( o t | 𝐪 , 𝐨 < t ) R ( 𝐪 , 𝐨 ) ] ] = 𝔼 𝐪 ∼ p 𝒬 ​ [ 𝔼 𝐨 ∼ π θ ( ⋅ | 𝐪 ) ​ [ ∑ t = 1 | 𝐨 | ∇ θ ​ log ​ π θ ​ ( o t | 𝐪 , 𝐨 < t ) ​ ∑ t ′ = t | 𝐨 | r ⁡ ( 𝐪 , 𝐨 ≤ t ′ ) ] ] = 𝔼 𝐪 ∼ p 𝒬 ​ [ 𝔼 𝐨 ∼ π θ ( ⋅ | 𝐪 ) ​ [ ∑ t = 1 | 𝐨 | ∇ θ ​ log ​ π θ ​ ( o t | 𝐪 , 𝐨 < t ) ​ ( ∑ t ′ = t | 𝐨 | r ⁡ ( 𝐪 , 𝐨 ≤ t ′ ) − B ⁡ ( 𝐪 , 𝐨 < t ) ) ] ] , \begin{split}\nabla_{\theta}\mathcal{J}(\pi_{\theta})&=\underset{{{\mathbf{q}}\sim p_{\mathcal{Q}}}}{\mathbb{E}}\left[\underset{{\mathbf{o}}\sim\pi_{\theta}(\cdot|{\mathbf{q}})}{\mathbb{E}}[\nabla_{\theta}\log\pi_{\theta}({\mathbf{o}}|{\mathbf{q}})R({\mathbf{q}},{\mathbf{o}})]\right]\\ &=\underset{{{\mathbf{q}}\sim p_{\mathcal{Q}}}}{\mathbb{E}}\left[\underset{{\mathbf{o}}\sim\pi_{\theta}(\cdot|{\mathbf{q}})}{\mathbb{E}}[\nabla_{\theta}\sum_{t=1}^{|{\mathbf{o}}|}\log\pi_{\theta}(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})R({\mathbf{q}},{\mathbf{o}})]\right]\\ &=\underset{{{\mathbf{q}}\sim p_{\mathcal{Q}}}}{\mathbb{E}}\left[\underset{{\mathbf{o}}\sim\pi_{\theta}(\cdot|{\mathbf{q}})}{\mathbb{E}}[\sum_{t=1}^{|{\mathbf{o}}|}\nabla_{\theta}\log\pi_{\theta}(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})\sum_{t^{\prime}=t}^{|{\mathbf{o}}|}r({\mathbf{q}},{\mathbf{o}}_{\leq t^{\prime}})]\right]\\ &=\underset{{{\mathbf{q}}\sim p_{\mathcal{Q}}}}{\mathbb{E}}\left[\underset{{\mathbf{o}}\sim\pi_{\theta}(\cdot|{\mathbf{q}})}{\mathbb{E}}\left[\sum_{t=1}^{|{\mathbf{o}}|}\nabla_{\theta}\log\pi_{\theta}(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})\left(\sum_{t^{\prime}=t}^{|{\mathbf{o}}|}r({\mathbf{q}},{\mathbf{o}}_{\leq t^{\prime}})-B({\mathbf{q}},{\mathbf{o}}_{<t})\right)\right]\right],\end{split} (5) where B ⁡ ( 𝐪 , 𝐨 < t ) B({\mathbf{q}},{\mathbf{o}}_{<t}) is a variance reduction term, which is invariant with respect to o t o_{t} so that 𝔼 o t ∼ π θ ( ⋅ | 𝐪 , 𝐨 < t ) ​ [ ∇ θ ​ log ​ π θ ​ ( o t | 𝐪 , 𝐨 < t ) ​ B ​ ( 𝐪 , 𝐨 < t ) ] \displaystyle\underset{o_{t}\sim\pi_{\theta}(\cdot|{\mathbf{q}},{\mathbf{o}}_{<t})}{\mathbb{E}}[\nabla_{\theta}\log\pi_{\theta}(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})B({\mathbf{q}},{\mathbf{o}}_{<t})] = 𝔼 o t ∼ π θ ( ⋅ | 𝐪 , 𝐨 < t ) ​ [ ∇ θ ​ log ​ π θ ​ ( o t | 𝐪 , 𝐨 < t ) ] ​ B ​ ( 𝐪 , 𝐨 < t ) \displaystyle=\underset{o_{t}\sim\pi_{\theta}(\cdot|{\mathbf{q}},{\mathbf{o}}_{<t})}{\mathbb{E}}[\nabla_{\theta}\log\pi_{\theta}(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})]B({\mathbf{q}},{\mathbf{o}}_{<t}) = [ ∑ o t π θ ​ ( o t | 𝐪 , 𝐨 < t ) ​ ∇ θ ​ log ⁡ π θ ​ ( o t | 𝐪 , 𝐨 < t ) ] ​ B ​ ( 𝐪 , 𝐨 < t ) \displaystyle=[\sum_{o_{t}}\pi_{\theta}(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})\nabla_{\theta}\log\pi_{\theta}(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})]B({\mathbf{q}},{\mathbf{o}}_{<t}) = [ ∑ o t ∇ θ π θ ​ ( o t | 𝐪 , 𝐨 < t ) ] ​ B ​ ( 𝐪 , 𝐨 < t ) \displaystyle=[\sum_{o_{t}}\nabla_{\theta}\pi_{\theta}(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})]B({\mathbf{q}},{\mathbf{o}}_{<t}) = [ ∇ θ ∑ o t π θ ( o t | 𝐪 , 𝐨 < t ) ] B ( 𝐪 , 𝐨 < t ) \displaystyle=[\nabla_{\theta}\sum_{o_{t}}\pi_{\theta}(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})]B({\mathbf{q}},{\mathbf{o}}_{<t}) = [ ∇ θ 1 ] ​ B ​ ( 𝐪 , 𝐨 z < t ) = 0 . \displaystyle=[\nabla_{\theta}1]B({\mathbf{q}},{\mathbf{o}}_{z<t})=0. Typically, we set B ⁡ ( 𝐪 , 𝐨 < t ) = 𝔼 𝐨 ≥ t ∼ π θ ( ⋅ | 𝐪 , 𝐨 < t ) ​ [ ∑ t ′ = t | 𝐨 | r ⁡ ( 𝐪 , 𝐨 ≤ t ′ ) ] B({\mathbf{q}},{\mathbf{o}}_{<t})=\underset{{\mathbf{o}}_{\geq t}\sim\pi_{\theta}(\cdot|{\mathbf{q}},{\mathbf{o}}_{<t})}{\mathbb{E}}[\sum_{t^{\prime}=t}^{|{\mathbf{o}}|}r({\mathbf{q}},{\mathbf{o}}_{\leq t^{\prime}})] , which is the expected cumulative reward in the future (also known as the value of the current state), and denote A ⁡ ( o t | 𝐪 , 𝐨 < t ) = ∑ t ′ = t | 𝐨 | r ⁡ ( 𝐪 , 𝐨 ≤ t ′ ) − B ⁡ ( 𝐪 , 𝐨 < t ) A(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})=\sum_{t^{\prime}=t}^{|{\mathbf{o}}|}r({\mathbf{q}},{\mathbf{o}}_{\leq t^{\prime}})-B({\mathbf{q}},{\mathbf{o}}_{<t}) as the advantage. In the case of outcome reward, ∑ t ′ = t | 𝐨 | r ⁡ ( 𝐪 , 𝐨 ≤ t ′ ) = ∑ t = 1 | 𝐨 | r ⁡ ( 𝐪 , 𝐨 ≤ t ) = R ⁡ ( 𝐪 , 𝐨 ) \sum_{t^{\prime}=t}^{|{\mathbf{o}}|}r({\mathbf{q}},{\mathbf{o}}_{\leq t^{\prime}})=\sum_{t=1}^{|{\mathbf{o}}|}r({\mathbf{q}},{\mathbf{o}}_{\leq t})=R({\mathbf{q}},{\mathbf{o}}) .

By setting B ⁡ ( 𝐪 , 𝐨 < t ) = mean ⁡ ( { R ⁡ ( 𝐪 , 𝐨 1 ) , … , R ⁡ ( 𝐪 , 𝐨 G ) } ) B({\mathbf{q}},{\mathbf{o}}_{<t})=\operatorname{mean}({\{R({\mathbf{q}},{\mathbf{o}}_{1}),\dots,R({\mathbf{q}},{\mathbf{o}}_{G})\}}) , the policy gradient of Eq. 5 becomes ∇ θ 𝒥 ​ ( π θ ) = 𝔼 𝐪 ∼ p 𝒬 ​ [ 𝔼 { 𝐨 i } i = 1 G ∼ π θ ( ⋅ | 𝐪 ) ​ [ 1 G ​ ∑ i = 1 G ∑ t = 1 | 𝐨 | ∇ θ ​ log ​ π θ ​ ( o i , t | 𝐪 , 𝐨 i , < t ) ​ A ~ i , t ] ] , \begin{split}\nabla_{\theta}\mathcal{J}(\pi_{\theta})&=\underset{{{\mathbf{q}}\sim p_{\mathcal{Q}}}}{\mathbb{E}}\left[\underset{\{{\mathbf{o}}_{i}\}_{i=1}^{G}\sim\pi_{\theta}(\cdot|{\mathbf{q}})}{\mathbb{E}}[\frac{1}{G}\sum_{i=1}^{G}\sum_{t=1}^{|{\mathbf{o}}|}\nabla_{\theta}\log\pi_{\theta}(o_{i,t}|{\mathbf{q}},{\mathbf{o}}_{i,<t})\tilde{A}_{i,t}]\right],\end{split} (6) where A ~ i , t = R ⁡ ( 𝐪 , 𝐨 i ) − mean ⁡ ( { R ⁡ ( 𝐪 , 𝐨 1 ) , … , R ⁡ ( 𝐪 , 𝐨 G ) } ) std ⁡ ( { R ⁡ ( 𝐪 , 𝐨 1 ) , … , R ⁡ ( 𝐪 , 𝐨 G ) } ) . \tilde{A}_{i,t}=\frac{R({\mathbf{q}},{\mathbf{o}}_{i})-\operatorname{mean}({\{R({\mathbf{q}},{\mathbf{o}}_{1}),\dots,R({\mathbf{q}},{\mathbf{o}}_{G})\}})}{{\color[rgb]{1,0,0}\xcancel{\operatorname{std}({\{R({\mathbf{q}},{\mathbf{o}}_{1}),\dots,R({\mathbf{q}},{\mathbf{o}}_{G})\}})}}}. We adopt the PPO ( Schulman et al., 2017b ) objective to compute Eq. 6 : 𝒥 ⁡ ( π θ ) = 𝔼 [ 𝐪 ∼ p 𝒬 , { 𝐨 i } i = 1 G ∼ π θ o ​ l ​ d ( ⋅ | 𝐪 ) ] 1 G ​ ∑ i = 1 G 1 | 𝐨 i | ​ ∑ t = 1 | 𝐨 i | { min ⁡ [ π θ ​ ( o i , t | 𝐪 , 𝐨 i , < t ) π θ o ​ l ​ d ​ ( o i , t | 𝐪 , 𝐨 i , < t ) ​ A ~ i , clip ​ ( π θ ​ ( o i , t | 𝐪 , 𝐨 i , < t ) π θ o ​ l ​ d ​ ( o i , t | 𝐪 , 𝐨 i , < t ) , 1 − ϵ , 1 + ϵ ) ​ A ~ i ] } , \footnotesize\begin{split}\mathcal{J}(\pi_{\theta})&=\mathbb{E}{[{\mathbf{q}}\sim p_{\mathcal{Q}},\{{\mathbf{o}}_{i}\}_{i=1}^{G}\sim\pi_{\theta_{old}}(\cdot|{\mathbf{q}})]}\\ &\frac{1}{G}\sum_{i=1}^{G}{\color[rgb]{1,0,0}\xcancel{\frac{1}{|{\mathbf{o}}_{i}|}}}\sum_{t=1}^{|{\mathbf{o}}_{i}|}\left\{\min\left[\frac{\pi_{\theta}(o_{i,t}|{\mathbf{q}},{\mathbf{o}}_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|{\mathbf{q}},{\mathbf{o}}_{i,<t})}\tilde{A}_{i},\text{clip}\left(\frac{\pi_{\theta}(o_{i,t}|{\mathbf{q}},{\mathbf{o}}_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|{\mathbf{q}},{\mathbf{o}}_{i,<t})},1-\epsilon,1+\epsilon\right)\tilde{A}_{i}\right]\right\},\end{split} from which we conclude that both std \operatorname{std} and | 𝐨 | |{\mathbf{o}}| should not appear in the RL objective.

Unbiasedness of A ~ i , t \tilde{A}_{i,t} . We note that A ~ i , t \tilde{A}_{i,t} computed above is equivalent to that of REINFORCE Leave-One-Out (RLOO) ( Ahmadian et al., 2024 ; Kool et al., 2019 ) up to a scaling factor, which can be subsumed into the learning rate without affecting the RL dynamics. Specifically, G G − 1 ⋅ A ~ i , t = G G − 1 ​ R ​ ( 𝐪 , 𝐨 i ) − G G − 1 ​ 1 G ​ ∑ j = 1 G R ⁡ ( 𝐪 , 𝐨 j ) = G G − 1 ​ R ​ ( 𝐪 , 𝐨 i ) − 1 G − 1 ​ ∑ j = 1 , j ≠ i G R ⁡ ( 𝐪 , 𝐨 j ) − 1 G − 1 ​ R ​ ( 𝐪 , 𝐨 i ) = A ^ i , t RLOO . \begin{split}{\color[rgb]{0.6,0,0.4}\frac{G}{G-1}}\cdot\tilde{A}_{i,t}&={\color[rgb]{0.6,0,0.4}\frac{G}{G-1}}R({\mathbf{q}},{\mathbf{o}}_{i})-{\color[rgb]{0.6,0,0.4}\frac{G}{G-1}}\frac{1}{G}\sum_{j=1}^{G}R({\mathbf{q}},{\mathbf{o}}_{j})\\ &={\color[rgb]{0.6,0,0.4}\frac{G}{G-1}}R({\mathbf{q}},{\mathbf{o}}_{i})-\frac{1}{G-1}\sum_{j=1,j\neq i}^{G}R({\mathbf{q}},{\mathbf{o}}_{j})-\frac{1}{G-1}R({\mathbf{q}},{\mathbf{o}}_{i})\\ &=\hat{A}^{\text{RLOO}}_{i,t}.\end{split}

## Appendix B Detailed Benchmark Results

We show the detailed benchmark results for three scales (1.5B, 3B and 7B) in Table 4 . We also include the instruct models at the same scale and R1-Distill models for comparison. Note that since we employ the Qwen2.5-Math base models, which have a context length of 4k, we thus limit the generation budget at 3k for all baselines compared. For models that are trained for a longer context (OpenReasoner-Zero end R1-Distill-Qwen), we also report their performance at 8k generation budget.

## Appendix C Extended Empirical Results

In this section we present two extended empirical results for (1) the ablation of different bias terms in GRPO and (2) statistical significance of Dr. GRPO’s results. We RL-tune the Qwen2.5-1.5B base model on a mixture of 3K diverse math questions drawn from ASDiv, MATH, and AIME (pre-2023).

Fig. 8 shows the training and evaluation curves for the following variants: Dr. GRPO, GRPO w/o length normalization, GRPO w/o standard deviation (std) normalization and Vanilla GRPO. From the middle subplot, we observe that both Dr. GRPO and the variant without length normalization generate shorter responses compared to the other two. This confirms that the length bias term has a more significant influence on response length–consistent with our expectations.

In terms of performance, Dr. GRPO and the other ablated variants consistently outperform vanilla GRPO in both training rewards and evaluation accuracy. This indicates that removing bias terms (either length or std) improves policy learning, validating our motivation for Dr. GRPO.

Fig. 9 compares GRPO and Dr. GRPO across three independent runs. We observe that Dr. GRPO consistently demonstrates statistically significant improvements–both in token efficiency and final accuracy–across different random seeds.

## Appendix D Keyword-based Detection and LLM-Based Identification of Self-Reflection Behaviors

We construct a pool of carefully selected keywords and phrases that signal self-reflection behaviors in the LLM’s responses. However, LLM-generated responses often contain hallucinations and off-topic content, leading to the presence of simple, ambiguous keywords that do not necessarily indicate genuine self-reflection. For instance, terms like “wait” and “try again” frequently result in false positive detections. To reduce false positives, we maintain a small, highly selective keyword pool consisting of terms that are strongly indicative of self-reflection. In our experiment, the keyword pool is limited to: recheck, rethink, reassess, reevaluate, re-evaluate, reevaluation, re-examine, reexamine, reconsider, reanalyze, double-check, check again, think again, verify again, and go over the steps.

We present the occurrences of various keywords in the responses generated by different models in Figure 10 . Interestingly, different model families emphasize different keywords. For instance, phrases such as “check again”, “double-check”, “re-evaluate”, “re-examine”, “recheck”, “reconsider”, and “verify again” appear most frequently in the Qwen2.5 family. In contrast, “re-evaluate”, “re-examine”, and “verify again” do not appear in the responses of the DeepSeek family, while Llama models frequently use the phrase “think again.” We hypothesize that this phenomenon results from differences in the pretraining data, particularly in relation to reasoning and mathematics.

Although we meticulously select the keyword pool, it may still be insufficient to identify some implicit behaviors of self-reflection that do not contain a specific keyword. Additionally, it can lead to false positives, as illustrated in Case (a) of Figure 11 . To address these limitations and more accurately assess the self-reflection capability of base models, we leverage stronger LLMs ( GPT-4o-mini in our experiments) to analyze the responses and determine whether they exhibit explicit self-reflection (e.g., keywords like ”recheck” and ”reevaluate”) or implicit self-reflection (e.g., more sophisticated patterns that cannot be easily captured through keyword matching). This approach helps distinguish true self-reflection behaviors from superficial or incidental use of related terms.

While LLM-based detection effectively filters out false positives from keyword-based detection and identifies implicit self-reflection behaviors, it can still misclassify responses, particularly when they are lengthy and complex. For instance, Case (b) in Figure 11 shows a false positive in LLM-based detection, where the response is categorized as self-reflection by the LLM but does not actually exhibit self-reflection. This type of error can be filtered out by keyword-based detection. To enhance robustness, we integrate keyword-based and LLM-based detection through cross-validation. The combined detection results, along with the individual results from keyword-based and LLM-based methods, are presented in Figure 12 .

## Appendix E Examples of Aha Moment in DeepSeek-V3-Base

Fig. 13 shows two examples to demonstrate that the DeepSeek-V3-Base model already exhibits the so-called “aha moment” even before the RL-tuning.

## Appendix F Comparison Between DeepSeek-V3-Base and DeepSeek-R1-Zero

We analyze DeepSeek-V3-Base and DeepSeek-R1-Zero to understand changes in model behavior during R1-Zero training. In Fig. 14 , we present the breakdown of response categories across difficulty levels for 500 MATH questions evaluated on both models. The results indicate that most incorrect responses are corrected after RL training, demonstrating substantial performance gains from R1-Zero training. Meanwhile, we find an increase in unformatted responses, which aligns with the observation in Liu et al. (2025b) .

In Table 5 , we report the average response lengths across categories. Note that truncated responses would fall into any of the other three categories if a larger context size were used; thus, we exclude them from the table. The results show a substantial increase in response lengths across all categories, including correct responses, consistent with the results in the Fig. 3 of Guo et al. (2025) . However, the average length of incorrect responses is notably longer than that of correct responses. We hypothesize this is because more challenging questions generally require longer responses due to increased reasoning complexity, and incorrect responses are more likely to originate from harder questions, resulting in a longer average length.

Self-reflection does not necessarily imply higher accuracy. To investigate whether self-reflection behaviors are associated with model performance during the inference (acknowledging that self-reflection may improve exploration during training—a potential positive effect outside this section’s scope), we analyze questions that elicit at least one response with self-reflection from DeepSeek-R1-Zero across eight trials. For each question, we sample 100 responses and divide them into two groups: those with self-reflection and those without. We then compute the accuracy difference between these two groups for each question. As shown in Fig. 15 , the results indicate that nearly half responses with self-reflection do not achieve higher accuracy than those without self-reflection, suggesting that self-reflection does not necessarily imply higher inference-stage accuracy for DeepSeek-R1-Zero.

## Appendix G Detailed Experimental Settings

All our experiments are performed on 8 × \times A100 GPUs and finished in about one day. We enable the actor-learner collocation supported by Oat ( Liu et al., 2025a ) to optimize the training efficiency. We show the experimental configurations in Table 6 .

## Appendix H Prompts Used for GPT-As-A-Judge

Prompt for checking the model’s question-answering ability.

Prompt for LLM-based detection to determine whether a response contains self-reflection behaviors.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.