Understanding R1-Zero-Like Training: A Critical Perspective β part 2/4
Cleaned-text code points [14001, 27934). Bibliography omitted here; source.md includes it.
In Deepseek-R1-Zero ( Guo et al., 2025 ) , a notable trend is the consistent increase in response length throughout the training process. This is frequently interpreted as an indication of the development of advanced reasoning abilities such as self-reflection. Recent studies ( Pan et al., 2025 ; Zeng et al., 2025 ; Hu et al., 2025 ) have replicated this phenomenon using various algorithms and implementations. However, we argue that the observed increase in response length may also be attributed to a bias inherent in the GRPO ( Shao et al., 2024 ) objective function : π₯ G β R β P β O ( Ο ΞΈ ) = πΌ πͺ βΌ p π¬ , { π¨ i } i = 1 G βΌ Ο ΞΈ o β l β d ( β
| πͺ ) 1 G β β i = 1 G 1 | π¨ i | β β t = 1 | π¨ i | { min β‘ [ Ο ΞΈ β ( o i , t | πͺ , π¨ i , < t ) Ο ΞΈ o β l β d β ( o i , t | πͺ , π¨ i , < t ) β A ^ i , t , clip β ( Ο ΞΈ β ( o i , t | πͺ , π¨ i , < t ) Ο ΞΈ o β l β d β ( o i , t | πͺ , π¨ i , < t ) , 1 β Ο΅ , 1 + Ο΅ ) β A ^ i , t ] } , \footnotesize\begin{split}\mathcal{J}_{GRPO}&(\pi_{\theta})=\mathbb{E}_{{\mathbf{q}}\sim p_{{\mathcal{Q}}},\{{\mathbf{o}}_{i}\}_{i=1}^{G}\sim\pi_{\theta_{old}}(\cdot|{\mathbf{q}})}\\ &\frac{1}{G}\sum_{i=1}^{G}{\color[rgb]{1,0,0}\frac{1}{|{\mathbf{o}}_{i}|}}\sum_{t=1}^{|{\mathbf{o}}_{i}|}\left\{\min\left[\frac{\pi_{\theta}(o_{i,t}|{\mathbf{q}},{\mathbf{o}}_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|{\mathbf{q}},{\mathbf{o}}_{i,<t})}\hat{A}_{i,t},\text{clip}\left(\frac{\pi_{\theta}(o_{i,t}|{\mathbf{q}},{\mathbf{o}}_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|{\mathbf{q}},{\mathbf{o}}_{i,<t})},1-\epsilon,1+\epsilon\right)\hat{A}_{i,t}\right]\right\},\end{split} (3) where A ^ i , t = R β‘ ( πͺ , π¨ i ) β mean β‘ ( { R β‘ ( πͺ , π¨ 1 ) , β¦ , R β‘ ( πͺ , π¨ G ) } ) std β‘ ( { R β‘ ( πͺ , π¨ 1 ) , β¦ , R β‘ ( πͺ , π¨ G ) } ) , \hat{A}_{i,t}=\frac{R({\mathbf{q}},{\mathbf{o}}_{i})-\operatorname{mean}({\{R({\mathbf{q}},{\mathbf{o}}_{1}),\dots,R({\mathbf{q}},{\mathbf{o}}_{G})\}})}{{\color[rgb]{1,0,0}\operatorname{std}({\{R({\mathbf{q}},{\mathbf{o}}_{1}),\dots,R({\mathbf{q}},{\mathbf{o}}_{G})\}})}}, with the return R β‘ ( πͺ , π¨ i ) R({\mathbf{q}},{\mathbf{o}}_{i}) typically only including the outcome verifiable reward in LLM reasoning (the analysis also applies to process reward cases).
Compared to the objective function in Eq. 2 , GRPO introduces two biases (see also Fig. 4 ): β’ Response-level length bias : This arises from dividing by | π¨ i | |{\mathbf{o}}_{i}| . For positive advantages ( A ^ i , t > 0 \hat{A}_{i,t}>0 , indicating a correct response), this bias results in greater gradient updates for shorter responses, leading the policy to favor brevity in correct answers. Conversely, for negative advantages ( A ^ i , t < 0 \hat{A}_{i,t}<0 , indicating an incorrect response), longer responses are penalized less due to their larger | π¨ i | |{\mathbf{o}}_{i}| , causing the policy to prefer lengthier responses among incorrect ones.
β’ Question-level difficulty bias : This is caused by dividing the centered outcome reward by std β‘ ( { R β‘ ( πͺ , π¨ 1 ) , β¦ , R β‘ ( πͺ , π¨ G ) } ) \operatorname{std}(\{R({\mathbf{q}},{\mathbf{o}}_{1}),\dots,R({\mathbf{q}},{\mathbf{o}}_{G})\}) . Questions with lower standard deviations (e.g., those that are too easy or too hard, with the outcome rewards being almost all 1 or 0) are given higher weights during policy updates. While advantage normalization is a common trick in RL ( Andrychowicz et al., 2021 ) , it is typically computed across an entire batch. In contrast, question-level normalization results in varying weights in the objective for different questions, leading to a difficulty bias in optimization.
Length Bias Also Exists in Open-Source PPO Implementations . We also examined several popular open-source implementations of vanilla PPO algorithms for LLM post-training. To our surprise, all of these implementations normalize the loss by response length (see and Table 2 ), which misaligns with the PPO objective as defined in Eq. 2 . This formulation-implementation misalignment was present even before the publication of GRPO. We speculate that the misalignment might originate from the pretraining stage ( Shoeybi et al., 2019 ) , where all tokens are packed into a fixed-length context and normalizing the loss by the context length (i.e., computing loss.mean(-1) ) improves the numerical stability. However, in the RL-tuning stage , typical implementations ( von Werra et al., 2020 ) normalize the loss by the response length, which is not a constant, introducing an unintended length bias.
Comparison between a typical open-source PPO loss implementation that is biased (red) and our implementation (green). MAX_TOKENS is a global constant during the entire training (unless budget curriculum is enabled), which specifies the maximum number of generation tokens. Other constants also work with differences in gradient norm.
### 3.2 Dr. GRPO: Group Relative Policy Optimization Done Right
To avoid the aforementioned optimization bias in GRPO, we propose to simply remove the 1 | π¨ i | {\color[rgb]{1,0,0}\frac{1}{|{\mathbf{o}}_{i}|}} and std β‘ ( { R β‘ ( πͺ , π¨ 1 ) , β¦ , R β‘ ( πͺ , π¨ G ) } ) {\color[rgb]{1,0,0}\operatorname{std}({\{R({\mathbf{q}},{\mathbf{o}}_{1}),\dots,R({\mathbf{q}},{\mathbf{o}}_{G})\}})} normalization terms. Meanwhile, to faithfully implement the unbiased optimization objective, we could replace the mask.sum(axis=dim) with a constant value (e.g., generation budget) in the masked_mean function in , as highlighted by the line in green. Notably, these simple modifications recover the PPO objective in Eq. 2 , with the advantage estimated by Monte Carlo return with an unbiased baseline ( Sutton & Barto, 2018 ) . We give detailed derivations in App. A . We refer to our new optimization algorithm as Dr. GRPO . We next experimentally validate its effectiveness.
Experimental settings . We implement our algorithm using Oat ( Liu et al., 2025a ) , a modular, research-friendly and efficient LLM RL framework. We adopt the Qwen2.5-1.5B base model and the R1 template ( Template 1 ) for online RL-tuning. We implement the verification-based reward function using Math-Verify 2 2 2 https://github.com/huggingface/Math-Verify . , with the following minimalistic rule: R β‘ ( πͺ , π¨ ) = { 1 if π¨ contains the correct final answer to πͺ 0 otherwise R({\mathbf{q}},{\mathbf{o}})=\begin{cases}1&\text{if ${\mathbf{o}}$ contains the correct final answer to ${\mathbf{q}}$}\\ 0&\text{otherwise}\end{cases} We run RL on questions sampled from the MATH ( Hendrycks et al., 2021 ) training dataset, and compare the vanilla GRPO with the proposed Dr. GRPO. We evaluate the online model on five benchmarks: AIME2024, AMC, MATH500, Minerva Math and OlympiadBench. More experimental details including hyperparameters can be found in App. G .
Results . We report various metrics in Fig. 5 to demonstrate that Dr. GRPO can effectively mitigate the optimization bias and lead to better token efficiency . In particular, we first note that both GRPO and Dr. GRPO exhibit similar trend to DeepSeek-R1-Zero ( Guo et al., 2025 ) , namely their response length increases along with training reward (Plots 1 & 2). However, we observe that GRPO tends to continually generate longer responses even when the reward improvement slows down (Plot 2). Although such a phenomenon is often referred to as the βemergenceβ of long-CoT through RL ( Zeng et al., 2025 ; Hu et al., 2025 ) , we argue that it is also confounded by the response-level length bias ( Sec. 3.1 ) during optimization 3 3 3 We note that both Zeng et al. (2025) and Hu et al. (2025) employ PPO, which is unbiased by formulation. However, their loss implementations still introduce the length bias (see ). . In contrast, by computing the unbiased policy gradients, Dr. GRPO prevents the response length from growing wildly during training (Plot 2). Moreover, on evaluation benchmarks, the length of incorrect responses is substantially reduced by Dr. GRPO compared to the baseline (Plot 4), suggesting that an unbiased optimizer also mitigates overthinking ( Chen et al., 2024 ) .
### 3.3 A Duet of Template and Question Set Coverage in RL dynamics
Recall that the Qwen2.5-Math base models can readily answer questions with high accuracy without any prompt template ( Sec. 2.2 ). Based on this intriguing observation, we are interested in how different templates affect the RL training. Furthermore, given the general belief that larger question set coverage leads to better performance ( Luo et al., 2025 ; Hu et al., 2025 ) , we also study the interaction between different templates and different levels of question coverage.
Experimental settings . Starting from the Qwen2.5-Math-1.5B base model, we apply R1 template, Qwen-Math template and No template respectively to run RL using Dr. GRPO. All experiments are repeated for different question sets that are detailed in Table 3 .
Results . Fig. 6 shows the RL curves of different runs, from which we can make several interesting observations: 1) Templates determine the performance of the initial policies, but RL can improve all policies to a comparable performance of βΌ 40 % \sim 40\% (given a proper question set); 2) When using the R1 template, question sets have a significant impact on the dynamics of RL, with too narrow coverage leading to lower plateau performance. However, when using the Qwen-Math template, the best final performance is attained by RL on GSM-8K, demonstrating that training on much simpler (and o.o.d.) questions can largely improve (nearly double) the test accuracy on harder questions. From these observations, we draw the following insights: β’ The Qwen2.5-Math-1.5B base model already possesses strong math-solving capabilities (see the starting point in the right plot of Fig. 6 ). Applying templates in fact destroys the capability before RL reconstructs it. This implies that we should be more conservative in claiming the huge gains brought about by pure RL.
β’ When there is a large mismatch between base models and templates (e.g., R1 template mismatches Qwen2.5-Math-1.5B), the policy improvement mainly comes from RL-tuning, thus requiring question set to have good coverage (left plot of Fig. 6 ). Otherwise , even a small and completely o.o.d. question set could induce the reasoning ability equally well, by reinforcing useful reasoning behaviors instead of infusing new knowledge .
### 3.4 Domain-Specific Pretraining Improves RL Ceiling
Recent successful R1-Zero-like replications of math reasoners mostly employ Qwen2.5 base models as the initial policies ( Zeng et al., 2025 ; Cui et al., 2025 ; Hu et al., 2025 ) , which are already strong math solvers and exhibit self-reflection patterns ( Sec. 2.2 and 2.3 ). In this section we hope to explore the other side: can R1-Zero-like training succeed on originally weak (in terms of math reasoning) base models? We answer this question affirmatively, with the observation that math pretraining would improve the ceiling of RL .
Experimental settings . We adopt the Llama-3.2-3B base model as our starting point, and use the unbiased Dr. GRPO algorithm for RL-tuning with the R1 template. We hypothesize that domain-specific pretraining would help RL, hence we adopt the Llama-3.2-3B-FineMath 4 4 4 https://huggingface.co/HuggingFaceTB/FineMath-Llama-3B . , which is continual pretrained on the FineMath dataset ( Allal et al., 2025 ) . Moreover, as we hypothesize that Qwen2.5 models are likely to be pretrained on concatenated question-response texts ( Sec. 2.2 ), we similarly prepare a concatenated dataset from NuminaMath-1.5 ( Li et al., 2024b ) , and continual pretrain Llama-3.2-3B-FineMath for 2 epochs with learning rate 1e-5. We refer to the concatanated continual pretrained model as Llama-3.2-3B-NuminaQA .
Results . We present the RL curves of different base models in the left plot of Fig. 7 . We observe that RL can even improve the vanilla Llama base model, but the gain is minimal. After continual pretraining (and concatenated continual pretraining) to embed math domain knowledge, Llama models can show much stronger RL performance, validating our hypothesis. We also revisit the GRPOβs optimization bias with the Llama base model. The right plot of Fig. 7 compares the model performance and response length trained with GRPO and Dr. GRPO. We can clearly see that GRPO can produce the βdouble-increaseβ phenomenon, potentially leading to a misperception that long-CoT can also emerge on Llama models after math pretraining. Unfortunately, the increase of length might be due to the optimization bias ( Sec. 3.1 ), which can be effectively mitigated by the proposed Dr. GRPO ( Sec. 3.2 & right plot of Fig. 7 ).
## 4 Closing Remarks
We have taken a critical perspective to examine base models used for R1-Zero-like training, as well as algorithms used for RL. Through the analysis, we demystified how pretraining biases influence RL outcomes and how optimization choices, like GRPO, can unintentionally shape model behavior. With the proposed Dr. GRPO, we offer a simple fix that improves token efficiency while preserving reasoning performance. Our results show that scaling RL can be both effective and efficientβsometimes, less really is more.
## Appendix A Policy Gradient Derivations
In the context of RL for LLM post-training, we typically maximize the value of π₯ β‘ ( Ο ΞΈ ) = πΌ πͺ βΌ p π¬ β [ πΌ π¨ βΌ Ο ΞΈ ( β
| πͺ ) β [ R β‘ ( πͺ , π¨ ) ] ] , \mathcal{J}(\pi_{\theta})=\underset{{{\mathbf{q}}\sim p_{\mathcal{Q}}}}{\mathbb{E}}\left[\underset{{\mathbf{o}}\sim\pi_{\theta}(\cdot|{\mathbf{q}})}{\mathbb{E}}[R({\mathbf{q}},{\mathbf{o}})]\right], (4) where R β‘ ( πͺ , π¨ ) = β t = 1 | π¨ | r β‘ ( πͺ , π¨ β€ t ) R({\mathbf{q}},{\mathbf{o}})=\sum_{t=1}^{|{\mathbf{o}}|}r({\mathbf{q}},{\mathbf{o}}_{\leq t}) is the return ( Sutton & Barto, 2018 ) of the trajectory πͺ ; π¨ {\mathbf{q}};{\mathbf{o}} , and r β‘ ( πͺ , π¨ β€ t ) r({\mathbf{q}},{\mathbf{o}}_{\leq t}) represents the token-level reward for t t -th token in response π¨ {\mathbf{o}} .