Understanding R1-Zero-Like Training: A Critical Perspective — part 1/4

Cleaned-text code points [0, 14001). Bibliography omitted here; source.md includes it.
# Understanding R1-Zero-Like Training: A Critical Perspective

###### Abstract

DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can directly enhance the reasoning capabilities of LLMs without supervised fine-tuning. In this work, we critically examine R1-Zero-like training by analyzing its two core components: base models and RL . We investigate a wide range of base models, including DeepSeek-V3-Base, to understand how pretraining characteristics influence RL performance. Our analysis reveals that DeepSeek-V3-Base already exhibit “Aha moment” , while Qwen2.5 base models demonstrate strong reasoning capabilities even without prompt templates , suggesting potential pretraining biases. Additionally, we identify an optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs) during training. To address this, we introduce Dr. GRPO , an unbiased optimization method that improves token efficiency while maintaining reasoning performance. Leveraging these insights, we present a minimalist R1-Zero recipe that achieves 43.3 % 43.3\% accuracy on AIME 2024 with a 7B base model, establishing a new state-of-the-art.

https://github.com/sail-sg/understand-r1-zero 1 1 1 Developed with the LLM RL framework Oat: https://github.com/sail-sg/oat .

## 1 Introduction

DeepSeek-R1-Zero ( Guo et al., 2025 ) revolutionizes the pipeline of large language model (LLM) post-training by introducing the R1-Zero-like training paradigm : directly applying RL to base LLMs without relying on supervised fine-tuning (SFT) as a preliminary step. This new paradigm is appealing due to its simplicity and the demonstrated RL scaling phenomenon : the model reasoning capabilities improve along with a continual increase in model’s response length. This phenomenon is also accompanied by the “Aha moment”, at which the model learns emergent skills such as self-reflections.

In this paper, we aim to understand R1-Zero-like training by studying two essential components: base models and RL . In the first part, we investigate various attributes of base models, with the focus on the Qwen2.5 model family ( Yang et al., 2024a ; Yang et al., 2024b ) , which has been used in recent attempts to reproduce R1-Zero ( Pan et al., 2025 ; Zeng et al., 2025 ; Liu et al., 2025b ; Hu et al., 2025 ) , as well as DeepSeek-V3-Base ( Liu et al., 2024 ) , from which the real R1-Zero model was RL-tuned. In the second part, we identify the bias in optimization of GRPO ( Shao et al., 2024 ) , which may lead to progressively longer incorrect responses. To this end, we propose a simple modification to eliminate the bias, i.e., to get GRPO Done Right ( Dr. GRPO ), which leads to better token efficiency (highlighted in Fig. 1 ).

Our analysis on base models and RL suggests a minimalist recipe for R1-Zero-like training: we RL-tune Qwen2.5-Math-7B using the (unbiased) Dr. GRPO algorithm on MATH ( Hendrycks et al., 2021 ) level 3-5 questions with the Qwen-Math template, and achieve state-of-the-art performance ( Fig. 2 ) with only 27 27 hours compute on 8 × 8\times A100 GPUs. We hope our findings presented in this paper, models released, and the codebase open-sourced could benefit future research in the field. As an overview, we summarize the takeaways of this paper below:

## 2 Analysis on Base Models

In this section, we scrutinize a wide range of base models, including the Qwen-2.5 family ( Yang et al., 2024a ; Yang et al., 2024b ) , Llama-3.1 ( Grattafiori et al., 2024 ) and DeepSeek series ( Liu et al., 2024 ; Shao et al., 2024 ; Guo et al., 2025 ) , asking them 500 500 questions sampled from the MATH ( Hendrycks et al., 2021 ) training set and analyzing their responses.

### 2.1 R1-Zero Trainability: Templates Construct Exploratory Base Policies

Since training from a base model is a fundamental setting of the R1-Zero-like paradigm, we first investigate whether widely used open-source base models, which are typically trained for sentence completion (i.e., p θ  ( 𝐱 ) p_{\theta}({\mathbf{x}}) ), can have their question-answering capabilities effectively elicited through appropriate templates, thereby functioning as a question-answering base policy π θ ( ⋅ | 𝐪 ) \pi_{\theta}(\cdot|{\mathbf{q}}) . In addition to the R1 template ( Template 1 ) in Guo et al. (2025) , we consider the Qwen-Math template ( Template 2 ) used by Zeng et al. (2025) , as well as No template ( Template 3 ):

Experimental settings . We include Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5-7B, Llama-3.1-8B, DeepSeek-Math-7B and DeepSeek-V3-Base-685B for experiments. For each model, we first apply No template to get the model responses, then let GPT-4o-mini to judge whether the model responses are in an answering format (regardless of quality) or in a sentence-completion pattern. We record the percentage of responses that tend to answer the question as the metric. We then apply both R1 template and Qwen-Math template to obtain model responses, and determine the most suitable template for each model based on the metric. Finally, we evaluate the pass@8 accuracy of each model with the corresponding template to assess whether the base policies can explore rewarding trajectories for RL improvement.

Results . The left plot of Fig. 3 shows how well base models (with or without templates) answer the provided questions. We observe that Llama and DeepSeek models all improve the answering ability by employing the proper template (R1 template). However, Qwen2.5 models work best (with 100 % 100\% answering rate) when no template is used. This intriguing property motivates further investigation which will be discussed in Sec. 2.2 . Meanwhile, the lowest answering rate with no template suggests that DeepSeek-V3-Base is a nearly pure base model. This observation motivates us to explore whether a pure base model like DeepSeek-V3-Base demonstrates the Aha moment ( Sec. 2.3 ). The middle plot of Fig. 3 shows the pass@8 accuracy of different base models (with template) at different sampling temperatures. This metric can serve as an indicator of base policy’s exploration ability. For example, if a base policy cannot even sample a single trajectory that leads to the correct final answer, it is impossible for RL to improve the policy because there is no reward signal. Our results demonstrate that all tested models are exploratory (thus ready for RL), with Qwen2.5 models performing the best (even surpassing DeekSeek-V3-Base). This might partially explain that most R1-Zero projects ( Zeng et al., 2025 ; Hu et al., 2025 ) are based on Qwen2.5 models.

### 2.2 Qwen-2.5 Models Unlock the Best Performance When Discarding Template

We next dig into the intriguing observation (c.f. Fig. 3 (Left)) that all Qwen2.5 base models readily serve as chat models even without any template. We take a step further to evaluate the reasoning ability of Qwen2.5-Math models on five standard benchmarks: AIME 2024 ( Li et al., 2024a ) , AMC ( Li et al., 2024a ) , MATH500 ( Hendrycks et al., 2021 ) , Minerva Math ( Lewkowycz et al., 2022 ) , and OlympiadBench ( He et al., 2024 ) . Following common practice, we use greedy decoding and limit the sampling budget to 3000 tokens.

As shown in Table 1 , not using any template can drastically boost the average performance, resulting in an improvement of about 60 % 60\% compared to the traditional 4-shot prompting. Since Qwen2.5-Math ( Yang et al., 2024b ) uses chat model’s data (question-answer pairs) during the pretraining stage, we hypothesize that they might pretrain on the concatenated text to maximize log ⁡ p θ  ( 𝐪 , 𝐨 ) \log p_{\theta}({\mathbf{q}};{\mathbf{o}}) directly. If our hypothesis turns out true, we shall be more careful about using Qwen2.5 models to reproduce DeepSeek-R1-Zero, since the base models are already SFT-like without templates.

### 2.3 Aha Moment Already Appears in Base Models Including DeepSeek-V3-Base

One of the most inspiring results of DeepSeek-R1-Zero is the emergence of self-reflection behaviors, a.k.a., Aha moment, through pure RL training. A few prior studies ( Liu et al., 2025b ; Yeo et al., 2025 ) have suggested that there may not be Aha moment in open-source R1 replications because the base models they use already exhibit self-reflection keywords. However, they have not tested DeepSeek-V3-Base, on which the real R1-Zero model was RL-tuned. We complete this missing piece by hosting DeepSeek-V3-Base-685B ourselves and investigating its responses to the 500 500 MATH questions with the R1 template. From the right plot of Fig. 3 , we can observe that DeepSeek-V3-Base also generates a decent amount of self-reflections, further validating the claims of Liu et al. (2025b) . We also show examples in App. E ( Fig. 13 ) where DeepSeek-V3-Base generates keywords such as “Aha” and “wait”.

An additional important question is whether self-reflection behaviors are associated with improved model performance after RL training. To investigate this, we host DeepSeek-R1-Zero and analyze its responses to the same questions from the MATH dataset. Although self-reflection behaviors occur more frequently in R1-Zero, we observe that these behaviors are not positively correlated with higher accuracy. Detailed analysis can be found in App. F .

## 3 Analysis on Reinforcement Learning

Language model generation can be formulated as a token-level Markov Decision Process (MDP) ℳ = ( 𝒮 , 𝒜 , r , p 𝒬 ) {\mathcal{M}}=({\mathcal{S}},{\mathcal{A}},r,p_{{\mathcal{Q}}}) . At each generation step t t , the state s t ∈ 𝒮 s_{t}\in{\mathcal{S}} is the concatenation of the input question and the output response generated so far: s t = 𝐪 ; 𝐨 < t = [ q 1 , … , q M , o 1 , … , o t − 1 ] s_{t}={\mathbf{q}};{\mathbf{o}}_{<t}=[q_{1},\dots,q_{M},o_{1},\dots,o_{t-1}] . The policy π θ ( ⋅ | s t ) \pi_{\theta}(\cdot|s_{t}) will select the next token o t o_{t} from the vocabulary 𝒜 {\mathcal{A}} , resulting in a deterministic transition to the next state s t + 1 = s t ; [ o t ] s_{t+1}=s_{t};[o_{t}] . The generation process starts from sampling an initial state s 1 = 𝐪 ∼ p 𝒬 s_{1}={\mathbf{q}}\sim p_{{\mathcal{Q}}} from a set of questions, and stops when the autoregressive policy generates the [eos] token or exhausts the budget.

Typically, we maximize the entropy-regularized objective ( Schulman et al., 2017a ) : 𝒥 ( π θ ) = 𝔼 𝐪 ∼ p 𝒬 [ 𝔼 𝐨 ∼ π θ ( ⋅ | 𝐪 ) [ R ( 𝐪 , 𝐨 ) ] − β 𝔻 K  L [ π θ ( ⋅ | 𝐪 ) ) | | π ref ( ⋅ | 𝐪 ) ] ] , \mathcal{J}(\pi_{\theta})=\underset{{{\mathbf{q}}\sim p_{\mathcal{Q}}}}{\mathbb{E}}\left[\underset{{\mathbf{o}}\sim\pi_{\theta}(\cdot|{\mathbf{q}})}{\mathbb{E}}[R({\mathbf{q}},{\mathbf{o}})]-\beta{\mathbb{D}}_{KL}[\pi_{\theta}(\cdot|{\mathbf{q}}))||\pi_{\text{ref}}(\cdot|{\mathbf{q}})]\right], (1) where R ⁡ ( 𝐪 , 𝐨 ) = ∑ t = 1 | 𝐨 | r ⁡ ( s t , o t ) R({\mathbf{q}},{\mathbf{o}})=\sum_{t=1}^{|{\mathbf{o}}|}r(s_{t},o_{t}) is the return ( Sutton & Barto, 2018 ) of the trajectory 𝐪 ; 𝐨 {\mathbf{q}};{\mathbf{o}} , and π ref \pi_{\text{ref}} is a reference policy. The KL regularization term is usually adopted ( β > 0 \beta>0 ) for reinforcement learning from human feedback ( Christiano et al., 2017 ) , where r r is a reward model learned from data collected by π ref \pi_{\text{ref}} . In this case, regularization helps prevent π θ \pi_{\theta} from deviating too far from the distribution where the reward model is accurate ( Jaques et al., 2019 ; Stiennon et al., 2020 ) . However, RL-tuning reasoning models typically employs rule-based verifiers as r r ( Lambert et al., 2024 ) , eliminating the concerns of distributional shift. This allows us to remove the KL term, which not only saves the memory and computation required by π ref \pi_{\text{ref}} during training, but also potentially leads to better performance for R1-Zero-like training ( Hu et al., 2025 ) . We will assume β = 0 \beta=0 throughout this paper.

Policy optimization algorithms . To optimize π θ \pi_{\theta} with the above objective ( Eq. 1 with β = 0 \beta=0 ), Proximal Policy Optimization (PPO) ( Schulman et al., 2017b ) maximizes the following surrogate objective: 𝒥 P  P  O  ( π θ ) = 𝔼 𝐪 ∼ p 𝒬 , 𝐨 ∼ π θ old ( ⋅ | 𝐪 ) ∑ t = 1 | 𝐨 | { min ⁡ [ π θ  ( o t | 𝐪 , 𝐨 < t ) π θ old  ( o t | 𝐪 , 𝐨 < t )  A ^ t , clip  ( π θ  ( o t | 𝐪 , 𝐨 < t ) π θ old  ( o t | 𝐪 , 𝐨 < t ) , 1 − ϵ , 1 + ϵ )  A ^ t ] } , \footnotesize\begin{split}{\mathcal{J}}_{PPO}(\pi_{\theta})&=\mathbb{E}_{{\mathbf{q}}\sim p_{{\mathcal{Q}}},{\mathbf{o}}\sim\pi_{\theta_{\text{old}}}(\cdot|{\mathbf{q}})}\\ &\sum_{t=1}^{|{\mathbf{o}}|}\left\{\min\left[\frac{\pi_{\theta}(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})}{\pi_{\theta_{\text{old}}}(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})}\hat{A}_{t},\text{clip}(\frac{\pi_{\theta}(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})}{\pi_{\theta_{\text{old}}}(o_{t}|{\mathbf{q}},{\mathbf{o}}_{<t})},1-\epsilon,1+\epsilon)\hat{A}_{t}\right]\right\},\end{split} (2) where π θ old \pi_{\theta_{\text{old}}} is the policy before the update, ϵ \epsilon is the clipping hyperparameter, and A ^ t \hat{A}_{t} is an estimator of the advantage function of the t t -th token. A standard way to estimate A ^ t \hat{A}_{t} is to compute the Generalized Advantage Estimation (GAE) ( Schulman et al., 2015 ) with a learned value model V ϕ V_{\phi} . However, in the context of LLM RL-tuning, learning the value model is computationally expensive, so methods that estimate A ^ t \hat{A}_{t} without V ϕ V_{\phi} are practically preferred. For example, Shao et al. (2024) proposed GRPO, which first samples a group of responses { 𝐨 1 , … , 𝐨 G } \{{\mathbf{o}}_{1},\dots,{\mathbf{o}}_{G}\} per question and computes their returns 𝐑 = { R 1 , … , R G } \mathbf{R}=\{R_{1},\dots,R_{G}\} , then sets the advantage of all tokens from 𝐨 i {\mathbf{o}}_{i} as A ^ t = R i − mean ⁡ ( 𝐑 ) std ⁡ ( 𝐑 ) \hat{A}_{t}=\frac{R_{i}-\operatorname{mean}(\mathbf{R})}{\operatorname{std}(\mathbf{R})} .

### 3.1 GRPO Leads to Biased Optimization