Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? — part 1/4
Cleaned-text code points [0, 15899). Bibliography omitted here; source.md includes it.
marginparsep has been altered. topmargin has been altered. marginparpush has been altered.
The page layout violates the ICML style.
Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you.
We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again.
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
Yang Yue 1 ∗ † {}^{\,1\,*\,\dagger\,} , Zhiqi Chen 1 ∗ {}^{\,1\,*} , Rui Lu 1 {}^{\,1\,} , Andrew Zhao 1 {}^{\,1\,} , Zhaokai Wang 2 {}^{\,2\,} , Yang Yue 1 {}^{\,1\,} , Shiji Song 1 {}^{\,1\,} , and Gao Huang 1 🖂 {}^{\,1\,\textrm{\Letter}}
1 {}^{1\,} LeapLab, Tsinghua University 2 {}^{2\,} Shanghai Jiao Tong University
∗ Equal Contribution † Project Lead 🖂 {}^{\textrm{\Letter}} Corresponding Author
## 1 Introduction
The development of reasoning-centric large language models (LLMs), such as OpenAI-o1 Jaech et al. (2024) , DeepSeek-R1 Guo et al. (2025) , and Kimi-1.5 Team et al. (2025) , has significantly advanced the frontier of LLM capabilities, particularly in solving complex logical tasks involving mathematics and programming. In contrast to traditional instruction-tuned approaches that rely on human-curated annotations Achiam et al. (2023) ; Grattafiori et al. (2024) , the key driver behind this leap forward is large-scale Reinforcement Learning with Verifiable Rewards (RLVR) Lambert et al. (2024) ; Guo et al. (2025) . RLVR starts with a pretrained base model or one fine-tuned on long chains of thought (CoT) data, optimizing it via reinforcement learning based on simple, automatically computable rewards. These rewards are determined by whether the model’s output matches a ground-truth solution in mathematics or passes unit tests in code, thus enabling scalability without human labeling. This framework has gained significant attention due to its simplicity and practical effectiveness. In traditional RL settings such as game playing ( e . g ., Atari, Go), agents often autonomously discover new strategies and surpass even human-level performance through self-improvement Mnih et al. (2015) ; Silver et al. (2017) . Inspired by this success, it is widely believed that RLVR similarly enables LLMs to autonomously develop novel reasoning patterns, including enumeration, self-reflection, and iterative refinement, surpassing the capabilities of their base models Guo et al. (2025) . Consequently, RLVR has been considered a promising path toward continuously self-evolving LLMs, potentially bringing us closer to more powerful intelligence Guo et al. (2025) .
However, despite its empirical success, the underlying effectiveness of current RLVR remains underexamined. This raises a fundamental question: Does current RLVR genuinely enable LLMs to acquire novel reasoning abilities–similar to how traditional RL discovers new strategies through exploration–or does it simply utilize reasoning patterns already in the base model?
To rigorously answer this question, we must first assess the reasoning capability boundaries of both base and RLVR-trained models. Traditional evaluation metrics rely on average score from greedy decoding or nucleus sampling Holtzman et al. (2020) , which reflects average-case behavior. However, these metrics risk underestimating the true potential of a model, especially if it fails on difficult problems after limited attempts, despite being capable of solving them with more sampling. To overcome this limitation, we adopt the pass@ k metric Brown et al. (2024) , where a problem is considered solved if any of the k k sampled outputs is correct. By allowing multiple attempts, pass@ k reveals whether a model has the potential to solve a problem. The average pass@ k across a dataset thus reflects the proportion of problems a model can potentially solve within k k trials, offering a more robust view of its reasoning boundary. This provides a rigorous test on whether the RLVR training yields fundamentally transcending capacity, enabling the model to solve problems that the base model cannot.
Using the pass@ k metric, we conduct extensive experiments across various benchmarks, covering multiple LLM families, model sizes, and RLVR algorithms to compare base models with their RLVR-trained counterparts. We uncover several surprising findings that offer a more comprehensive assessment of the effectiveness of current RLVR training and reveal the gap between existing RLVR methods and the ideal goals of RL-discovering genuinely new reasoning strategies:
∙ \bullet Current RLVR models often exhibit narrower reasoning coverage than their base models. In pass@ k k curves, although RLVR models outperform their base models at small k k , it is surprising that base models consistently surpass RLVR models across all benchmarks and LLM families as k k increases. This suggests that current RLVR training does not expand, and even reduce the scope of reasoning over solvable problems. Manual inspection of model responses shows that, for most problems, the base model can produce at least one correct CoT, implying that it can already generate correct reasoning paths for problems that were previously considered only solvable for RLVR models.
∙ \bullet Reasoning paths generated by current RLVR model already exist in its base model. To further investigate this phenomenon, we analyze the accuracy distribution. The results show that although RLVR improves average performance ( i . e ., pass@1) by sampling more efficiently on problems already solvable by the base model, it does not enable the model to solve new problems. Further perplexity analysis reveals that the reasoning paths produced by RLVR models already exist within the output distribution of the base model. These findings indicate that RLVR does not introduce fundamentally new reasoning capabilities and that the reasoning capacity of current RLVR models remains bounded by that of its base model. This effect of RLVR is illustrated in Figure 1 (left).
∙ \bullet Current RLVR algorithms perform similarly and remain far from optimal. Treating the base model as an upper bound, we define the sampling efficiency gap ( Δ SE \Delta_{\text{SE}} ), shown in Figure 8 (top), as the difference between an RL model’s pass@1 and the base model’s pass@ k k (with k = 256 k=256 as a proxy for upper-bound performance). This metric quantifies how closely an RL algorithm approaches the optimal bound. Across all algorithms (e.g., PPO, GRPO, Reinforce++), Δ SE \Delta_{\text{SE}} shows only minor variation yet remains consistently large, suggesting that current RLVR methods, while improving sampling efficiency, are still far from optimal.
∙ \bullet RLVR and distillation are fundamentally different. While RLVR improves reasoning scores by more efficiently sampling high-reward outputs, it does not elicit new reasoning capabilities and remains constrained within the base model’s capacity. In contrast, distillation can transfer new reasoning patterns from a stronger teacher to the student. As a result, distilled models often demonstrate an expanded reasoning scope beyond that of the base model.
In conclusion, our findings show that current RLVR methods, while improving sampling efficiency, rarely elicit novel reasoning beyond the base model. This highlights a gap between existing RLVR methods and the goals of reinforcement learning, underscoring the need for improved RL paradigms such as better exploration, continual data scaling, fine-grained process signal, and multi-turn agent interaction.
## 2 Preliminaries
In this section, we first outline the fundamentals of RLVR, then introduce the pass@ k k metric to evaluate reasoning boundaries, and explain why it is preferred over alternatives like best-of- N N .
### 2.1 Reinforcement Learning with Verifiable Rewards
Verifiable Rewards. Let π θ \pi_{\theta} be an LLM with parameters θ \theta that generates a token sequence 𝐲 = ( y 1 , … , y T ) \mathbf{y}=(y_{1},\dots,y_{T}) conditioned on a natural-language prompt x x . A deterministic verifier 𝒱 \mathcal{V} returns a binary reward: r = 𝒱 ( x , 𝐲 ) ∈ { 0 , 1 } , r=\mathcal{V}(x,\mathbf{y})\in\{0,1\}, where r = 1 r=1 if and only if the model’s final answer is exactly correct. A format reward may also be added to encourage the model to explicitly separate the reasoning process from the final answer. The goal of RL is to learn a policy to maximize the expected reward: J ( θ ) = 𝔼 x ∼ 𝒟 [ 𝔼 𝐲 ∼ π θ ( ⋅ | x ) [ r ] ] J(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{\mathbf{y}\sim\pi_{\theta}(\cdot|x)}[\,r\,]\right] , where 𝒟 \mathcal{D} is the distribution of prompts.
RLVR Algorithms. Proximal Policy Optimization (PPO) Schulman et al. (2017) proposed using the following clipped surrogate to maximize the objective: ℒ CLIP = 𝔼 [ min ( r t ( θ ) A t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A t ) ] , \mathcal{L}_{\text{CLIP}}=\mathbb{E}\left[\min\!\left(r_{t}(\theta)A_{t},\;\operatorname{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)A_{t}\right)\right], (1) where r t ( θ ) = π θ ( y t | x , 𝐲 < t ) / π θ old ( y t | x , 𝐲 < t ) r_{t}(\theta)=\pi_{\theta}(y_{t}|x,\mathbf{y}_{<t})/\pi_{\theta_{\text{old}}}(y_{t}|x,\mathbf{y}_{<t}) , and A t A_{t} is the advantage estimated by a value network V ϕ V_{\phi} . KL divergence term is optionally applied, to constrain the model from deviating too far from the original policy. More algorithms are introduced in Section C.5 .
Policy Gradient. PPO and its variants belong to the policy gradient class of RL Williams (1992) ; Sutton et al. (1998) . These methods learn exclusively from on-policy samples , i . e ., samples generated by the current LLM. In the context of verifiable rewards, the training objective generally maximizes the log-likelihood of samples with correct answers and minimizes the likelihood of those with incorrect answers.
Zero RL Training applies RL directly to the base model without any supervised fine-tuning (SFT) Guo et al. (2025) . To clearly study the effect of RLVR, we follow this zero-RL setting for all math tasks using pretrained models as start model. However, for coding and visual reasoning tasks, open-source work typically uses instruction-tuned models as starting points, primarily due to the training instability and limited effectiveness of using a pure zero-RL setting. Following this convention, we compare the finetuned model with its RLVR-trained counterpart to focus solely on the effect of RLVR.
### 2.2 Metrics for LLM Reasoning Capacity Boundary
Pass@ k k Metrics. Accurately measuring the reasoning ability boundary of base and RL models is challenging, as methods like greedy decoding or the average of nucleus samplings Holtzman et al. (2020) only reflect average-case performance. To accurately measure the reasoning ability boundary, we extend the commonly used pass@ k metric from code generation Chen et al. (2021) to all tasks with verifiable rewards. Given a problem, we sample k k outputs from the model. The pass@ k k value for this question is 1 if at least one of the k k samples passes verification; otherwise, it is 0. The average pass@ k k value over the dataset reflects the proportion of problems in the dataset that the model can solve within k k trials, providing a rigorous evaluation of the reasoning capacity coverage of LLMs. We adopt an unbiased, low-variance estimator for computing to calculate pass@ k k , as detailed in Section A.2 .
Comparison with Best-of- N N and Majority Voting. Best-of- N N Cobbe et al. (2021) and majority voting are practical methods for selecting correct answers, but they may overlook a model’s full reasoning potential. In contrast, we use pass@ k k not to assess practical utility but to investigate the boundaries of reasoning capacity. If a model produces a correct solution in any of the k k samples, we treat the problem as within its potential scope. Thus, if RL enhances reasoning, the RL-trained model should succeed in more such problems than the base model. Methods like Best-of- N N or majority voting may miss these successes if the correct answer is not selected by the verifier or voting.
Random Guessing Issue . For coding tasks, where a compiler and predefined unit test cases are used as verifiers, the pass@ k k value can accurately reflect whether the model can solve the problem. In mathematics , the issue of “guessing” can become pronounced as k k increases, where a model may generate an incorrect CoT but still accidentally arrive at the correct answer. To address this, we manually check the correctness of CoT for a subset of model outputs as detailed in Section 3.1 . By combining results on math with manually checking and coding, we rigorously evaluate the scope of LLM’s reasoning capacity. Another caveat is that, with an astronomically large k k , even uniform sampling over the token dictionary would stumble upon the correct reasoning path–though this is infeasible within today’s time and compute budgets. Crucially, we find that the base model already produces correct outputs at realistic values ( k = 128 k=128 or 1024 1024 ), well within practical resource limits.
## 3 RLVR’s Effect on Reasoning Capacity Boundary
With the evaluation metrics for reasoning boundaries established, we now conduct a comprehensive evaluation of the base and RLVR models through extensive experiments. Our analysis is organized by task category, covering three representative domains: mathematics, code generation, and visual reasoning. The overall experimental setup is summarized in Table 1 .
Evaluation Protocol. For sampling procedures for both base and RLVR models, we use a temperature of 0.6 and a top- p p value of 0.95, allowing a maximum generation of 16,384 tokens. We also show the effect of different temperature settings in Figure 17 . For evaluation of the base model, a common practice is to include few-shot examples in the prompt to guide the output Grattafiori et al. (2024) ; Yang et al. (2024) ; Liu et al. (2024) . However, to ensure a fair and unbiased comparison, we deliberately avoid using few-shot prompts for base models, eliminating any potential confounding effects on reasoning that might be introduced by in-context examples. For evaluating both the base and RLVR models, we use the same zero-shot prompt as in RLVR training, or the default prompt provided by the benchmark, ensuring a consistent setup across both models. Interestingly, although base models often produce unformatted or non-sensical responses without few-shot guidance, we observe that with sufficient sampling, they are still capable of generating correctly formatted outputs and successfully solving complex problems. Prompt templates for training and evaluation are provided in Appendix D .
### 3.1 RLVR for Mathematical Reasoning
Models and Benchmarks. In math problems, models are required to generate a reasoning process ( i . e ., CoT) along with the final answer. To ensure the robustness of conclusions, we experiment with multiple LLM families, primarily Qwen2.5 (7B/14B/32B base variants) Yang et al. (2024) and additionally LLaMA-3.1-8B Grattafiori et al. (2024) . We adopt RLVR models released by SimpleRLZoo Zeng et al. (2025) , which train zero-RL models using GRPO on GSM8K and the MATH training set, with correctness reward only, excluding any format-based reward. We compare the pass@ k k curves of base and zero-RL models on benchmarks of varying difficulty: GSM8K Cobbe et al. (2021) , MATH500 Hendrycks et al. (2021) , Minerva Lewkowycz et al. (2022) , Olympiad He et al. (2024) , AIME24, and AMC23. Additionally, we include the RLVR model Oat-Zero-7B and DAPO-32B Liu et al. (2025b) ; Yu et al. (2025) . These two models are characterized by strong performance on the challenging AIME24 benchmark.