Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? — part 3/4

Cleaned-text code points [31364, 46576). Bibliography omitted here; source.md includes it.

Scaling plays a central role in the capabilities of contemporary LLMs. It remains an important question whether the conclusions drawn continue to hold as model size increases. For many large models, isolating the effect of RLVR is not feasible. For example, in the case of GPT-o1, the base model is not publicly accessible. Qwen3-235B Yang et al. (2025a) is trained through multiple stages, including RLVR and long-context CoT supervised fine-tuning, which makes it impossible to disentangle the impact of RLVR alone. For Deepseek-R1-Zero, the absence of a publicly hosted API forced us to self-host the model, but throughput was limited to around 50 tokens per second at a maximum sequence length of 32k, rendering pass@ k k evaluation currently impractical. As a more tractable alternative, we selected the Magistral-Medium-2506 API to conduct a preliminary set of experiments. This model is trained using pure RL, with Mistral-Medium-3-2505 as the starting model Rastogi et al. (2025) . Although the model size is not disclosed, Magistral-Medium performs comparably to Deepseek-R1 and is positioned near the frontier in terms of reasoning capability.

We queried the models using a maximum context length of 40k as the original paper does. Once again, we observed that RLVR provides significant gains at low k k , but little or no improvement at higher k k . Specifically, at k = 1 k=1 , the RLVR-enhanced model solves approximately 7 more problems on AIME24 and 8 more on AIME25 compared to its base version. However, as k k increases, the performance gap steadily narrows. These observations suggest that our conclusion continues to hold even for current, highly capable, near-frontier reasoning models. Whether this trend persists as more compute, such as pre-training scale budgets, is dedicated to RL training remains a critical question for the future of LLM reasoning.

## 5 Discussion

In Section 3 and Section 4 , we identified key limitations of RLVR in enhancing LLM reasoning capabilities. In this section, we explore possible underlying factors that may explain why RLVR remains bounded by the reasoning capacity of the base model.

Discussion 1: Key Differences Between Traditional RL and RLVR for LLMs are Vast Action Space and Pretrained Priors. Traditional RL such as AlphaGo Zero and the DQN series Silver et al. (2017) ; Mnih et al. (2015) ; Yue et al. (2023) can continuously improve the performance of a policy in environments like Go and Atari games without an explicit upper bound . There are two key differences between traditional RL and RLVR for LLMs. First, the action space in language models is exponentially larger than that of Go or Atari games Ramamurthy et al. (2023) . RL algorithms were not originally designed to handle such a vast action space, which makes it nearly impossible to explore the reward signal effectively if training starts from scratch. Therefore, the second distinction is that RLVR for LLMs starts with a pretrained base model with useful prior, whereas traditional RL in Atari and GO games often begins from scratch. This pretrained prior guides the LLM in generating reasonable responses, making the exploration process significantly easier, and the policy can receive positive reward feedback.

Discussion 2: Priors as a Double-Edged Sword in This Vast Action Space. Since the sampling of responses is guided by the pretrained prior, the policy may struggle to explore new reasoning patterns beyond what the prior already provides. Specifically, in such a complex and highly combinatorial space, most responses generated by naive token-level sampling exploration are constrained by the base model’s prior. Any sample deviating from the prior is highly likely to produce invalid or non-sensical outputs, leading to negative outcome reward . As discussed in Section 2.1 , policy gradient algorithms aim to maximize the log-likelihood of responses within the prior that receive positive rewards, while minimizing the likelihood of responses outside the prior that receive negative rewards. As a result, the trained policy tends to produce responses already present in the prior, constraining its reasoning ability within the boundaries of the base model. From this perspective, training RL models from a distilled model may temporarily provide a beneficial solution, as distillation helps inject a better prior.

Possible Future Work. As discussed above, inefficient exploration mechanisms in a vast action space and the reliance on binary outcome rewards may be the root causes of the limitations observed in current RLVR settings. To fundamentally address these challenges, several directions may be worth exploring:

• Efficient exploration strategies in high-level abstraction. High-level exploration mechanisms such as AlphaEvolve Novikov et al. (2025) , which perform self-evolution in a program-level abstraction space, may be crucial for navigating the vast action space. Such strategies could facilitate the discovery of out-of-prior reasoning patterns and previously unseen knowledge structures.

• Data scale via curriculum. A curriculum can begin by training on easier subproblems, allowing the model to improve sampling efficiency and acquire essential meta-skills. By increasing success rates on simpler tasks before tackling harder ones, such a curriculum may hierarchically reduce the exploration space and lift performance from nearly zero to non-zero on challenging parent tasks, thereby enabling RLVR to obtain meaningful rewards Zhang et al. (2025) ; Li et al. (2025) . Although traces of such hierarchical relationships occasionally appear in current RLVR training data, and their effects have been observed in recent work Chen et al. (2025b) , realizing their full potential will require a more deliberate, large-scale data-RL iteration pipeline that ensures sufficient coverage of meta-skills as well as appropriate relationships between easy and hard problems.

• Process reward and fine-grained credit assignment. Compared to purely binary outcome rewards, incorporating intermediate signals to guide the reasoning trajectory may significantly improve exploration efficiency and steer exploration toward more promising solution paths.

• Agentic RL. Current RLVR reasoning are limited to single-turn response, whereas iterative refinement based on feedback is crucial for IMO-level reasoning Huang and Yang (2025) . It also lacks the ability to actively collect new information by using search tools or conducting experiments. A multi-turn agentic RL paradigm, featuring richer interactions with environment feedback, could allow models to generate novel experiences and learn from them. This emerging agent framework has been described as the beginning of an "era of experience" Silver and Sutton (2025) .

## 6 Related Work

We summarize key related works on the analysis of RLVR here and provide a more comprehensive discussion in Appendix B . While recent RLVR methods have achieved impressive empirical results Guo et al. (2025) ; Lambert et al. (2024) , their fundamental impact on reasoning remains underexplored. Several studies Liu et al. (2025a) ; Zhao et al. (2025b) ; Shah et al. (2025) suggest that reflective behaviors in RLVR models originate from the base models rather than being learned through reinforcement learning. Dang et al . Dang et al. (2025) observed a decline in pass@ k k performance post-RLVR training, but their analysis was limited in scope. More importantly, they did not explore the relationship between the base model and the RL model. Deepseek-Math Shao et al. (2024) also observed similar trends, but their study was limited to a single instruction-tuned model and two math benchmarks. In contrast, our work systematically investigates a wide range of models, tasks, and RL algorithms to accurately assess the effects of current RLVR methods and models. We further provide in-depth analyses, including accuracy distributions, reasoning coverage, perplexity trends, and comparison against distilled models, offering a comprehensive understanding of RLVR’s capabilities and limitations.

## 7 Conclusion and Limitations

RLVR is widely regarded as a promising approach to enable LLMs to continuously self-improve and acquire novel reasoning capabilities. In this paper, we systematically investigate the effect of current RLVR methods on the reasoning capacity boundaries of LLMs. Surprisingly, our findings reveal that current RLVR rarely elicits fundamentally new reasoning patterns; instead, the reasoning capabilities of RLVR-trained models remain bounded by those of their base models. These results indicate that current RLVR methods have not fully realized the potential of reinforcement learning to elicit novel reasoning abilities in LLMs through exploration and exploitation. This limitation may stem from the lack of effective exploration strategies in the vast language space as we discussed in Section 5 . Exploration in high-level abstraction, fine-grained credit assignment, and multi-turn agent-environment interactions may alleviate this problem. We hope the community will continue developing methods along these dimensions to unlock the potential of reinforcement learning to discover genuinely novel reasoning strategies.

Despite our best efforts, this study has several limitations. Although we have attempted to evaluate as many strong, publicly available pure-RLVR models as possible, our analysis is constrained by the fact that many of the most capable models and training pipelines remain proprietary. Moreover, RL for LLM is rapidly evolving, and emerging techniques may mitigate some of the limitations identified here. Consequently, our conclusions should be interpreted with awareness of these practical constraints.

## Author Contributions

All authors made valuable contributions to the experimental design, analysis, and iteration, as well as to the writing, editing, and overall management of the project.

• Yang Yue (乐洋) led the project, first discovered the phenomenon where RL pass@k is surpassed by the base model, and proposed the idea; designed the experiments and partially conducted experiments; took primary responsibility for writing the manuscript.

• Zhiqi Chen conducted substantial experiments, including pass@k evaluation across models and benchmarks, and the perplexity analysis; contributed to discussions, figure creation, and manuscript review.

• Rui Lu contributed to inspiration of the idea and conceptualization of the project, story writing and manual check of AI reasoning trajectory.

• Andrew Zhao contributed to discussions on experimental design, proposed the perplexity-based analysis, and contributed to the early implementation of the RL training code.

• Zhaokai Wang contributed to discussions of RLVR’s effect on reasoning boundary, writing, proofreading, and comprehensive manuscript review.

• Yang Yue (乐阳) contributed to the training of visual reasoning model, discussions, proofreading and figure refinement.

• Gao Huang & Shiji Song supervised the research, and assisted in writing the paper.

## Acknowledgements

This work is supported in part by the National Key R&D Program of China under Grant 2022ZD0114903, the National Natural Science Foundation of China under Grants 42327901 and U24B20173, and the Scientific Research Innovation Capability Support Project for Young Faculty under Grant ZYGXQNJSKYCXNLZCXM-I20.

## Appendix

## Appendix Contents

## Appendix A Implementation Details

### A.1 RLVR Algorithms

To reduce memory and computational overhead, several critic-free variants have been proposed. GRPO Shao et al. (2024) estimates the advantage with a normalized reward within a group of responses to the same question: A i = [ r i − mean ⁡ ( r ) ] / std ⁡ ( r ) A_{i}=[r_{i}-\operatorname{mean}(\textbf{r})]/{\operatorname{std}(\textbf{r})} , where r = { r 1 , … , r G } \textbf{r}=\{r_{1},\ldots,r_{G}\} denotes the set of rewards for a group of G G sampled responses. RLOO Ahmadian et al. (2024) instead adopts a leave-one-out baseline within each batch ℬ \mathcal{B} . Its advantage is defined as A i = r i − 1 | ℬ | − 1 ​ ∑ j ≠ i r j A_{i}=r_{i}-\tfrac{1}{|\mathcal{B}|-1}\sum_{j\neq i}r_{j} .

### A.2 Low-Variance pass@ k k Estimation

Directly computing pass@ k k using only k k sampled outputs per problem can lead to high variance. To mitigate this, we follow the unbiased estimation method proposed by Chen et al . Chen et al. (2021) . Specifically, for each problem x i x_{i} from the evaluation dataset 𝒟 \mathcal{D} , we generate n n samples ( n ≥ k n\geq k ) and count the number of correct samples as c i c_{i} . The unbiased estimator of pass@ k k over the dataset is given by: pass@ ​ k := 𝔼 x i ∼ 𝒟 ​ [ 1 − ( n − c i k ) ( n k ) ] \text{pass@}k:=\mathbb{E}_{x_{i}\sim\mathcal{D}}\left[1-\frac{\binom{n-c_{i}}{k}}{\binom{n}{k}}\right] (2) With this formulation, we can easily estimate pass@ k k with low variance across all k ≤ n k\leq n .

In our experiments, we set n n to the largest ( i . e ., rightmost) k k value in the pass@ k k curves, typically 128, 256, or 1024. For example, in Figure 2 , we use n = 128 n=128 for MATH500, Minerva, and GSM8K, and n = 1024 n=1024 for AMC23 and AIME24. For the Olympiad benchmark, we set n = 128 n=128 for the Qwen models and n = 1024 n=1024 for LLaMA-3.1-8B, due to its relatively lower base model capacity.

## Appendix B More Related Works

Reinforcement Learning for LLM Reasoning. Since the emergence of LLMs, the post-training phase has proven crucial to enhance problem solving and reasoning abilities Ouyang et al. (2022) . This stage typically falls into three main categories: supervised fine-tuning using human-curated or distilled data Wang et al. (2023) , self-improvement iteration Zelikman et al. (2022) ; Gulcehre et al. (2023) , and reinforcement learning Ouyang et al. (2022) . Previously, a reward model or preferences between responses were employed for reward modeling Ouyang et al. (2022) ; Rafailov et al. (2023) . Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has gained significant traction as a method to improve the reasoning capabilities of LLMs in domains such as mathematics and programming Lambert et al. (2024) ; Shao et al. (2024) . An encouraging landmark work is OpenAI’s o1 model Jaech et al. (2024) , which was among the first large-scale applications of RL for reasoning, achieving state-of-the-art results at the time of its release. Following this, Deepseek-R1 Guo et al. (2025) became the first open-weight model to match or surpass the performance of o1. A significant innovation introduced with R1 is the “zero” setting, where reinforcement learning is applied directly to the base LLM, bypassing any intermediate supervised tuning. This approach inspired a wave of open-source efforts to replicate or extend R1’s methodology and improve RL algorithms Zeng et al. (2025) ; Liu et al. (2025b) ; Yu et al. (2025) ; Liu and Zhang (2025) ; Zhao et al. (2025a) ; Wang et al. (2025) . In parallel, reinforcement learning has also gained attention in the multimodal domain, driving advancements in multimodal reasoning Chen et al. (2025a) ; Shen et al. (2025) ; Zheng et al. (2025) .