Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning — part 1/4

Cleaned-text code points [0, 15771). Bibliography omitted here; source.md includes it.
# Rethinking RL for LLM Reasoning: It’s Sparse Policy Selection, Not Capability Learning

###### Abstract

Reinforcement learning has become the standard for improving reasoning in large language models, yet evidence increasingly suggests that RL does not teach new strategies; it redistributes probability mass over solutions the base model already contains. In this work we ask: if RL merely steers the model toward paths it already knows, is the RL optimization loop itself necessary? Through token‑level analysis across multiple model families and RL algorithms, we find that RL’s beneficial footprint is a sparse, predictable correction concentrated at high‑entropy decision points where the model is uncertain which branch to take. Only 1–3% of token positions are affected, the promoted token always lies within the base model’s top‑5 alternatives, and targeted corrections at those few positions causally recover a large fraction of RL’s accuracy gain, while random corrections fail. The base model’s own entropy identifies these positions without any RL‑trained model, and the entire correction is low‑dimensional, representable in a tiny fraction of model parameters. These findings reframe reasoning improvement as sparse policy selection , not capability acquisition. We translate this insight into ReasonMaxxer , a minimal RL‑free method that applies contrastive loss only at entropy‑gated decision points, using a few hundred base‑model rollouts and no online generation. Across three model families, six scales, and six math reasoning benchmarks, ReasonMaxxer matches or exceeds full RL performance while requiring only tens of problems and minutes of single‑GPU training, a reduction in training cost of roughly three orders of magnitude .

## 1 Introduction

Reinforcement learning with verifiable rewards (RLVR) has become the dominant paradigm for improving reasoning in large language models ( Guo et al., 2025 ; Shao et al., 2024 ; Zeng et al., 2025 ) . Systems such as DeepSeek-R1 ( Guo et al., 2025 ) , OpenAI o1 ( Jaech et al., 2024 ) , and Qwen3 ( Yang et al., 2025a ) demonstrate substantial gains from this pipeline, and the field has broadly adopted RL, typically GRPO ( Shao et al., 2024 ) or PPO ( Schulman et al., 2017 ) , as the standard post-training method for mathematical and code reasoning. The implicit assumption underlying this paradigm is that RL, similar to how it discovers novel strategies in games ( Silver et al., 2017 ) , enables LLMs to acquire genuinely new reasoning patterns through reward-driven exploration. A growing body of evidence challenges this assumption. Yue et al. (2025) show that while RL improves pass@ 1 1 , base models achieve higher pass@ k k at large k k : the base model’s sampling distribution already contains correct solutions that RL merely promotes. Davis and Recht (2025) prove that popular RL algorithms with binary rewards all reduce to stochastic gradient ascent on monotone transforms of the probability of a correct answer, and that such optimization is only profitable when the base model already succeeds non-trivially. Zhang et al. (2025) confirm this through controlled experiments: RL produces genuine gains only at the model’s edge of competence, on problems that are difficult but not yet out of reach. At the token level, Wang et al. (2025c) identify that RL’s improvements concentrate at high-entropy “forking tokens” where the model is uncertain which reasoning path to follow, and show that restricting gradient updates to these tokens matches training on all tokens. From a structural angle, Park et al. (2025) find that RL operates through a small number of emergent attention heads. Collectively, these findings converge on an emerging picture: RL primarily steers the model toward committing to solution paths that the base model already contains, rather than inventing genuinely new reasoning strategies.

Despite this growing understanding, a critical gap remains. The works that identify this structure still operate inside the RL framework: Wang et al. (2025c) make RL more efficient rather than eliminating it, Yue et al. (2025) call for improved RL paradigms, and Karan and Du (2025) offer only inference‑time alternatives. The natural next question is whether we can precisely characterize RL’s token‑level effect and, if that characterization is simple enough, whether the RL optimization loop itself is necessary .

In this paper, we answer that question through a systematic token-level analysis across multiple model families and RL algorithms. We find that RL’s behavioral footprint is strikingly simple: it modifies only 1–3% of token positions, does not introduce tokens outside the base model’s top-5 candidates, and concentrates edits at high-entropy decision points where the model is uncertain which reasoning branch to take. Using oracle intervention with random controls, we establish that the specific token chosen at these positions matters causally, recovering a large share of RL’s gain, while random corrections fail. Crucially, these decision points can be located without any RL-trained model: the base model’s own token entropy, which peaks at the positions RL edits, provides a strong proxy for where intervention is useful. We further show that the full correction is low-dimensional, representable in a tiny fraction of model parameters. Together, these findings reframe reasoning improvement as a sparse policy selection problem: committing to the right branch at a handful of uncertainty points , rather than acquiring new capabilities through expensive exploration.

To test this reframing directly, we construct ReasonMaxxer , a minimal RL-free method that exploits the identified structure. ReasonMaxxer generates a small set of rollouts from the base model, uses entropy gating to locate decision points, and applies an advantage-weighted contrastive loss exclusively at those positions, while anchoring all other tokens to the base distribution. The method requires no RL, no online generation, and no large-scale compute: it maximizes reasoning performance with a shoestring budget. Across three model families and multiple scales, ReasonMaxxer matches or exceeds the performance of models trained with full RL, yet uses only tens of problems, hundreds of rollouts, and minutes of single-GPU training, reducing training cost by roughly three orders of magnitude . That so simple a method suffices challenges the prevailing assumption that heavy RL infrastructure is necessary for reasoning improvement.

Our contributions are as follows: • Mechanistic characterization of RL for reasoning. Through token‑level analysis across multiple model families and RL algorithms, we show that RL’s beneficial effect is a sparse, entropy‑localized reranking of tokens the base model already favors, and we establish causality through oracle intervention with random controls.

• An RL‑free method that matches full RL. We introduce ReasonMaxxer , which applies contrastive fine‑tuning only at entropy‑gated decision points using the base model’s own rollouts. It matches or exceeds RL‑trained models on math reasoning benchmarks while using orders‑of‑magnitude less compute and data.

• Evidence that heavy RL is not a prerequisite. By showing that a lightweight method can replicate RL’s reasoning improvement, we demonstrate that the problem RL solves in this domain is sparse policy selection, not capability acquisition. This suggests that the community’s default investment in full RL pipelines for outcome‑based reasoning may be excessive relative to the problem’s complexity .

## 2 Background and Experimental Setup

### 2.1 Reinforcement Learning with Verifiable Rewards

We briefly review the RL algorithms used by the baseline models in our study. Given a prompt q q with ground-truth answer a a , RLVR generates G G rollouts { o i } i = 1 G \{o^{i}\}_{i=1}^{G} from the current policy π θ \pi_{\theta} and assigns each a binary reward R i = 𝟏  [ match  ( o i , a ) ] R^{i}=\mathbf{1}[\texttt{match}(o^{i},a)] . The dominant algorithm among the baselines we evaluate is Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) , which computes per-rollout advantages via group normalization: A ^ i = R i − mean  ( { R j } j = 1 G ) std  ( { R j } j = 1 G ) , \hat{A}^{i}=\frac{R^{i}-\text{mean}(\{R^{j}\}_{j=1}^{G})}{\text{std}(\{R^{j}\}_{j=1}^{G})}, (1) and updates the policy by maximizing a clipped surrogate objective applied uniformly across all token positions. This uniform application is a key point of contrast with our approach: GRPO distributes gradient across every token in every rollout, despite the evidence (presented in § 3 ) that only a small fraction of positions carry the useful signal. Several baselines use alternative algorithms that share the same core structure. Open-Reasoner-Zero ( Hu et al., 2025 ) employs Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) with GAE, while other recent work explores REINFORCE-style variants such as RLOO ( Ahmadian et al., 2024 ) . All of these methods optimize the same underlying objective: increasing the probability of tokens that lead to correct answers, with the primary differences lying in advantage estimation and regularization strategies. Our mechanistic analysis in § 3 studies models trained with GRPO, PPO, and RLOO, and finds the same sparse-correction pattern across all three.

### 2.2 Token-Level Entropy and Decision Points

For an autoregressive language model π θ \pi_{\theta} , the token-level generation entropy at position t t is defined as H t = − ∑ v ∈ 𝒱 π θ ( v ∣ q , o < t ) log π θ ( v ∣ q , o < t ) , H_{t}=-\sum_{v\in\mathcal{V}}\pi_{\theta}(v\mid q,o_{<t})\log\pi_{\theta}(v\mid q,o_{<t}), (2) where 𝒱 \mathcal{V} is the vocabulary and o < t o_{<t} denotes the tokens generated so far. Positions with high H t H_{t} correspond to points where the model distributes probability mass across multiple plausible continuations rather than committing to a single token. Recent work has identified these high-entropy positions as functionally significant: Wang et al. (2025c) show that they act as “forks” steering the model toward different reasoning pathways, and Agarwal et al. (2025) demonstrate that minimizing entropy without labeled data can improve reasoning performance. We refer to positions where H t H_{t} exceeds a threshold τ \tau as decision points , the subset of the generation where the model’s commitment to a reasoning path is genuinely uncertain.

### 2.3 Models and Baselines

The RL algorithms used by the baselines were introduced in Section 2.1 (GRPO, PPO, and their variants). Table 1 summarises the model families and the specific publicly available RL‑trained checkpoints we used in our experiments across the paper. All baselines are trained with verifiable outcome rewards on mathematical reasoning problems.

## 3 What RL Actually Changes: Sparse Corrections at Decision Points

Recent work suggests that RL for reasoning primarily steers the model toward solutions it already knows rather than inventing new strategies ( Yue et al., 2025 ; Davis and Recht, 2025 ; Zhang et al., 2025 ) . To understand what this steering looks like at the token level, we compare the outputs of a base model and its RL‑trained counterpart on the same set of prompts.

Our investigation addresses three questions:

1. How often, and at what kind of positions, does the RL model disagree with the base model? (§ 3.1 )

2. Do these token‑level disagreements cause the observed accuracy gain? (§ 3.2 )

3. Can we locate the critical positions without access to the RL model, using only signals from the base model? (§ 3.3 )

We focus on the four base/RL‑tuned pairs introduced in § 2.3 and evaluate on MATH‑500 with deterministic decoding ( T = 0 T{=}0 ).

### 3.1 Disagreement Is Rare, Conservative, and Concentrated at Decision Points

For each prompt, we generate a response from the base model and, at every token position, we record which token the RL‑tuned teacher model would have preferred given the identical prefix. Positions are then classified as follows: Unshifted : \displaystyle\textsc{Unshifted}: arg ⁡ max ⁡ π base = arg ⁡ max ⁡ π teacher , \displaystyle\arg\max\pi_{\text{base}}=\arg\max\pi_{\text{teacher}}, (3) Reranked : \displaystyle\textsc{Reranked}: arg max π teacher ≠ arg max π base , arg max π teacher ∈ Top - 5 ( π base ) , \displaystyle\arg\max\pi_{\text{teacher}}\neq\arg\max\pi_{\text{base}},\;\arg\max\pi_{\text{teacher}}\in\mathrm{Top}\hbox{-}5(\pi_{\text{base}}), Shifted : \displaystyle\textsc{Shifted}: arg ⁡ max ⁡ π teacher ∉ Top  -  5  ( π base ) . \displaystyle\arg\max\pi_{\text{teacher}}\notin\mathrm{Top}\hbox{-}5(\pi_{\text{base}}). In words, reranked means the teacher promotes a token that was already among the base model’s top‑5 candidates, whereas shifted would indicate a genuinely new preference.

The results, summarized in Fig. 1 and Table 2 , paint a clear picture. Only 1.0–4.1% of all token positions are reranked, and we observe zero shifted positions in any pair. The teacher’s preferred token is, on average, the second most likely token under the base model (mean rank 2.14–2.39). Moreover, the reranked positions have 5–12 × \times higher base‑model entropy than unchanged positions. Thus, RL’s edits are not only extremely sparse; they are also highly predictable: they occur exactly at high‑entropy decision points where the model is uncertain which reasoning branch to follow (cf. § 2.2 ). RL does not introduce novel tokens; it consistently elevates one of the base model’s top alternatives at moments of uncertainty. This explains why prior work observed low perplexity between RL‑trained and base models Yue et al. (2025) : the promoted token was already a plausible candidate.

### 3.2 Correcting Only the Disagreements Recovers RL Performance

Having established where the two models differ, we now ask whether these differences are causally responsible for the RL model’s higher accuracy. We design an oracle intervention: during deterministic generation from the base model, at every position where the teacher disagrees (i.e., the reranked positions from Table 2 ), we replace the base token with the teacher’s preferred token and continue generating from the corrected prefix. As a control, we instead insert a randomly chosen alternative from the base model’s top‑20 ( random substitution ).

Figure 2 shows the outcome. The oracle intervention reproduces the teacher’s pass@1 exactly on every pair, while the random substitution baseline performs no better than the base model (often worse). The fraction of tokens touched by the oracle equals the rerank percentages from Table 2 (1.0–4.1%). Hence, the RL model’s entire accuracy advantage can be attributed to a tiny set of precise token choices at decision points. In short, a handful of token corrections can redirect the full reasoning trajectory ; RL’s benefit is not a diffuse effect but is concentrated at a few branch points where the choice of continuation determines the solution path.

### 3.3 Entropy Alone Identifies the Critical Positions

The oracle experiment relies on the teacher to both locate and correct the important tokens. For a practical RL‑free method, we need to locate these positions without the teacher. The strong correlation observed in § 3.1 suggests that base‑model entropy might serve this role. We therefore test an entropy‑gated intervention: we replace the base token with the teacher’s preferred token at every position where the base‑model entropy exceeds a threshold τ \tau , without using any information about the teacher’s preferences. This probe tells us how well entropy alone can substitute for the teacher’s knowledge of where to intervene.