Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning — part 3/4

Cleaned-text code points [31213, 46867). Bibliography omitted here; source.md includes it.

Figure 4 shows pass@1 on MATH‑500 and GSM8K as τ \tau varies from 1.0 to 2.2, alongside the corresponding mean fraction of tokens that are gated as decision points (annotated below each τ \tau ). Performance is robust over a broad range: the optimal MATH‑500 score (0.50) is achieved at τ = 1.4 \tau=1.4 (5.2% of tokens), and the optimal GSM8K score (0.72) at the same threshold, matching the RL model. A second peak appears at τ = 1.8 \tau=1.8 (2.6% of tokens) where the decision fraction closely matches RL’s observed intervention rate of 2.11% (Table 2 ). This indicates that ReasonMaxxer does not require a precise replication of RL’s sparsity; as long as the threshold selects a plausible set of high‑entropy positions, the contrastive signal is effective. The broad plateau confirms that entropy gating is a reliable, teacher‑free proxy for locating RL’s intervention sites, as argued in Section 3.3 .

#### Necessity of the contrastive term.

To isolate the contribution of the negative gradient, we compare the full ReasonMaxxer loss against a variant where only correct rollouts ( A i > 0 A_{i}>0 ) are used (positive‑only training, equivalent to supervised fine‑tuning on correct trajectories). On Qwen2.5‑1.5B, positive‑only training raises MATH‑500 pass@1 from 0.298 (base) to 0.398, a non‑trivial improvement that confirms the value of targeting decision points. However, it remains far below the RL model (0.496) and full ReasonMaxxer (0.502). The contrastive term that suppresses incorrect decisions thus contributes roughly half of the total gain over the base model, and it is the combination of positive reinforcement and negative suppression that together capture RL’s full policy‑steering effect. This directly supports the design choice: a two‑sided contrastive loss exploits the edge‑of‑competence signal discussed in Section 5.1 , teaching the adapter not only which tokens to prefer but also which tokens to avoid.

## 7 Related Work

Many of the works most directly relevant to this paper were already discussed in the introduction (Section 1 ). Here we provide a more complete discussion, situating our study within the broader literature.

#### What RL does for reasoning.

A growing body of work has questioned whether RLVR expands or merely refines the base model’s reasoning capabilities. Yue et al. (2025) apply pass@ k k analysis to show that RL‑trained models’ reasoning paths lie within the base model’s sampling distribution. Davis and Recht (2025) prove that popular RL algorithms with binary rewards reduce to stochastic gradient ascent on monotone transforms of the probability of a correct answer, implying that optimisation is profitable only when the base model already succeeds non‑trivially. Zhang et al. (2025) confirm this through controlled experiments, finding that RL produces genuine gains only at the model’s edge of competence. Wang et al. (2025d) demonstrate that a single training example can yield large improvements, suggesting that the corrective signal RL imparts is highly compressible. Our work provides a token‑level mechanistic characterisation that unifies these observations.

#### Entropy and decision points in LLM reasoning.

Wang et al. (2025c) identify high‑entropy “forking tokens” as the locus of RL’s gradient signal and show that restricting GRPO updates to these positions matches training on all tokens. Agarwal et al. (2025) demonstrate that entropy minimisation without labeled data improves reasoning performance. Park et al. (2025) find that RL operates through a small number of emergent attention heads. Together, these studies indicate that RL’s effect on large language models is concentrated in a small number of structural and representational units. The present work builds on these insights by establishing causality through oracle intervention and by showing that the sparse signal can be captured without RL.

#### RL post‑training baselines.

Our main experiments compare against a diverse set of publicly available RL‑trained models that span multiple algorithms and training strategies. SimpleRL‑Zoo ( Zeng et al., 2025 ) provides GRPO‑trained checkpoints across ten base models, enabling systematic comparison. Open‑Reasoner‑Zero ( Hu et al., 2025 ) scales PPO on base models without distillation, demonstrating that vanilla PPO with GAE suffices for reasoning improvement. PRIME ( Cui et al., 2025 ) introduces implicit process rewards for online RL, combining outcome supervision with dense token‑level feedback. General‑Reasoner ( Ma et al., 2025 ) extends GRPO with a model‑based verifier across diverse domains beyond mathematics. On the distilled‑model track, DeepScaleR ( Luo et al., 2025 ) applies iterative GRPO with context‑length scaling, STILL‑3 ( Min et al., 2024 ) employs a three‑stage pipeline combining imitation, exploration, and self‑improvement, and Open‑RS3 ( Dang and Ngo, 2025 ) investigates GRPO under tight compute constraints.

#### RL‑free alternatives for reasoning.

Several methods improve reasoning without RL. STaR ( Zelikman et al., 2022 ) and rejection sampling fine‑tuning ( Yuan et al., 2023 ) train on the model’s own correct solutions using uniform token‑level losses. Best‑of‑ N N and MCMC sampling ( Karan and Du, 2025 ) improve reasoning at inference time without modifying the policy. DPO ( Rafailov et al., 2023 ) offers an offline preference‑based alternative that operates at the sequence level. The method proposed in this paper, ReasonMaxxer, differs from these approaches by explicitly targeting the sparse, entropy‑localised decision points identified in the mechanistic analysis.

#### Efficiency in reasoning.

Making LLM reasoning more computationally efficient has been approached from several complementary angles. On the training side, parameter‑efficient fine‑tuning methods such as LoRA ( Hu et al., 2021 ) and QLoRA ( Dettmers et al., 2024 ) have dramatically reduced the cost of adapting large models, and a growing body of work extends these ideas to reasoning specifically: Resa ( Wang et al., 2025a ) uses sparse autoencoder tuning to extract reasoning abilities from a source model and guide lightweight supervised fine‑tuning, while TINA ( Wang et al., 2025b ) shows that LoRA‑constrained RL can match full‑parameter RL on reasoning benchmarks at a fraction of the cost. On the inference side, a separate line of work targets the overthinking phenomenon in large reasoning models by terminating generation once sufficient confidence is reached; DEER ( Yang et al., 2025b ) proposes a training‑free early‑exit mechanism that monitors reasoning transition points and self‑truncates chain‑of‑thought, and LYNX ( Akgül et al., 2025 ) extends this idea with lightweight hidden‑state probes and conformal prediction for distribution‑free confidence control. Broader surveys such as ( Sui et al., 2025 ) provide structured taxonomies of these and related approaches. Together, these works point toward a trend where strong reasoning performance is sought with substantially lower computational overhead than that of current RL‑heavy pipelines.

## 8 Conclusion

We set out to answer whether the RL optimization loop is necessary for improving reasoning in LLMs. Through systematic token‑level analysis across multiple model families and RL algorithms, we showed that RL’s useful effect on math reasoning is a sparse, predictable, and low‑dimensional correction. We then demonstrated that this correction can be obtained without RL at all with ReasonMaxxer . These results reframe reasoning improvement as a sparse policy‑selection problem : the model already knows the necessary reasoning paths; it only needs to commit to the right branch at a handful of critical moments. The RL optimization loop, while capable of discovering this correction, is not a prerequisite for it. Our findings suggest that the community’s default investment in heavy RL infrastructure for post‑training may be disproportionate to the complexity of the problem that is actually being solved. Recognizing this simplicity opens the door to a generation of far more efficient post‑training methods.

## Appendix A Detailed KL‑LoRA Compression Ablations

Section 4 demonstrated that a rank‑32 QKVO adapter trained via KL distillation captures RL’s full correction on reasoning tasks. Table 4 reports a more aggressive compression study on Qwen2.5‑1.5B, varying both the rank and the targeted attention modules.

The rank‑8 𝐖 Q ​ K ​ V ​ O \mathbf{W}_{QKVO} adapter already matches the RL teacher, and even an aggressively small output‑projection adapter ( 𝐖 O \mathbf{W}_{O} only, rank 8, 688 K parameters) lags by only 1 point on MATH‑500. This suggests that RL’s correction can be expressed almost entirely through the output layer: the base model already attends to mostly the right evidence, and RL mainly changes how the attended information is written into the hidden state to produce better next‑token choices. We leave the exploration of such extremely compressed adapters for future work, noting that while they are sufficient to represent RL’s signal, learning the signal from scratch in such a constrained space may require different optimization strategies. Throughout the main paper we conservatively use the full rank‑32 𝐖 Q ​ K ​ V ​ O \mathbf{W}_{QKVO} configuration for ReasonMaxxer .

## Appendix B Implementation Details

We provide the full training and architectural details for the KL‑LoRA distillation experiments reported in Section 4 and for ReasonMaxxer (Section 5 ).

### KL‑LoRA Distillation

The KL‑LoRA adapters in Section 4 are trained with a manually implemented KL divergence over the teacher’s top‑64 logits. The teacher model (SimpleRL‑Zoo GRPO checkpoint) generates rollouts with temperature 0.6 and top‑ p p 0.95 (seed 44), and the distribution is cached. The student (base model + LoRA) is trained for three epochs with batch size 2 and gradient accumulation 8, using the AdamW optimizer, learning rate 10 − 4 10^{-4} , weight decay 10 − 2 10^{-2} , and a 10 % warmup ratio. The objective is averaged over generated‑token positions only. LoRA settings for the full‑rank and compressed variants are given in Table 5 .

### ReasonMaxxer

ReasonMaxxer adapters are trained with AdamW using default betas ( 0.9 , 0.999 ) (0.9,0.999) and ϵ = 10 − 8 \epsilon=10^{-8} , with a linear warmup followed by linear decay to zero. All models are trained for one epoch with batch size 1 and gradient accumulation over 8 steps; the exact number of optimizer steps therefore depends on the number of training sequences, but the hyperparameters are identical across model scales. The decision loss ℒ dec \mathcal{L}_{\text{dec}} and the KL anchor ℒ anchor \mathcal{L}_{\text{anchor}} are each averaged over their respective token masks (decision points for ℒ dec \mathcal{L}_{\text{dec}} , all other valid prediction tokens for ℒ anchor \mathcal{L}_{\text{anchor}} ). Decision points are defined exclusively on generated completion tokens; prompt tokens and padding positions are excluded. Advantages are clipped per rollout to [ − 2.5 , 2.5 ] [-2.5,2.5] before token weighting. The KL anchor is computed over all valid non‑decision prediction tokens in the truncated sequence (including both prompt and completion tokens), and the total loss is ℒ = ℒ dec + 0.2 ​ ℒ anchor \mathcal{L}=\mathcal{L}_{\text{dec}}+0.2\,\mathcal{L}_{\text{anchor}} . Table 6 summarises the common hyper‑parameters.

Family‑specific overrides prompt styles are listed in Table 7 . See Appendix C for exact templates).

## Appendix C Prompting and Answer Extraction

The exact prompt templates and answer extraction rules used for each model family are reported below. Within a family, the same template and extraction are applied to the base model, the RL baselines, and the ReasonMaxxer adapter, ensuring a fair comparison.

### Prompt Templates

#### Qwen2.5 and Qwen3.

Both families use a raw completion prompt without a chat template.

Solve the following math problem step by step. Put your final answer in \boxed{}. Problem: {problem} Solution:

#### Mistral‑7B.

The prompt follows a simple instruction‑answer format, without a chat template.

Question: {problem} Answer: Let’s think step by step.

#### DeepSeek‑R1‑Distill.

The native chat template is applied with a single user message.

Solve the following math problem step by step. Give a concise solution and put your final answer in \boxed{}. Problem: {problem}

## Appendix D Cost Estimation Details

Table 3 reports monetary training costs for all methods. Here we describe how those figures were obtained. Whenever a baseline paper explicitly reports wall‑clock time and hardware, we use those numbers directly. When such information is not published, we infer GPU‑hours from the official training scripts and documented hyperparameters, then convert to cost using RunPod on‑demand pricing as of Apr 28, 2026 ($2.49 per H100‑hour for H100‑80G instances; $2.69 per H100‑hour for H100‑SXM instances where specified; $1.89 per GPU‑hour for RTX Pro 6000 instances). In all cases we note whether the cost figure is a direct measurement or an estimate.

#### ReasonMaxxer.

All ReasonMaxxer costs are direct measurements on the authors’ hardware (4 × \times NVIDIA RTX Pro 6000 Blackwell, 96 GB GDDR7). The reported GPU‑hours include rollout generation, entropy scoring, a sweep over τ ∈ { 1.2 , 1.4 , 1.6 , 1.8 } \tau\in\{1.2,1.4,1.6,1.8\} , training, and final checkpoint selection on a hold‑out set. Cost is computed at the RunPod on‑demand rate for that GPU type.

#### SimpleRL‑Zoo.

For Qwen2.5‑7B and larger models, SimpleRL‑Zoo [ Zeng et al., 2025 ] directly reports the number of GPUs and training hours. For the 1.5B variant, which does not have a separately published wall‑clock, we estimate GPU‑hours by scaling the per‑step generation time of the 7B run by model size, holding the training configuration constant (1024 prompts × \times 8 rollouts × \times ∼ \sim 100 GRPO steps) 2 2 2 The public training command is available at https://github.com/hkust-nlp/simpleRL-reason . . The Mistral‑7B run uses the same hardware and step count as the Qwen2.5‑7B group.

#### Open‑Reasoner‑Zero.

Open‑Reasoner‑Zero [ Hu et al., 2025 ] does not report wall‑clock for any model size. We estimate GPU‑hours from the public PPO recipes in the project repository 3 3 3 See playground/orz_1p5b_ppo.py and playground/orz_7b_ppo.py in the official Open‑Reasoner‑Zero codebase: https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero . . The estimation procedure uses the documented hardware configuration (number of nodes, GPUs per node), the number of prompts and rollouts per step, and the step counts inferred from the training curves in the paper. Per‑step time is calibrated against SimpleRL‑Zoo’s published figures for a comparable model size, with a 1.5–2 × \times overhead factor to account for the PPO critic and GAE computation absent in GRPO. The final point estimates are reported as midpoints of plausible ranges; italicised costs in Table 3 reflect this estimation.

#### PRIME‑Zero.

PRIME‑Zero [ Cui et al., 2025 ] publishes per‑step wall‑clock time and the total number of RL steps. We multiply the two to obtain GPU‑hours and convert using the RunPod rate for the reported GPU type. The training data size uses the number of prompts consumed during RL, approximately 13K out of the full Eurus‑2‑RL dataset.

#### General‑Reasoner (Qwen3‑4B).

Ma et al. [ Ma et al., 2025 ] state that the 4B model is trained on 4 nodes × \times 8 H100 GPUs for around 2 days, giving ∼ \sim 1,536 H100‑hours. We adopt this figure directly.

#### DeepSeek‑based baselines.