Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning — part 2/4

Cleaned-text code points [15771, 31213). Bibliography omitted here; source.md includes it.

The blue bars in Fig. 2 show the performance of this entropy‑gated correction. With only an entropy threshold ( τ = \tau= 1.2), the intervention matches the teacher exactly on the 7B GRPO pair, closely approaches it on the PPO pair, and substantially improves over the base model on the other pairs, while touching only 1.2–8.3% of tokens. Entropy therefore acts as an effective, fully teacher‑free proxy for the decision points that RL would correct. Thus, the where of RL’s correction is predictable from the base model’s entropy alone ; the remaining challenge is to learn which token to substitute at those positions, a problem we solve with ReasonMaxxer (§ 5 ).

## 4 The Correction Is Low-Dimensional

Section 3 showed that RL’s beneficial effect is sparse in token space and predictable from the base model’s entropy. A natural next question is whether the correction is also simple in parameter space. If replicating the RL model’s behavior at decision points required high‑dimensional parameter changes, the observed token‑level sparsity might be an emergent property of a complex distributed computation, and the full RL optimization loop might still be necessary. Several studies have noted that such large‑scale RL can produce representations that look low‑dimensional only after the fact ( Park et al., 2025 ) . To test whether RL’s correction is inherently low‑dimensional, we measure how much adapter capacity is needed to capture it.

### 4.1 Distilling RL into a Low‑Rank Adapter

Our diagnostic is a KL‑LoRA distillation: we attach a LoRA adapter ( Hu et al., 2021 ) to the base model and train only the adapter parameters to minimise the token‑level Kullback–Leibler divergence between the adapter‑augmented model and the RL‑trained teacher: ℒ distill = ∑ t KL ( π teacher ( ⋅ ∣ x < t ) ∥ π base + Δ ​ θ ( ⋅ ∣ x < t ) ) . \mathcal{L}_{\text{distill}}=\sum_{t}\text{KL}\!\left(\pi_{\text{teacher}}(\cdot\mid x_{<t})\;\|\;\pi_{\text{base}+\Delta\theta}(\cdot\mid x_{<t})\right). (4) We cache the teacher’s top‑ k k logits on a set of rollouts generated by the teacher itself. The adapters are trained on only 100 randomly chosen problems . If a tiny adapter can absorb RL’s full distributional change from such a small number of problems, then that change must be fundamentally low‑dimensional.

### 4.2 A Small Adapter Captures RL’s Full Correction

Figure 3 presents the results for the four base/RL pairs studied in § 3 . On both MATH‑500 and GSM8K, a LoRA adapter with rank 32 applied to all attention projections (QKVO) matches the RL teacher’s accuracy, while modifying only 0.27–0.49% of the base model’s parameters.

The adapter sizes above each group (0.3% to 0.5%) make the low‑dimensional nature of RL’s correction immediately visible. The design is frugal by intent: using only 100 randomly chosen problems, the adapter sees just enough examples of the model’s behaviour at critical decision points to capture RL’s policy steering. This reinforces the insight from § 3 that RL’s signal is concentrated in a few high‑entropy locations; a small, targeted dataset suffices because the base model already possesses the necessary vocabulary and reasoning patterns. 1 1 1 Further compression is possible: a rank‑8 output‑projection adapter matches the full W Q ​ K ​ V ​ O W_{QKVO} adapter within a few points on MATH‑500 (Appendix A ), indicating that RL’s correction can be expressed almost entirely through the output layer. We conservatively use the full rank‑32 W Q ​ K ​ V ​ O W_{QKVO} configuration for ReasonMaxxer . Thus, RL’s correction is not only sparse in token space but also low‑dimensional in parameter space: a tiny adapter, on the order of a fraction of a percent of the model’s parameters, captures the entire distributional change.

#### From Representability to Learnability

The KL‑LoRA experiment shows that RL’s corrective signal is representable in a tiny parameter budget. Recent work has further demonstrated that learning such a signal from scratch with LoRA‑constrained RL can match full‑parameter RL, indicating that the solution is not only low‑dimensional but also accessible within a small parameter space ( Wang et al., 2025b ) . This simplicity suggests that the signal might be learnable without RL’s stochastic search, a hypothesis we test directly with ReasonMaxxer in the next section.

## 5 ReasonMaxxer – Entropy‑Gated Contrastive Fine‑Tuning

ReasonMaxxer translates the findings of Sections 3 and 4 into a direct, RL‑free training procedure. The method generates a small set of base‑model rollouts, selects token positions where the base model’s entropy is high, and applies a contrastive loss that encourages tokens leading to correct answers while penalizing those that lead to incorrect ones. The following subsections describe the problem selection, entropy‑based identification of decision points, and contrastive fine‑tuning.

### 5.1 Problem Selection: Exploiting the Edge of Competence

For a collection of math problems with verifiable answers, we sample K K completions per problem from the frozen base model at nonzero temperature and compare each completion against the ground‑truth answer. From this pool we keep exclusively problems where the base model’s pass rate lies strictly between 0 and 1: some rollouts are correct, others are incorrect.

This filter is the direct operationalisation of a property that both prior theoretical work ( Davis and Recht, 2025 ; Zhang et al., 2025 ) and our own oracle experiments (Section 3.2 ) have shown to be necessary for learning from outcome feedback. When the base model always succeeds on a problem, there is no incorrect behaviour to penalise; when it always fails, there is no correct behaviour to reinforce. Only the mixed‑success regime supplies the two‑sided contrastive signal that can distinguish good decisions from bad ones at the same decision points. The filter guarantees that every retained problem contributes this signal. In Section 6.3 we verify empirically that the exact width of the pass‑rate window is not critical; the existence of both correct and incorrect rollouts within a problem is what matters.

### 5.2 Decision‑Point Identification via Entropy

For each retained rollout we compute the per‑token entropy of the frozen base model (Eq. 2 ). A token position t t is designated as a decision point if H t > τ H_{t}>\tau , where τ \tau is a model‑family‑specific threshold chosen so that the marked positions correspond to roughly the top few percent of the model’s entropy distribution. We write 𝒟 = { t : H t > τ } \mathcal{D}=\{t:H_{t}>\tau\} .

This step rests directly on two findings from Section 3 . First, the positions where an RL‑trained teacher disagrees with the base model are precisely the high‑entropy positions (Table 2 , Fig. 1 ). Second, an entropy‑based gate can replace the teacher’s disagreement signal without loss of corrective power (Section 3.3 ). Consequently, 𝒟 \mathcal{D} is a fully teacher‑free, principled selection of the locations where the model’s behaviour most needs refinement. Because entropy is computed from the base model alone, this stage requires no external supervision beyond the rollouts already generated.

### 5.3 Advantage‑Weighted Contrastive Loss with Base Anchoring

Given a set of rollouts for a single problem, we compute a per‑rollout normalised advantage A i = r i − r ¯ σ r + ϵ , A_{i}=\frac{r_{i}-\bar{r}}{\sigma_{r}+\epsilon}, (5) where r i ∈ { 0 , 1 } r_{i}\!\in\!\{0,1\} indicates whether rollout i i arrived at the correct answer, and r ¯ , σ r \bar{r},\sigma_{r} are the mean and standard deviation of the correctness indicators for that problem. This normalisation centres the advantages so that correct and incorrect rollouts receive symmetric positive and negative weights, preventing class imbalance from distorting the gradient.

The training loss is the sum of two terms. At decision points we apply an advantage‑weighted cross‑entropy, ℒ dec = − ∑ t ∈ 𝒟 A i ⋅ log p θ ( x t ∣ x < t ) , \mathcal{L}_{\text{dec}}=-\sum_{t\in\mathcal{D}}A_{i}\cdot\log p_{\theta}(x_{t}\mid x_{<t}), (6) which increases the likelihood of the observed token when the rollout was correct ( A i > 0 A_{i}\!>\!0 ) and decreases it when the rollout was incorrect ( A i < 0 A_{i}\!<\!0 ). The model is therefore shaped to reproduce the token‑level choices that preceded a correct final answer and to avoid those that preceded an incorrect one.

At all positions outside the decision set 𝒟 \mathcal{D} , we minimise the Kullback–Leibler divergence to the frozen base model, ℒ anchor = ∑ t ∉ 𝒟 KL ( p base ( ⋅ ∣ x < t ) ∥ p θ ( ⋅ ∣ x < t ) ) . \mathcal{L}_{\text{anchor}}=\sum_{t\notin\mathcal{D}}\text{KL}\big(p_{\text{base}}(\cdot\mid x_{<t})\;\|\;p_{\theta}(\cdot\mid x_{<t})\big). (7) This anchor term preserves the base model’s behaviour everywhere that the mechanistic analysis found RL to have no effect, and it prevents the small adapter from overfitting to spurious correlations in the limited training set. The total loss is ℒ = ℒ dec + λ ​ ℒ anchor \mathcal{L}=\mathcal{L}_{\text{dec}}+\lambda\mathcal{L}_{\text{anchor}} , the λ \lambda balancing the two objectives.

Architecturally, ReasonMaxxer implements this loss through a LoRA adapter ( Hu et al., 2021 ) attached to the base model. The base model remains frozen; only the low‑rank adapter matrices are updated. This choice is the natural consequence of the low‑dimensionality established in Section 4 : if a rank‑32 adapter containing well under one percent of the model’s parameters can absorb RL’s entire distributional change, then the same parameter budget is more than sufficient to learn the contrastive signal directly from the base model’s own rollouts. Implementation details are in Appendix B .

## 6 Experiments

We benchmark ReasonMaxxer on six mathematical reasoning benchmarks against publicly available RL‑trained models spanning three model families and multiple RL algorithms, and we analyze its performance, efficiency, and critical design choices

### 6.1 Experimental Setup

#### Benchmarks and evaluation protocol.

We evaluate on six standard mathematical reasoning benchmarks: MATH‑500 ( Hendrycks et al., 2021 ) , GSM8K ( Cobbe et al., 2021 ) , AMC 2023, AIME 2024, Minerva Math ( Lewkowycz et al., 2022 ) , and OlympiadBench ( He et al., 2024 ) . For AMC 2023 and AIME 2024, which contain few problems, we report avg@8 (average pass@1 over eight independent generations) to reduce variance; for all other benchmarks we report standard pass@1 from a single generation. Experiment details are given in Appendix C .

#### Training configuration for ReasonMaxxer .

We implement ReasonMaxxer as a rank‑32 LoRA adapter on all attention projections, leaving the base model frozen. From a pool of 150 math problems balanced across difficulty levels we sample 20 rollouts per problem and retain 50 problems on which the base model exhibits mixed success, yielding 1000 training sequences. Entropy‑gated decision points are selected by sweeping the threshold τ \tau on a small held‑out set, and the adapter is trained with the advantage‑weighted contrastive loss described in Section 5 . We train for a single epoch and select the final checkpoint on a fixed 50‑problem validation split. Full hyper‑parameters and optimizer settings are provided in Appendix B .

#### Cost estimation.

To quantify efficiency, we report the estimated monetary cost of training for every method in Table 3 . For ReasonMaxxer , costs are directly measured from our runs on NVIDIA RTX Pro 6000 (96 GB) GPUs using RunPod on‑demand pricing and include rollout generation, entropy scoring, the τ \tau sweep, and checkpoint selection. Baseline costs are either taken from published reports or inferred from the official training scripts, hardware configuration, and on‑demand pricing of the corresponding GPU type (detailed in Appendix D ). All costs are rounded to the nearest US dollar; italicised figures indicate estimates where exact numbers were not publicly documented.

### 6.2 Results and Analysis

Table 3 presents the full set of results. We structure the discussion around three key findings.

#### ReasonMaxxer matches full RL on clean comparisons.

The most direct test compares ReasonMaxxer against publicly available RL trained models that were trained from the same raw base models without additional distillation or SFT stages. On Qwen2.5‑1.5B, ReasonMaxxer achieves 50.2% on MATH‑500 versus SimpleRL‑Zoo’s 49.6% and Open‑Reasoner‑Zero’s 43.6%, while costing $4 compared to $200 and $1200 respectively. On Qwen2.5‑7B, it reaches 70.6% vs. 65.6% (SimpleRL‑Zoo) and 73.2% (Open‑Reasoner‑Zero), again at a tiny fraction of the cost. The pattern holds across the Qwen2.5‑Math‑7B and Qwen2.5‑32B variants, as well as for the Mistral‑7B and Qwen3 models. In every case, we perform on par with or better than the RL baseline while reducing training cost by two to three orders of magnitude. This confirms that the sparse policy‑selection signal identified in Sections 3 – 4 is not merely a diagnostic artifact; it is the signal that RL itself ultimately captures, and ReasonMaxxer recovers it without the RL optimization.

#### Performance generalizes beyond pure RL settings.

Several baselines in Table 3 incorporate additional training strategies beyond outcome‑only RL: DeepSeek‑R1‑Distill‑1.5B baselines start from a distilled checkpoint, Qwen3‑4B’s General‑Reasoner uses a model‑based verifier and multi‑domain data, and STILL‑3 employs iterative RL on a curated dataset. Despite having access to none of these enhancements, ReasonMaxxer still matches or exceeds their accuracy on the majority of benchmarks. These results indicate that a substantial fraction of the gains attributed to sophisticated post‑training pipelines actually originates from the same sparse policy‑selection mechanism that ReasonMaxxer isolates and directly optimizes.

#### Efficiency and scalability.

The computational and data efficiency of ReasonMaxxer are equally notable. In terms of compute, ReasonMaxxer completes in single‑digit GPU‑hours across all models, while the RL baselines require hundreds to tens of thousands of GPU‑hours (Table 3 ); the average training cost is less than $10, compared with $100 to $100,000 for the RL baselines. In terms of data, ReasonMaxxer trains on 50 problems. SimpleRL‑Zoo uses approximately 8,000 MATH problems for its GRPO runs, and Open‑Reasoner‑Zero trains on 57,000 math and reasoning problems. This gap of over two orders of magnitude in training data is not an incidental optimisation; it follows directly from the mechanistic picture established in Sections 3 and 4 . Because RL’s useful signal is concentrated at a sparse set of high‑entropy decision points, a handful of mixed‑success problems supplies sufficient contrastive supervision to capture the full policy‑steering correction. For the same reason, ReasonMaxxer generates rollouts once and offline, and trains only a lightweight adapter; its cost scales with the number of training sequences rather than with the product of model size and on‑policy iterations, keeping it practical even for larger models. In summary, ReasonMaxxer demonstrates that the essential reasoning improvement from outcome‑based RL can be obtained by a simple, data‑efficient contrastive procedure, challenging the necessity of heavy RL infrastructure.

### 6.3 Ablation Studies

#### Sensitivity to τ \tau .