Learning to Reason without External Rewards — part 3/4

Cleaned-text code points [30738, 46369). Bibliography omitted here; source.md includes it.

This paper introduces Intuitor , an instantiation of Reinforcement Learning from Internal Feedback (RLIF) that uses a model’s intrinsic self-certainty as its sole reward signal, eliminating the need for external supervision or gold-standard solutions. Our experiments show that Intuitor matches the performance of supervised RLVR methods like GRPO on mathematical reasoning and achieves competitive, sometimes better generalization to out-of-domain tasks such as code generation and instruction following. It also promotes structured reasoning and leverages online self-certainty to guard against reward exploitation. These findings highlight the transformative potential of RLIF, signaling a meaningful step toward AI systems that improve through introspection and unlock rich latent capabilities. Looking forward, this paradigm opens the door to AI agents capable of autonomous skill acquisition in novel domains and scalable self-improvement—even as they approach or surpass the limits of human oversight. Future directions include integrating RLIF with external reward methods like RLHF or RLVR to tackle increasingly complex real-world challenges, and advancing the development of more robust, generalizable, and truly autonomous learning systems.

## Ethics Statement

Our research is based on publicly available datasets and open-source language models, mitigating concerns related to private data or human subjects. The goal of our work is to enhance the reasoning capabilities of language models through self-supervision, which we believe is a positive step toward more transparent and robust AI systems. We have made our code publicly available to ensure transparency and allow for full scrutiny of our methods and findings. We do not foresee any direct negative societal impacts or ethical concerns arising from this work.

## Reproducibility Statement

To ensure the reproducibility of our results, we provide all source code and training configurations in https://github.com/sunblaze-ucb/Intuitor . The Experimental Setup section and Appendix B detail all hyperparameters, software versions (including the Open-R1 framework), and evaluation setups. Furthermore, Appendix C.1 includes the exact prompts used during training and evaluation. These resources should allow for the complete replication of our experiments and validation of our findings.

## Appendix A Additional Background

### A.1 From External Supervision to Internal Feedback

To provide additional context, we review existing RL-based fine-tuning paradigms and their limitations, which motivate our exploration of Reinforcement Learning from Internal Feedback (RLIF).

Current RL fine-tuning approaches for LLMs primarily fall into two categories: those relying on external human feedback (RLHF) and those using verifiable, often task-specific, rewards (RLVR).

In RLHF ( Ziegler et al., 2019 ; Ouyang et al., 2022 ) , the policy π θ \pi_{\theta} is optimized to align with human preferences, typically encapsulated by a learned reward model r ϕ r_{\phi} . The objective is: max π θ 𝔼 o ∼ π θ ​ ( q ) [ r ϕ ( q , o ) − β KL [ π θ ( o | q ) ∥ π ref ( o | q ) ] ] \max_{\pi_{\theta}}\mathbb{E}_{o\sim\pi_{\theta}(q)}\left[r_{\phi}(q,o)-\beta\mathrm{KL}[\pi_{\theta}(o|q)\|\pi_{\text{ref}}(o|q)]\right] (3) Online RL algorithms like PPO ( Schulman et al., 2017 ) generate samples from π θ \pi_{\theta} , evaluate them using r ϕ r_{\phi} , and update π θ \pi_{\theta} to maximize this objective. However, the reward model r ϕ r_{\phi} is crucial yet fragile; introducing it can lead to “reward hacking,” and retraining it is resource-intensive, complicating the training pipeline ( Gao et al., 2023 ) .

RLVR, on the other hand, substitutes the learned reward model with an automatically verifiable signal. This has proven effective in promoting reasoning capabilities, especially in domains like mathematics ( Guo et al., 2025 ) . The RLVR objective is: max π θ 𝔼 o ∼ π θ ​ ( q ) [ v ( q , o ) − β KL [ π θ ( o | q ) ∥ π ref ( o | q ) ] ] \max_{\pi_{\theta}}\mathbb{E}_{o\sim\pi_{\theta}(q)}\left[v(q,o)-\beta\mathrm{KL}[\pi_{\theta}(o|q)\|\pi_{\text{ref}}(o|q)]\right] (4) where v ⁡ ( q , o ) v(q,o) is a verifiable reward function. For instance, in mathematical problem-solving, v ⁡ ( q , o ) v(q,o) might be: v ⁡ ( q , o ) = { α if output ​ o ​ is correct 0 otherwise. v(q,o)=\begin{cases}\alpha&\text{if output }o\text{ is correct}\\ 0&\text{otherwise.}\end{cases} . RLVR is often implemented using algorithms like REINFORCE ( Williams, 1992 ) , PPO or GRPO. Despite their simplicity, verifiable rewards still rely on gold-standard answers or test executions, which are costly and domain-specific ( Liu et al., 2025 ; Team et al., 2025 ) . RLVR faces challenges in extending beyond math and code to tasks involving ambiguity or subjective reasoning.

## Appendix B Additional Experimental Details

### B.1 Influence of the KL Penalty

We further investigate how the magnitude of the KL penalty influences Intuitor , as shown in Table 3 . On in-domain benchmarks (MATH500 and GSM8K), the choice of penalty has only a minor effect, but on out-of-domain tasks—LiveCodeBench (code generation) and CRUXEval-O (code reasoning)—model accuracy is highly sensitive to this hyper-parameter. Because Intuitor does not receive explicit feedback from generated responses during training, the KL penalty serves as a critical regularization mechanism. It prevents the policy from drifting too far from the initial model distribution, acting as a safeguard against degeneration. These findings highlight the importance of careful KL tuning in general-purpose reinforcement learning setups, especially when targeting robust generalization across domains.

### B.2 Scaling to Larger Models

We extend Intuitor to larger base models, including Qwen2.5-7B, Qwen2.5-14B, and Qwen3-14B. However, we find that the original training recipe triggers severe behavioral collapse at the very start of training. Even before any updates, the 7B model solves the given problem and then immediately proceeds to tackle an unrelated one; this tendency becomes more pronounced as training progresses.

To stabilize learning, we simplify the system prompt, reduce the learning rate to 1 × 10 − 6 1\times 10^{-6} , and increase the number of sampled responses per problem to sixteen. These settings represent our first, untuned trial, and a comprehensive hyperparameter sweep is beyond the scope of this paper. Because the system prompt is the only additional signal the model receives during Intuitor fine‑tuning, we expect its careful calibration to exert a particularly strong influence on training dynamics. With these adjustments, Intuitor trains smoothly on both larger models. The corresponding evaluation results and training dynamics are reported in Table 4 and Figure 9 .

### B.3 Generalization Across Model Families

To assess the generalizability of Intuitor across different model families, we apply it to Llama3.2-3B-Instruct ( Meta AI, 2024 ) and the fully open OLMo-2-1124-7B-SFT model ( OLMo et al., 2024 ) .

As shown in Table 5 and Figure 10 , Intuitor improves the performance of Llama3.2, with both accuracy and response length showing steady improvement throughout the training process, indicating meaningful optimization gains under Intuitor .

Similarly, results on OLMo-2 (Table 6 and Figure 11 ) confirm that Intuitor provides consistent training improvements. These experiments demonstrate its robustness and applicability beyond the Qwen model family. Furthermore, since OLMo-2 is a fully open-source model with available training data and code, it also addresses concerns about data contamination in the evaluation dataset.

### B.4 Comparison with Alternative Reward Signals

Contemporary research has found that applying a negative token-level entropy reward can improve a model’s reasoning performance without requiring external labels ( Agarwal et al., 2025 ; Prabhudesai et al., 2025 ) . However, since low entropy often correlates with repetitive loops ( Holtzman et al., 2019 ) , using negative entropy alone as an RL reward risks driving the model into a collapsed state. In other words, without sufficient supervised training to push the base model away from degenerate behavior, the model risks falling into a repetition trap from which it cannot recover. As we observe a nontrivial amount of repetitive responses in Qwen2.5‐1.5B, we test this hypothesis by applying GRPO with the negative‐entropy reward: u EM = − 1 | o | ⋅ | 𝒱 | ∑ i = 1 | o | ∑ j = 1 | 𝒱 | p π θ ( j | q , o < i ) ⋅ log ( p π θ ( j | q , o < i ) ) . \displaystyle u_{\text{EM}}=-\frac{1}{|o|\cdot|\mathcal{V}|}\sum_{i=1}^{|o|}\sum_{j=1}^{|\mathcal{V}|}p_{\pi_{\theta}}(j|q,o_{<i})\cdot\log\left(p_{\pi_{\theta}}(j|q,o_{<i})\right). Figure 12 (left) validates our prediction. Entropy minimization (EM) exacerbates repetition, and after a few updates, the model converges to producing the same character regardless of the prompt. By contrast, Intuitor enhances performance without triggering collapse (Figure 4 ). Even when the base model is sufficiently strong to avoid collapse during the early stages of entropy minimization training, it remains more prone to later degeneration because entropy provides a weaker confidence signal compared to self-certainty. As shown in Figure 13 , we train both EM and Intuitor under identical settings using Qwen2.5-3B for two epochs. The results show that while both methods initially reach similar peak performance, Intuitor stabilizes around this peak, whereas EM exhibits a steady decline, with a consistent bias toward longer responses. These findings highlight self-certainty as a more robust and effective signal for RLIF.

To further validate the efficacy of Intuitor , we also trained Qwen2.5‐3B using a random reward baseline ( Shao et al., 2025 ) , where each response was assigned a reward of 0 or 1 with equal probability. Figure 12 (right) shows that this random reward scheme severely degrades the model’s performance in a chat‐style RL setting, demonstrating that the performance gains observed with Intuitor are indeed non‐trivial.

Sharpening mechanisms ( Huang et al., 2025 ) have also been proposed to improve the policy, which use the logarithm of the probability assigned to the completion as the reward, log ⁡ ( ∏ i = 1 | o | p π θ ​ ( o i ∣ q , o < i ) ) \log\left(\prod_{i=1}^{|o|}p_{\pi_{\theta}}(o_{i}\mid q,o_{<i})\right) . However, this unnormalized probability is inherently length-biased toward shorter completions, since each conditional probability is at most one, and their product decreases with sequence length. This effect is especially pronounced for long chains of reasoning. We conducted an experiment using this reward under the same setup as before, training Qwen2.5-3B with the GRPO loss. As shown in Figure 14 , both the completion length and the reward decrease rapidly as training progresses, indicating degeneration of the policy. This empirical observation is consistent with our analysis. We further tested a length-normalized variant, 1 | o | ​ ∑ i = 1 | o | log ⁡ p π θ ​ ( o i ∣ q , o < i ) \frac{1}{|o|}\sum_{i=1}^{|o|}\log p_{\pi_{\theta}}(o_{i}\mid q,o_{<i}) , as the reward on Qwen2.5-1.5B. While normalization removes the short-length bias, it introduces the opposite tendency. The model can increase reward by producing longer completions. This reward is quickly exploited and training destabilizes. In contrast, Intuitor consistently improves accuracy on both models, demonstrating substantially stronger robustness.

### B.5 Ablation on Optimization Strategy: Policy Gradient vs. Direct Optimization

One possible approach is to optimize self-certainty directly by minimizing the negative self-certainty as a loss function. Although this strategy rapidly increases the target metric, it creates an incentive for reward hacking in which the model inflates its own certainty without genuine improvement in task performance. As illustrated in Figure 15 , direct optimization produces an initial rise in accuracy, suggesting that self-certainty is correlated with useful learning signals, but it ultimately results in model collapse. By comparison, the advantage weighted gradient policy optimization implemented in Intuitor incorporates self-certainty only as a relative weighting factor. This formulation mitigates reward hacking, stabilizes the optimization process, and consistently achieves superior performance relative to direct optimization.

### B.6 The Effect of Rollout Size

To examine how the number of rollouts per question affects the training of Intuitor , we conduct an additional experiment using a rollout size of 14 and compare the resulting validation accuracy with previous settings. As shown in Table 7 , increasing the rollout size leads to higher accuracy on both GSM8K and MATH500. A larger rollout size reduces the variance of the advantage estimates computed using self-certainty, which in turn improves training stability and generalization.

### B.7 Attempts at Combining Golden Answers and Self-Certainty

We also investigate whether combining self-certainty with golden-answer supervision can yield further performance improvements. Our first attempt uses a simple weighted sum of the advantages from Intuitor and GRPO, forming a combined advantage: A ′ = 1 2 ​ A Intuitor + 1 2 ​ A GRPO A^{\prime}=\tfrac{1}{2}A_{\textsc{Intuitor}}+\tfrac{1}{2}A_{\text{GRPO}} However, as shown in Table 8 , this straightforward combination not only fails to improve accuracy but in fact performs worse than GRPO alone.

Next, we experiment with a two-stage training scheme. We first train using Intuitor on MATH for one epoch, then switch to GRPO for an additional epoch. Interestingly, this alternating approach yields better performance than training with GRPO for two full epochs. One possible explanation is that Intuitor helps the model establish more confident and coherent reasoning trajectories, allowing subsequent GRPO training to better identify and reinforce correct reasoning traces. A more thorough investigation is needed to develop principled methods for combining these two types of signals, which remains a promising direction for future research.

### B.8 Sequential Training Across Domains

We further investigate how training on one domain with Intuitor affects subsequent training on another domain. Specifically, we compare Qwen2.5-3B trained directly on the MATH dataset using Intuitor with a model first trained on Codeforces and then fine-tuned on MATH using Intuitor . As shown in Table 9 , the model that was pretrained on Codeforces achieves higher accuracy on both GSM8K and MATH500 after being trained on MATH. This suggests that Intuitor training on one domain does not hinder later learning on another domain. In fact, pretraining on Codeforces appears to improve downstream mathematical reasoning performance.

### B.9 Standard Deviation of Response Correctness

To assess whether training with Intuitor reduces the variance of model responses, we track during training the average standard deviation of correctness within each rollout group and the standard deviation of correctness across all completions at each step. As shown in Figure 16 , the step-wise standard deviation remains largely stable for both methods, exhibiting minimal fluctuation. Meanwhile, the average within-group standard deviation decreases under both training procedures and converges to a similar level. Overall, we find no strong evidence that Intuitor reduces within-group correctness variance more aggressively than GRPO.

### B.10 Failure Case Analysis