# Learning to Reason without External Rewards

###### Abstract

Training large language models (LLMs) for complex reasoning via Reinforcement Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on costly, domain-specific supervision. We explore Reinforcement Learning from Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic signals without external rewards or labeled data. We propose Intuitor , an RLIF method that uses a model’s own confidence—termed self-certainty —as its sole reward signal. Intuitor replaces external rewards in Group Relative Policy Optimization (GRPO) with self-certainty scores, enabling fully unsupervised learning. Experiments demonstrate that Intuitor matches GRPO’s performance on mathematical benchmarks while achieving better generalization to out-of-domain tasks like code generation, without requiring gold solutions or test cases. Our findings show that intrinsic model signals can drive effective learning across domains, offering a scalable alternative to RLVR for autonomous AI systems where verifiable rewards are unavailable. Code is available at https://github.com/sunblaze-ucb/Intuitor .

## 1 Introduction

Reinforcement learning has become essential for enhancing large language model capabilities. Early work focused on Reinforcement Learning from Human Feedback (RLHF), which aligns model outputs with human values through reward models trained on preference data ( Ouyang et al., 2022 ) . Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) replace learned reward models with automatically verifiable signals, such as exact answer matching in mathematical problem-solving, demonstrating improved reasoning capabilities in models like DeepSeek-R1 ( Guo et al., 2025 ; Lambert et al., 2024 ) .

Despite these successes, both RLHF and RLVR face fundamental limitations that constrain their broader applicability. RLHF requires extensive human annotation, making it expensive and potentially biased ( Gao et al., 2023 ) . RLVR, while avoiding learned reward models, demands domain-specific verifiers and gold-standard solutions. In mathematics, this requires expert annotation of solutions; in code generation, it necessitates comprehensive test suites and execution environments ( Liu et al., 2023 ; Liu and Zhang, 2025 ; Team et al., 2025 ; Xiaomi, 2025 ) . These requirements limit RLVR to carefully curated domains and complicate deployment in open-ended scenarios. Moreover, outcome-oriented verifiable rewards limit transferability to other domains. These challenges motivate exploration of more general and scalable reward paradigms, leading to a critical research question: Can LLMs enhance their reasoning abilities by relying solely on intrinsic, self-generated signals, without recourse to external verifiers or domain-specific ground truth?

In this paper, we introduce and explore such a paradigm: Reinforcement Learning from Internal Feedback (RLIF) , where models optimize intrinsic feedback to improve performance without external rewards or supervision. The motivation for RLIF extends to future scenarios where models develop superhuman capabilities that become difficult for humans to evaluate directly ( Burns et al., 2023 ) , requiring self-improvement through intrinsic mechanisms ( Oudeyer and Kaplan, 2007 ) .

Under the RLIF paradigm, we propose Intuitor , a novel reinforcement learning approach leveraging a model’s own confidence as an intrinsic reward. This builds on observations that LLMs exhibit lower confidence on difficult problems ( Farquhar et al., 2024 ; Kuhn et al., 2023 ; Kang et al., 2024 ; Kang et al., 2025 ) ; optimizing for confidence should improve reasoning capabilities. Specifically, we use self-certainty ( Kang et al., 2025 ) , the average KL divergence between the model’s output distribution and a uniform distribution, as our confidence measure. This metric has proven useful for distinguishing high-quality responses from flawed ones ( Kang et al., 2025 ; Ma et al., 2025 ) . Building on this insight, Intuitor guides learning through self-generated signals, eliminating the need for external supervision or handcrafted rewards. The implementation of Intuitor is simple, efficient, and effective: we replace the verifiable reward signal in existing RLVR frameworks, specifically Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) , with self-certainty scores, using the same policy gradient algorithm.

Our experiments demonstrate promising results. On the MATH dataset ( Hendrycks et al., 2021 ) with Qwen2.5-3B base ( Yang et al., 2024 ) , Intuitor matches the performance of GRPO without relying on any gold answers. As Intuitor rewards the generation trajectory rather than only the end result, it generalizes more effectively: training a Qwen2.5-3B base model on MATH yields a 65 % 65\% relative improvement on LiveCodeBench Code generation task ( Jain et al., 2024 ) versus no improvement for GRPO, and a 76 % 76\% gain on CRUXEval-O ( Gu et al., 2024 ) compared with 44 % 44\% for GRPO. Additionally, when we fine-tune the Qwen2.5-1.5B base model with Intuitor on the MATH corpus, a model that originally produces repetitive content and scores 0 % 0\% on LiveCodeBench learns to emit coherent reasoning chains and well-structured code, reaching 9.9 % 9.9\% accuracy after the tuning. Beyond the Qwen family, experiments with Llama ( Meta AI, 2024 ) and OLMo ( OLMo et al., 2024 ) models also show impressive gains, underscoring the strong generalization capabilities of Intuitor . As Intuitor requires only a clear prompt and no verifiable reward, it is broadly applicable across tasks, providing fresh evidence that pretrained LLMs possess richer latent behavioral priors than previously recognized.

Our contributions can be summarized as follows: • We introduce and explore Reinforcement Learning from Internal Feedback (RLIF), a novel reinforcement learning paradigm enabling LLMs to improve reasoning skills by leveraging intrinsic, self-generated signals, without reliance on external supervision or labeled data.

• We introduce Intuitor , an RLIF-based method that utilizes a model’s own internal confidence measure—termed self-certainty —as the sole intrinsic reward.

• We demonstrate that Intuitor matches supervised RL performance on in-domain tasks and achieves competitive, sometimes better out-of-domain generalization. We uncover emergent structured reasoning and enhanced instruction-following capabilities induced by intrinsic rewards.

## 2 Related Work

#### Reinforcement Learning from Human Feedback (RLHF).

RL has become instrumental in refining LLMs. Early pivotal work centered on Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ) , which aligns LLMs with human values by training a reward model on human preference data. While effective, RLHF is often resource-intensive due to the need for extensive human annotation ( Touvron et al., 2023 ) . Subsequent innovations like Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) aimed to simplify this by directly training models on preferences. The reliance on human-generated or model-approximated human preferences poses scalability challenges and introduces potential biases from the reward model itself ( Gao et al., 2023 ) .

#### Reinforcement Learning with Verifiable Rewards (RLVR).

RLVR emerged as a powerful alternative, particularly for tasks with clear correctness criteria like mathematical reasoning and code generation ( Hu et al., 2025 ; Team et al., 2025 ; Xiaomi, 2025 ) . RLVR utilizes rule-based verification functions, such as exact answer matching ( Guo et al., 2025 ; Team et al., 2025 ; Xiaomi, 2025 ; Jaech et al., 2024 ) , to provide reward signals, thereby avoiding the complexities and potential pitfalls of learned reward models. This approach has sparked significant advances, with models like DeepSeek-R1 ( Guo et al., 2025 ) achieving state-of-the-art reasoning capabilities. The development of robust policy optimization algorithms like GRPO ( Shao et al., 2024 ) and its variants ( Luo et al., 2025 ; Liu et al., 2025 ) has further solidified RLVR’s success. Nevertheless, RLVR’s applicability is largely confined to domains where verifiable gold solutions or exhaustive test cases can be constructed, and its predominant focus on outcome-based rewards can limit generalization to dissimilar tasks or those requiring nuanced, process-oriented feedback.

#### Intrinsic Signals and Self-Play in LLM Optimization.

Self-play and intrinsic rewards enable autonomous model improvement. Methods like SPIN ( Chen et al., 2024 ) and Self-Rewarding LMs ( Yuan et al., 2024 ) use the model itself for feedback. Earlier work like STaR ( Zelikman et al., 2022 ) relies on outcome evaluation, while others explore procedural generalization ( Poesia et al., 2024 ; Cheng et al., 2024 ) . Concurrent works such as Genius, TTRL, SRT, and Absolute Zero ( Xu et al., 2025 ; Zuo et al., 2025 ; Shafayat et al., 2025 ; Zhao et al., 2025 ) leverage unlabeled queries for RL but are often restricted to specific task distributions. Song et al. (2025) examine LLM self-improvement through the generation–verification gap, while Huang et al. (2025) study it through the lens of sharpening dynamics. Intuitor aligns with this direction, offering a lightweight, general-purpose approach using self-certainty as a confidence-based intrinsic reward, enabling single-agent RL across diverse tasks without explicit feedback or gold labels.

## 3 Method

### 3.1 Reinforcement Learning from Internal Feedback (RLIF)

To overcome the limitations of RLHF’s costly human annotation and RLVR’s domain-specific supervision, we propose Reinforcement Learning from Internal Feedback (RLIF). Instead of depending on external evaluation, RLIF uses the model’s own assessment of its outputs as feedback. This offers several advantages: it reduces reliance on supervision infrastructure, provides task-agnostic reward signals, and supports learning in domains where external verification is unavailable. The optimization objective for policy π θ \pi_{\theta} is: max π θ 𝔼 o ∼ π θ ​ ( q ) [ u ( q , o ) − β KL [ π θ ( o | q ) ∥ π ref ( o | q ) ] ] \max_{\pi_{\theta}}\mathbb{E}_{o\sim\pi_{\theta}(q)}\left[u(q,o)-\beta\mathrm{KL}[\pi_{\theta}(o|q)\|\pi_{\text{ref}}(o|q)]\right] (1) where q q is an input query, o o is the generated output, π ref \pi_{\text{ref}} is an initial reference policy, and β \beta controls the KL divergence to prevent excessive deviation from π ref \pi_{\text{ref}} . Here, u ⁡ ( q , o ) u(q,o) is an intrinsic signal derived from the model’s internal state or computation, rather than external verification. The key challenge lies in identifying intrinsic signals that correlate with output quality and can effectively guide learning.

Concurrent research explores related concepts within the RLIF paradigm. For example, Entropy Minimized Policy Optimization (EMPO) ( Zhang et al., 2025 ) minimizes LLM predictive entropy on unlabeled questions in a latent semantic space. SEED-GRPO ( Chen et al., 2025 ) uses the semantic entropy of generated sequences, combined with ground truth rewards, to modulate policy updates. Reinforcement Learning with a Negative Entropy Reward (EM-RL) ( Agarwal et al., 2025 ) employs a reward signal based solely on the negative sum of token-level entropy, akin to REINFORCE but without labels. These methods highlight the growing interest and potential of leveraging intrinsic signals for LLM training under the RLIF framework.

### 3.2 Intuitor : Policy Optimization with Self-Certainty

We propose Intuitor , a novel RLIF method that utilizes a model’s own confidence as the sole intrinsic reward signal u ⁡ ( q , o ) u(q,o) . Our choice of model confidence as the intrinsic reward is motivated by observations that LLMs often exhibit lower confidence when encountering unfamiliar tasks or lacking sufficient knowledge ( Kang et al., 2024 ) . Conversely, higher confidence frequently correlates with correctness. By rewarding increased self-confidence, Intuitor encourages to iteratively “practice” and refine its reasoning pathways until it becomes more confident in its outputs.

We adopt the self-certainty metric from Kang et al. (2025) , defined as the average KL divergence between a uniform distribution U U over the vocabulary 𝒱 \mathcal{V} and the model’s next-token distribution: Self-certainty ( o | q ) ≔ 1 | o | ∑ i = 1 | o | KL ( U ∥ p π θ ( ⋅ | q , o < i ) ) = − 1 | o | ⋅ | 𝒱 | ∑ i = 1 | o | ∑ j = 1 | 𝒱 | log ( | 𝒱 | ⋅ p π θ ( j | q , o < i ) ) \displaystyle\textbf{Self-certainty}(o|q)\coloneqq\frac{1}{|o|}\sum_{i=1}^{|o|}\mathrm{KL}(U\parallel p_{\pi_{\theta}}(\cdot|q,o_{<i}))=-\frac{1}{|o|\cdot|\mathcal{V}|}\sum_{i=1}^{|o|}\sum_{j=1}^{|\mathcal{V}|}\log\left(|\mathcal{V}|\cdot p_{\pi_{\theta}}(j|q,o_{<i})\right) where o < i o_{<i} are the previously generated tokens and p ⁡ ( j | q , o < i ) p(j|q,o_{<i}) is the model’s predicted probability for token j j at step i i . Higher self-certainty values indicate greater confidence.

Self-certainty is related to a KL divergence where the model’s prediction is the second argument, KL ( U ∥ p π θ ) \mathrm{KL}(U\parallel p_{\pi_{\theta}}) . This contrasts with entropy (or reverse KL divergence from uniform). Critically, self-certainty is reported to be less prone to biases towards longer generations, a common issue with perplexity or entropy-based measures ( Fang et al., 2024 ; Kang et al., 2025 ) , making it a potentially more reliable indicator of intrinsic confidence. Kang et al. (2025) demonstrate that self-certainty is effective for selecting high-quality answers from multiple candidates, and uniquely among different confidence measures, its utility improves with more candidates. Optimizing for self-certainty thus encourages the model to generate responses that it deems more convincing. The RL process can achieve this by, for instance, guiding the model to produce more detailed reasoning steps, thereby increasing the model’s conviction in its final answer. This mechanism is more nuanced than simply increasing the probability of the most likely output; it involves modifying the generation process itself to build confidence.

To optimize the objective in Equation 1 , various policy gradient algorithms can be employed. Informed by the recent success in models such as DeepSeek-R1 ( Guo et al., 2025 ) and its widespread adoption of GRPO in open-source projects, we utilize GRPO to optimize for self-certainty. The overall pipeline for this GRPO-based instantiation of Intuitor is illustrated in Figure 2 .

The core idea behind the optimization is to sample multiple candidate outputs for a given query and use their relative rewards to estimate advantages for policy updates. For each query q ∼ P ⁡ ( Q ) q\sim P(Q) , GRPO samples a group of G G outputs o 1 , … , o G {o_{1},\ldots,o_{G}} using a behavior policy π θ old \pi_{\theta_{\text{old}}} (e.g., a previous iteration or the SFT model). The target policy π θ \pi_{\theta} is then optimized by maximizing: 𝒥 GRPO ( θ ) = 𝔼 q ∼ P ⁡ ( Q ) , { o i } i = 1 G ∼ π θ old ( ⋅ | q ) [ 1 G ∑ i = 1 G 1 | o i | ∑ t = 1 | o i | ( min [ c i , t ( θ ) A ^ i , t , clip ε ( c i , t ( θ ) ) A ^ i , t ] − β 𝔻 KL ( π θ ∥ π ref ) ) ] \displaystyle\mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}_{\begin{subarray}{c}q\sim P(Q),\\ \{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|q)\end{subarray}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\left(\min\left[c_{i,t}(\theta)\hat{A}_{i,t},\;\operatorname{clip}_{\varepsilon}\left(c_{i,t}(\theta)\right)\hat{A}_{i,t}\right]-\beta\mathbb{D}_{\mathrm{KL}}\bigl(\pi_{\theta}\|\pi_{\mathrm{ref}}\bigr)\right)\right]

where c i , t ​ ( θ ) = π θ ​ ( o i , t ∣ q , o i , < t ) π θ old ​ ( o i , t ∣ q , o i , < t ) c_{i,t}(\theta)=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})} is the importance weight, clip ε \operatorname{clip}_{\varepsilon} is the function that clips to [ 1 − ε , 1 + ε ] [1-\varepsilon,1+\varepsilon] . Hyperparameters ϵ \epsilon (for clipping) and β \beta (for KL penalty strength) control stability and exploration, and A ^ i , t \hat{A}_{i,t} is the advantage estimate.

Integration of Self-Certainty. The key innovation in Intuitor is replacing external rewards with self-certainty scores in GRPO’s advantage computation. Specifically, each output o i o_{i} is scored by: u i = Self-certainty ​ ( o i | q ) , A ^ i , t = u i − mean ​ ( { u 1 , u 2 , ⋯ , u G } ) std ​ ( { u 1 , u 2 , ⋯ , u G } ) . u_{i}=\text{Self-certainty}(o_{i}|q),\quad\hat{A}_{i,t}=\frac{u_{i}-\text{mean}(\{u_{1},u_{2},\cdots,u_{G}\})}{\text{std}(\{u_{1},u_{2},\cdots,u_{G}\})}. (2) This formulation enables the policy to favor outputs that the model itself considers more confident. The complete Intuitor training pipeline operates by sampling multiple candidate outputs for each query, computing self-certainty scores for each candidate, using these scores to estimate advantages within the group, and updating the policy to increase the likelihood of generating high-confidence outputs. This process requires no external supervision, making it a self-reinforcing learning loop.

## 4 Experimental Setup

Training Setup. Both GRPO and Intuitor are trained with the Open-R1 framework ( Face, 2025 ) on the training split of the MATH dataset ( Hendrycks et al., 2021 ) , which contains 7,500 problems. We use Qwen2.5-1.5B and Qwen2.5-3B ( Yang et al., 2024 ) as backbone models, with a chat-based prompting format throughout. Given the models’ initially weak instruction-following abilities, we do not require them to disentangle intermediate reasoning from final answers. Each update processes 128 problems, generating 7 candidate solutions per problem, with a default KL penalty of β = 0.005 \beta=0.005 . For a fair comparison, GRPO and Intuitor share identical hyperparameters (see Appendix) without additional tuning. We also evaluate a GRPO variant, denoted GRPO-PV in Table 1 , which uses plurality voting 1 1 1 Self-consistency uses a plurality rule, selecting the most frequent answer even without majority support, while majority voting requires > 50 % >50\% support and otherwise yields no winner ( De Condorcet and others, 2014 ) . as a proxy for ground truth. This follows the approach from TTRL ( Zuo et al., 2025 ) , which shows that self-consistency-based rewards can match the performance of golden answers when training on inference data.

Intuitor for Code Generation ( Intuitor -Code). To assess generalization beyond mathematical reasoning, we apply Intuitor to the Codeforces code generation dataset ( Li et al., 2022 ) . This variant, denoted Intuitor -Code in Table 1 , modifies the setup as follows: the number of sampled completions per problem is increased to 14; the learning rate is reduced from 3 × 10 − 5 3\times 10^{-5} to 1 × 10 − 5 1\times 10^{-5} ; and the KL penalty is increased to β = 0.01 \beta=0.01 . For simplicity, we limit the run to 50 steps, utilizing a total of 3,200 problems.

Evaluation. Evaluations generally use the same chat-style prompting format as in training, except for MMLU-Pro ( Wang et al., 2024 ) , where we follow the benchmark’s original prompt format. Greedy decoding is used for all completions. Experiments were conducted on NVIDIA A100 GPUs, each with 40GB of memory. We evaluate performance on the following benchmarks (1) Math reasoning : MATH500 and GSM8K, using the lighteval library ( Habib et al., 2023 ) . (2) Code reasoning : CRUXEval-O ( Gu et al., 2024 ) , using the ZeroEval framework ( Lin, 2024 ) , and LiveCodeBench v6 (LCB) ( Jain et al., 2024 ) . (3) Instruction following : AlpacaEval 2.0 with length-controlled win rates ( Dubois et al., 2024 ) , judged by GPT-4.1 ( OpenAI, 2025 ) .

## 5 Results and Analysis

We evaluate the effectiveness of Intuitor by addressing the following research questions: • (RQ1) How does the overall performance of Intuitor compare to supervised RLVR methods?

• (RQ2) How does intrinsic feedback influence the model’s qualitative behavior?

• (RQ3) How robust is online self-certainty when used as an intrinsic reward signal during training?

Table 1 presents main evaluation results, and Figure 4 illustrates response length evolution during training. On in-domain MATH and GSM8K datasets, Intuitor and GRPO-PV (both golden-answer-free) achieve performance comparable to GRPO (using golden answers). This aligns with TTRL ( Zuo et al., 2025 ) , where plurality voting approximated golden answers without significant performance loss. While Intuitor performs slightly worse than GRPO overall, on MATH it produces longer responses and demonstrates markedly improved code generation, suggesting enhanced reasoning capabilities.

### 5.1 Learning to Follow Instructions

Intuitor significantly enhances instruction-following. Initially, the pretrained Qwen2.5-1.5B struggles with chat-style prompts, scoring < < 10% on all chat-template tasks (Table 1 ) and generating repetitive, nonsensical output, which inflates average response lengths (Figure 4 ). Fine-tuning with Intuitor sharply reduces such gibberish, decreases completion lengths, and enables non-trivial performance across all evaluated benchmarks. Furthermore, on the MATH dataset, Intuitor substantially improves the Length Control Win Rate on AlpacaEval for both Qwen2.5-1.5B and Qwen2.5-3B, surpassing GRPO under identical settings. This demonstrates robust gains in instruction adherence.

### 5.2 Fostering Structured Reasoning

Rapid Initial Learning. Self-certainty, a continuous and inherently process-aware reward derived from the model’s internal assessment across all tokens, contrasts with binary rewards. This internal signal may encourage LLMs to follow more effective learning trajectories. Given comparable final performance between GRPO and Intuitor , we assess early-stage learnability by comparing in-domain accuracy at training step 10. As shown in Table 2 , Intuitor consistently outperforms GRPO on both GSM8K and MATH benchmarks for Qwen2.5-1.5B and Qwen2.5-3B, highlighting its advantage in rapid initial learning.

Cross-Task Generalization. Figure 4 illustrates performance trajectories on MATH500 (in-domain) and LiveCodeBench (transfer task) for models trained on the MATH dataset. For both Intuitor and GRPO, accuracy improvements on LiveCodeBench emerge later in training, following initial gains on MATH500. Notably, LiveCodeBench performance continues to improve even after MATH500 accuracy plateaus. This pattern suggests that initial in-domain learning (on MATH) facilitates subsequent generalization to code generation tasks (LiveCodeBench).

Emergence of Long-Form Reasoning. While large models like Deepseek-R1 achieve long-form reasoning through extensive RL, Intuitor enables smaller models to develop structured reasoning with limited data. On CRUXEval-O (Figure 5 ), models trained with Intuitor often exhibit free-form reasoning before summarizing it within the instructed JSON block, despite prompts requiring reasoning directly in JSON. A similar pattern of pre-code natural language reasoning is observed on LiveCodeBench. This emergent pre-reasoning may contribute to Intuitor ’s strong performance on these benchmarks.

### 5.3 Understanding Emergent Long-Form Reasoning

When LLMs encounter unfamiliar questions, they sample from a distribution of possible answers ( Kang et al., 2024 ) . Self-certainty reflects the model’s internal assessment of its output coherence. By reinforcing high-confidence responses, Intuitor encourages more elaborate reasoning, potentially improving the model’s comprehension of its own outputs. While not explicitly targeting benchmark accuracy, this enhancement in output quality and structure leads to more reliable answers and better generalization.

We analyze models trained with Intuitor on code corpora by examining outputs for ten randomly selected LiveCodeBench questions across different training steps. Figure 6 shows the evolution of output types alongside model accuracy. The results reveal a clear progression: models first learn to generate valid Python code (evidenced by improved accuracy and fewer invalid responses), then develop pre-code reasoning to facilitate self-understanding. Further inspection of generations confirms that models progressively elaborate their reasoning throughout training, supporting our hypothesis that Intuitor encourages traces that the model itself can better understand.

To quantify this effect, we classify outputs from successive checkpoints into three categories: invalid code (”No Answer”), valid code without reasoning (”No Reasoning”), and valid code with explicit reasoning (”Reasoning”). Figure 6 (a) illustrates how these proportions evolve during training alongside LiveCodeBench accuracy. The model first reduces invalid outputs and improves code correctness before incorporating pre-code reasoning, reflecting an emergent emphasis on self-explanatory traces. Figure 6 (b) demonstrates how training with Intuitor leads to structured reasoning before code generation. Additional evidence appears in Figure 8 , where Intuitor -trained models assign significantly higher confidence to their generated responses compared to baseline models, as discussed further in Section 5.4 .

### 5.4 Online Self-Certainty Prevents Reward Exploitation

Over-optimization against static reward models is a known failure mode in reinforcement learning ( Gao et al., 2023 ) . To assess the robustness of self-certainty as a reward, we compare offline self-certainty (rewards from a fixed base model) with online self-certainty (rewards from the evolving policy model), using a reduced batch size of 224 responses per gradient update.

Figure 7 demonstrates that the offline annotator is susceptible to exploitation. Around the 100th update step, the policy model learns to inflate its self-certainty reward by appending an auxiliary, already-solved problem to its answer for the given question. This exploitation manifests as a sharp increase in response length (dashed line) and a concurrent collapse in validation accuracy (solid line). In contrast, the online annotator, whose reward signal co-evolves with the policy, prevents such reward hacking and maintains stable training.

To further evaluate the quality of self-certainty as a reward signal, we analyze the distribution of self-certainty scores from policies trained with Intuitor and GRPO on MATH500 responses (Figure 8 ). We employ Mann–Whitney U tests to determine if correct responses achieve significantly higher self-certainty scores than incorrect ones. Both GRPO and Intuitor models exhibit significantly higher average self-certainty scores, indicating that GRPO also enhances the model’s self-assessment capabilities. Notably, policies trained with online self-certainty (i.e., Intuitor ) show no signs of reward hacking. The Intuitor policy yields the lowest p p -values and largest effect sizes ( r r ) in the Mann-Whitney U tests (Figure 8 , inset). This indicates it is most effective at discriminating its own correct and incorrect answers using self-certainty, even while assigning higher absolute confidence scores overall. These findings underscore the potential of Intuitor for robust training on larger datasets.

### 5.5 Ablation Studies

To comprehensively validate Intuitor ’s design and robustness, we conducted extensive ablation studies, with full details provided in Appendix B due to page limitations. Key findings are: (1) KL term: Varying the KL penalty (Sec. B.1 ) shows a stability–performance trade-off; moderate values yield the best accuracy. (2) Scaling: Intuitor scales to larger backbones (Qwen2.5-7B/14B, Qwen3-14B; Sec. B.2 ), delivering consistent gains in reasoning and generalization. (3) Architecture: On Llama-3.2 and OLMo-2 (Sec. B.3 ), Intuitor remains effective, indicating robustness across model families and sizes. (4) Reward design: Compared to entropy minimization ( Agarwal et al., 2025 ) and random rewards ( Shao et al., 2025 ) , Intuitor yields stable improvements, while the alternatives trigger catastrophic collapse (Sec. B.4 ). (5) Optimization strategy: Directly optimizing self-certainty as a loss function leads to reward hacking and performance collapse; our advantage-weighted policy-gradient formulation avoids this and trains reliably (Sec. B.5 ).

## 6 Discussion and Future Research

Scalability and Generalization. Our experiments, constrained by computational resources, utilize relatively compact models trained on relatively small, unsupervised corpora. We aim to demonstrate the potential of a model’s self-certainty as a reward signal for policy optimization. The results show that this signal consistently promotes more coherent, well-justified, and interpretable explanations, indicating a path towards more autonomous learning. Future work could explore these benefits in larger foundation models (with hundreds of billions of parameters) and on more diverse, real-world datasets. Given that purely offline training with Intuitor led to performance degradation over time, scaling up will likely require periodic online updates to self-certainty estimates or hybrid offline-online schedules to maintain calibration.

Theoretical Analysis of RLIF. While we have empirically demonstrated the superior performance of using self-certainty as a reward for RLIF, the underlying theoretical mechanisms warrant further investigation. Huang et al. (2025) analyze LLM self-improvement as a “sharpening” mechanism, proposing a statistical framework to evaluate algorithm efficiency via sample complexity. Similarly, Yue et al. (2025) question whether RLVR functions primarily by sharpening the base model’s existing distribution. However, determining the theoretically optimal reward signal for RLIF and establishing the fundamental reasoning boundaries of LLMs remain open problems. These challenges highlight the need for future theoretical research to complement empirical findings.

Combining Reward Signals. To enable a direct comparison between self-certainty and golden-answer rewards, this paper focuses exclusively on a single reward signal. However, these signals are not mutually exclusive. Future work could explore combining them, for instance, by summation or by alternating based on the availability of golden answers. Furthermore, other reward signals, such as formatting rewards ( Guo et al., 2025 ) , could be additively combined to enhance performance. Integrating RLIF with methods like RLHF and RLVR may further advance LLM capabilities across various dimensions.

## 7 Conclusion

This paper introduces Intuitor , an instantiation of Reinforcement Learning from Internal Feedback (RLIF) that uses a model’s intrinsic self-certainty as its sole reward signal, eliminating the need for external supervision or gold-standard solutions. Our experiments show that Intuitor matches the performance of supervised RLVR methods like GRPO on mathematical reasoning and achieves competitive, sometimes better generalization to out-of-domain tasks such as code generation and instruction following. It also promotes structured reasoning and leverages online self-certainty to guard against reward exploitation. These findings highlight the transformative potential of RLIF, signaling a meaningful step toward AI systems that improve through introspection and unlock rich latent capabilities. Looking forward, this paradigm opens the door to AI agents capable of autonomous skill acquisition in novel domains and scalable self-improvement—even as they approach or surpass the limits of human oversight. Future directions include integrating RLIF with external reward methods like RLHF or RLVR to tackle increasingly complex real-world challenges, and advancing the development of more robust, generalizable, and truly autonomous learning systems.

## Ethics Statement

Our research is based on publicly available datasets and open-source language models, mitigating concerns related to private data or human subjects. The goal of our work is to enhance the reasoning capabilities of language models through self-supervision, which we believe is a positive step toward more transparent and robust AI systems. We have made our code publicly available to ensure transparency and allow for full scrutiny of our methods and findings. We do not foresee any direct negative societal impacts or ethical concerns arising from this work.

## Reproducibility Statement

To ensure the reproducibility of our results, we provide all source code and training configurations in https://github.com/sunblaze-ucb/Intuitor . The Experimental Setup section and Appendix B detail all hyperparameters, software versions (including the Open-R1 framework), and evaluation setups. Furthermore, Appendix C.1 includes the exact prompts used during training and evaluation. These resources should allow for the complete replication of our experiments and validation of our findings.

## References

Agarwal et al. (2025) S. Agarwal, Z. Zhang, L. Yuan, J. Han, and H. Peng The unreasonable effectiveness of entropy minimization in llm reasoning . arXiv preprint arXiv:2505.15134 . Cited by: §B.4 , §3.1 , §5.5 .

Burns et al. (2023) C. Burns, P. Izmailov, J. H. Kirchner, B. Baker, L. Gao, L. Aschenbrenner, Y. Chen, A. Ecoffet, M. Joglekar, J. Leike, et al. Weak-to-strong generalization: eliciting strong capabilities with weak supervision . arXiv preprint arXiv:2312.09390 . Cited by: §1 .

Chen et al. (2025) M. Chen, G. Chen, W. Wang, and Y. Yang SEED-grpo: semantic entropy enhanced grpo for uncertainty-aware policy optimization . arXiv preprint arXiv:2505.12346 . Cited by: §3.1 .

Chen et al. (2024) Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu Self-play fine-tuning converts weak language models to strong language models . arXiv preprint arXiv:2401.01335 . Cited by: §2 .

Cheng et al. (2024) P. Cheng, Y. Dai, T. Hu, H. Xu, Z. Zhang, L. Han, N. Du, and X. Li Self-playing adversarial language game enhances llm reasoning . Advances in Neural Information Processing Systems 37 , pp. 126515–126543 . Cited by: §2 .

De Condorcet et al. (2014) N. De Condorcet et al. Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix . Cambridge University Press . Cited by: footnote 1 .

Dubois et al. (2024) Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto Length-controlled alpacaeval: a simple way to debias automatic evaluators . arXiv preprint arXiv:2404.04475 . Cited by: §4 .

Face (2025) H. Face Open r1: a fully open reproduction of deepseek-r1 . External Links: Link Cited by: §4 .

Fang et al. (2024) L. Fang, Y. Wang, Z. Liu, C. Zhang, S. Jegelka, J. Gao, B. Ding, and Y. Wang What is wrong with perplexity for long-context language modeling? . arXiv preprint arXiv:2410.23771 . Cited by: §3.2 .

Farquhar et al. (2024) S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal Detecting hallucinations in large language models using semantic entropy . Nature 630 ( 8017 ), pp. 625–630 . Cited by: §1 .

Gao et al. (2023) L. Gao, J. Schulman, and J. Hilton Scaling laws for reward model overoptimization . In International Conference on Machine Learning , pp. 10835–10866 . Cited by: §A.1 , §1 , §2 , §5.4 .

Gu et al. (2024) A. Gu, B. Rozière, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. I. Wang Cruxeval: a benchmark for code reasoning, understanding and execution . arXiv preprint arXiv:2401.03065 . Cited by: §1 , §4 .

Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning . arXiv preprint arXiv:2501.12948 . Cited by: §A.1 , §1 , §2 , §3.2 , §6 .

Habib et al. (2023) N. Habib, C. Fourrier, H. Kydlíček, T. Wolf, and L. Tunstall LightEval: a lightweight framework for llm evaluation . External Links: Link Cited by: §4 .

Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the math dataset . arXiv preprint arXiv:2103.03874 . Cited by: §1 , §4 .

Holtzman et al. (2019) A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi The curious case of neural text degeneration . arXiv preprint arXiv:1904.09751 . Cited by: §B.4 .

Hu et al. (2025) J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H. Shum Open-reasoner-zero: an open source approach to scaling up reinforcement learning on the base model . arXiv preprint arXiv:2503.24290 . Cited by: §2 .

Huang et al. (2025) A. Huang, A. Block, D. J. Foster, D. Rohatgi, C. Zhang, M. Simchowitz, J. T. Ash, and A. Krishnamurthy Self-improvement in language models: the sharpening mechanism . In International Conference on Learning Representations (ICLR) , Cited by: §B.4 , §2 , §6 .

Jaech et al. (2024) A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai o1 system card . arXiv preprint arXiv:2412.16720 . Cited by: §2 .

Jain et al. (2024) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica Livecodebench: holistic and contamination free evaluation of large language models for code . arXiv preprint arXiv:2403.07974 . Cited by: §1 , §4 .

Kang et al. (2024) K. Kang, E. Wallace, C. Tomlin, A. Kumar, and S. Levine Unfamiliar finetuning examples control how language models hallucinate . arXiv preprint arXiv:2403.05612 . Cited by: §1 , §3.2 , §5.3 .

Kang et al. (2025) Z. Kang, X. Zhao, and D. Song Scalable best-of-n selection for large language models via self-certainty . In The Thirty-ninth Annual Conference on Neural Information Processing Systems , External Links: Link Cited by: §1 , §3.2 , §3.2 .

Kuhn et al. (2023) L. Kuhn, Y. Gal, and S. Farquhar Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation . arXiv preprint arXiv:2302.09664 . Cited by: §1 .

Lambert et al. (2024) N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. T \ \backslash ”ulu 3: pushing frontiers in open language model post-training . arXiv preprint arXiv:2411.15124 . Cited by: §1 .

Li et al. (2022) Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, T. Hubert, P. Choy, C. de Masson d’Autume, I. Babuschkin, X. Chen, P. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. Mankowitz, E. Sutherland Robson, P. Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals Competition-level code generation with alphacode . arXiv preprint arXiv:2203.07814 . Cited by: §4 .

Lin (2024) B. Y. Lin ZeroEval: A Unified Framework for Evaluating Language Models . External Links: Link Cited by: §4 .

Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation . Advances in Neural Information Processing Systems 36 , pp. 21558–21572 . Cited by: §1 .

Liu and Zhang (2025) J. Liu and L. Zhang Code-r1: reproducing r1 for code with reliable rewards . Note: https://github.com/ganler/code-r1 Cited by: §1 .

Liu et al. (2025) Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin Understanding r1-zero-like training: a critical perspective . arXiv preprint arXiv:2503.20783 . Cited by: §A.1 , §2 .

Luo et al. (2025) M. Luo, S. Tan, R. Huang, X. Shi, R. Xin, C. Cai, A. Patel, A. Ariyak, Q. Wu, C. Zhang, L. E. Li, R. A. Popa, and I. Stoica DeepCoder: a fully open-source 14b coder at o3-mini level . Note: https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51 Notion Blog Cited by: §2 .

Ma et al. (2025) W. Ma, J. He, C. Snell, T. Griggs, S. Min, and M. Zaharia Reasoning models can be effective without thinking . arXiv preprint arXiv:2504.09858 . Cited by: §1 .

Meta AI (2024) Meta AI Llama 3.2: revolutionizing edge ai and vision with open, customizable models . Note: https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ Accessed: 2025-05-16 Cited by: §B.3 , §1 .

OLMo et al. (2024) T. OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, N. Lambert, D. Schwenk, O. Tafjord, T. Anderson, D. Atkinson, F. Brahman, C. Clark, P. Dasigi, N. Dziri, M. Guerquin, H. Ivison, P. W. Koh, J. Liu, S. Malik, W. Merrill, L. J. V. Miranda, J. Morrison, T. Murray, C. Nam, V. Pyatkin, A. Rangapur, M. Schmitz, S. Skjonsberg, D. Wadden, C. Wilhelm, M. Wilson, L. Zettlemoyer, A. Farhadi, N. A. Smith, and H. Hajishirzi 2 olmo 2 furious . External Links: 2501.00656 , Link Cited by: §B.3 , §1 .

OpenAI (2025) OpenAI Introducing GPT‑4.1 in the API . Note: https://openai.com/index/gpt-4-1/ Accessed: 15 May 2025 Cited by: §4 .

Oudeyer and Kaplan (2007) P. Oudeyer and F. Kaplan What is intrinsic motivation? a typology of computational approaches . Frontiers in neurorobotics 1 , pp. 108 . Cited by: §1 .

Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback . Advances in neural information processing systems 35 , pp. 27730–27744 . Cited by: §A.1 , §1 , §2 .

Poesia et al. (2024) G. Poesia, D. Broman, N. Haber, and N. Goodman Learning formal mathematics from intrinsic motivation . Advances in Neural Information Processing Systems 37 , pp. 43032–43057 . Cited by: §2 .

Prabhudesai et al. (2025) M. Prabhudesai, L. Chen, A. Ippoliti, K. Fragkiadaki, H. Liu, and D. Pathak Maximizing confidence alone improves reasoning . External Links: 2505.22660 , Link Cited by: §B.4 .

Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model . Advances in Neural Information Processing Systems 36 , pp. 53728–53741 . Cited by: §2 .

Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms . arXiv preprint arXiv:1707.06347 . Cited by: §A.1 .

Shafayat et al. (2025) S. Shafayat, F. Tajwar, R. Salakhutdinov, J. Schneider, and A. Zanette Can large reasoning models self-train? . arXiv preprint arXiv:2505.21444 . Cited by: §2 .

Shao et al. (2025) R. Shao, S. S. Li, R. Xin, S. Geng, Y. Wang, S. Oh, S. S. Du, N. Lambert, S. Min, R. Krishna, et al. Spurious rewards: rethinking training signals in rlvr . arXiv preprint arXiv:2506.10947 . Cited by: §B.4 , §5.5 .

Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models . arXiv preprint arXiv:2402.03300 . Cited by: §1 , §2 .

Song et al. (2025) Y. Song, H. Zhang, C. Eisenach, S. Kakade, D. Foster, and U. Ghai Mind the gap: examining the self-improvement capabilities of large language models . In International Conference on Learning Representations (ICLR) , Cited by: §2 .

Team et al. (2025) K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. Kimi k1. 5: scaling reinforcement learning with llms . arXiv preprint arXiv:2501.12599 . Cited by: §A.1 , §1 , §2 .

Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: open foundation and fine-tuned chat models . arXiv preprint arXiv:2307.09288 . Cited by: §2 .

Wang et al. (2024) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. Mmlu-pro: a more robust and challenging multi-task language understanding benchmark . arXiv preprint arXiv:2406.01574 . Cited by: §4 .

Williams (1992) R. J. Williams Simple statistical gradient-following algorithms for connectionist reinforcement learning . Machine learning 8 , pp. 229–256 . Cited by: §A.1 .

Xiaomi (2025) Xiaomi MiMo: unlocking the reasoning potential of language model – from pretraining to posttraining . External Links: Link Cited by: §1 , §2 .

Xu et al. (2025) F. Xu, H. Yan, C. Ma, H. Zhao, Q. Sun, K. Cheng, J. He, J. Liu, and Z. Wu Genius: a generalizable and purely unsupervised self-training framework for advanced reasoning . arXiv preprint arXiv:2504.08672 . Cited by: §2 .

Yang et al. (2024) A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report . arXiv preprint arXiv:2412.15115 . Cited by: §1 , §4 .

Yuan et al. (2024) W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. E. Weston Self-rewarding language models . In Proceedings of the 41st International Conference on Machine Learning , Proceedings of Machine Learning Research , Vol. 235 , pp. 57905–57923 . Cited by: §2 .

Yue et al. (2025) Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, S. Song, and G. Huang Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? . arXiv preprint arXiv:2504.13837 . Cited by: §6 .

Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. Goodman Star: bootstrapping reasoning with reasoning . Advances in Neural Information Processing Systems 35 , pp. 15476–15488 . Cited by: §2 .

Zhang et al. (2025) Q. Zhang, H. Wu, C. Zhang, P. Zhao, and Y. Bian Right question is already half the answer: fully unsupervised llm reasoning incentivization . arXiv preprint arXiv:2504.05812 . Cited by: §3.1 .

Zhao et al. (2025) A. Zhao, Y. Wu, Y. Yue, T. Wu, Q. Xu, M. Lin, S. Wang, Q. Wu, Z. Zheng, and G. Huang Absolute zero: reinforced self-play reasoning with zero data . arXiv preprint arXiv:2505.03335 . Cited by: §2 .

Ziegler et al. (2019) D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving Fine-tuning language models from human preferences . arXiv preprint arXiv:1909.08593 . Cited by: §A.1 .

Zuo et al. (2025) Y. Zuo, K. Zhang, S. Qu, L. Sheng, X. Zhu, B. Qi, Y. Sun, G. Cui, N. Ding, and B. Zhou Ttrl: test-time reinforcement learning . arXiv preprint arXiv:2504.16084 . Cited by: §2 , §4 , §5 .

## LLM Usage Statement

Large Language Models were utilized solely as a general-purpose assist tool for paraphrasing and polishing the clarity, conciseness, and flow of the English writing in this paper. LLMs did not contribute to research ideation, experimental design, data analysis, or the generation of any core scientific content, arguments, or conclusions presented herein. The authors take full responsibility for all content within this submission.

## Appendix A Additional Background

### A.1 From External Supervision to Internal Feedback

To provide additional context, we review existing RL-based fine-tuning paradigms and their limitations, which motivate our exploration of Reinforcement Learning from Internal Feedback (RLIF).

Current RL fine-tuning approaches for LLMs primarily fall into two categories: those relying on external human feedback (RLHF) and those using verifiable, often task-specific, rewards (RLVR).

In RLHF ( Ziegler et al., 2019 ; Ouyang et al., 2022 ) , the policy π θ \pi_{\theta} is optimized to align with human preferences, typically encapsulated by a learned reward model r ϕ r_{\phi} . The objective is: max π θ 𝔼 o ∼ π θ ​ ( q ) [ r ϕ ( q , o ) − β KL [ π θ ( o | q ) ∥ π ref ( o | q ) ] ] \max_{\pi_{\theta}}\mathbb{E}_{o\sim\pi_{\theta}(q)}\left[r_{\phi}(q,o)-\beta\mathrm{KL}[\pi_{\theta}(o|q)\|\pi_{\text{ref}}(o|q)]\right] (3) Online RL algorithms like PPO ( Schulman et al., 2017 ) generate samples from π θ \pi_{\theta} , evaluate them using r ϕ r_{\phi} , and update π θ \pi_{\theta} to maximize this objective. However, the reward model r ϕ r_{\phi} is crucial yet fragile; introducing it can lead to “reward hacking,” and retraining it is resource-intensive, complicating the training pipeline ( Gao et al., 2023 ) .

RLVR, on the other hand, substitutes the learned reward model with an automatically verifiable signal. This has proven effective in promoting reasoning capabilities, especially in domains like mathematics ( Guo et al., 2025 ) . The RLVR objective is: max π θ 𝔼 o ∼ π θ ​ ( q ) [ v ( q , o ) − β KL [ π θ ( o | q ) ∥ π ref ( o | q ) ] ] \max_{\pi_{\theta}}\mathbb{E}_{o\sim\pi_{\theta}(q)}\left[v(q,o)-\beta\mathrm{KL}[\pi_{\theta}(o|q)\|\pi_{\text{ref}}(o|q)]\right] (4) where v ⁡ ( q , o ) v(q,o) is a verifiable reward function. For instance, in mathematical problem-solving, v ⁡ ( q , o ) v(q,o) might be: v ⁡ ( q , o ) = { α if output ​ o ​ is correct 0 otherwise. v(q,o)=\begin{cases}\alpha&\text{if output }o\text{ is correct}\\ 0&\text{otherwise.}\end{cases} . RLVR is often implemented using algorithms like REINFORCE ( Williams, 1992 ) , PPO or GRPO. Despite their simplicity, verifiable rewards still rely on gold-standard answers or test executions, which are costly and domain-specific ( Liu et al., 2025 ; Team et al., 2025 ) . RLVR faces challenges in extending beyond math and code to tasks involving ambiguity or subjective reasoning.

## Appendix B Additional Experimental Details

### B.1 Influence of the KL Penalty

We further investigate how the magnitude of the KL penalty influences Intuitor , as shown in Table 3 . On in-domain benchmarks (MATH500 and GSM8K), the choice of penalty has only a minor effect, but on out-of-domain tasks—LiveCodeBench (code generation) and CRUXEval-O (code reasoning)—model accuracy is highly sensitive to this hyper-parameter. Because Intuitor does not receive explicit feedback from generated responses during training, the KL penalty serves as a critical regularization mechanism. It prevents the policy from drifting too far from the initial model distribution, acting as a safeguard against degeneration. These findings highlight the importance of careful KL tuning in general-purpose reinforcement learning setups, especially when targeting robust generalization across domains.

### B.2 Scaling to Larger Models

We extend Intuitor to larger base models, including Qwen2.5-7B, Qwen2.5-14B, and Qwen3-14B. However, we find that the original training recipe triggers severe behavioral collapse at the very start of training. Even before any updates, the 7B model solves the given problem and then immediately proceeds to tackle an unrelated one; this tendency becomes more pronounced as training progresses.

To stabilize learning, we simplify the system prompt, reduce the learning rate to 1 × 10 − 6 1\times 10^{-6} , and increase the number of sampled responses per problem to sixteen. These settings represent our first, untuned trial, and a comprehensive hyperparameter sweep is beyond the scope of this paper. Because the system prompt is the only additional signal the model receives during Intuitor fine‑tuning, we expect its careful calibration to exert a particularly strong influence on training dynamics. With these adjustments, Intuitor trains smoothly on both larger models. The corresponding evaluation results and training dynamics are reported in Table 4 and Figure 9 .

### B.3 Generalization Across Model Families

To assess the generalizability of Intuitor across different model families, we apply it to Llama3.2-3B-Instruct ( Meta AI, 2024 ) and the fully open OLMo-2-1124-7B-SFT model ( OLMo et al., 2024 ) .

As shown in Table 5 and Figure 10 , Intuitor improves the performance of Llama3.2, with both accuracy and response length showing steady improvement throughout the training process, indicating meaningful optimization gains under Intuitor .

Similarly, results on OLMo-2 (Table 6 and Figure 11 ) confirm that Intuitor provides consistent training improvements. These experiments demonstrate its robustness and applicability beyond the Qwen model family. Furthermore, since OLMo-2 is a fully open-source model with available training data and code, it also addresses concerns about data contamination in the evaluation dataset.

### B.4 Comparison with Alternative Reward Signals

Contemporary research has found that applying a negative token-level entropy reward can improve a model’s reasoning performance without requiring external labels ( Agarwal et al., 2025 ; Prabhudesai et al., 2025 ) . However, since low entropy often correlates with repetitive loops ( Holtzman et al., 2019 ) , using negative entropy alone as an RL reward risks driving the model into a collapsed state. In other words, without sufficient supervised training to push the base model away from degenerate behavior, the model risks falling into a repetition trap from which it cannot recover. As we observe a nontrivial amount of repetitive responses in Qwen2.5‐1.5B, we test this hypothesis by applying GRPO with the negative‐entropy reward: u EM = − 1 | o | ⋅ | 𝒱 | ∑ i = 1 | o | ∑ j = 1 | 𝒱 | p π θ ( j | q , o < i ) ⋅ log ( p π θ ( j | q , o < i ) ) . \displaystyle u_{\text{EM}}=-\frac{1}{|o|\cdot|\mathcal{V}|}\sum_{i=1}^{|o|}\sum_{j=1}^{|\mathcal{V}|}p_{\pi_{\theta}}(j|q,o_{<i})\cdot\log\left(p_{\pi_{\theta}}(j|q,o_{<i})\right). Figure 12 (left) validates our prediction. Entropy minimization (EM) exacerbates repetition, and after a few updates, the model converges to producing the same character regardless of the prompt. By contrast, Intuitor enhances performance without triggering collapse (Figure 4 ). Even when the base model is sufficiently strong to avoid collapse during the early stages of entropy minimization training, it remains more prone to later degeneration because entropy provides a weaker confidence signal compared to self-certainty. As shown in Figure 13 , we train both EM and Intuitor under identical settings using Qwen2.5-3B for two epochs. The results show that while both methods initially reach similar peak performance, Intuitor stabilizes around this peak, whereas EM exhibits a steady decline, with a consistent bias toward longer responses. These findings highlight self-certainty as a more robust and effective signal for RLIF.

To further validate the efficacy of Intuitor , we also trained Qwen2.5‐3B using a random reward baseline ( Shao et al., 2025 ) , where each response was assigned a reward of 0 or 1 with equal probability. Figure 12 (right) shows that this random reward scheme severely degrades the model’s performance in a chat‐style RL setting, demonstrating that the performance gains observed with Intuitor are indeed non‐trivial.

Sharpening mechanisms ( Huang et al., 2025 ) have also been proposed to improve the policy, which use the logarithm of the probability assigned to the completion as the reward, log ⁡ ( ∏ i = 1 | o | p π θ ​ ( o i ∣ q , o < i ) ) \log\left(\prod_{i=1}^{|o|}p_{\pi_{\theta}}(o_{i}\mid q,o_{<i})\right) . However, this unnormalized probability is inherently length-biased toward shorter completions, since each conditional probability is at most one, and their product decreases with sequence length. This effect is especially pronounced for long chains of reasoning. We conducted an experiment using this reward under the same setup as before, training Qwen2.5-3B with the GRPO loss. As shown in Figure 14 , both the completion length and the reward decrease rapidly as training progresses, indicating degeneration of the policy. This empirical observation is consistent with our analysis. We further tested a length-normalized variant, 1 | o | ​ ∑ i = 1 | o | log ⁡ p π θ ​ ( o i ∣ q , o < i ) \frac{1}{|o|}\sum_{i=1}^{|o|}\log p_{\pi_{\theta}}(o_{i}\mid q,o_{<i}) , as the reward on Qwen2.5-1.5B. While normalization removes the short-length bias, it introduces the opposite tendency. The model can increase reward by producing longer completions. This reward is quickly exploited and training destabilizes. In contrast, Intuitor consistently improves accuracy on both models, demonstrating substantially stronger robustness.

### B.5 Ablation on Optimization Strategy: Policy Gradient vs. Direct Optimization

One possible approach is to optimize self-certainty directly by minimizing the negative self-certainty as a loss function. Although this strategy rapidly increases the target metric, it creates an incentive for reward hacking in which the model inflates its own certainty without genuine improvement in task performance. As illustrated in Figure 15 , direct optimization produces an initial rise in accuracy, suggesting that self-certainty is correlated with useful learning signals, but it ultimately results in model collapse. By comparison, the advantage weighted gradient policy optimization implemented in Intuitor incorporates self-certainty only as a relative weighting factor. This formulation mitigates reward hacking, stabilizes the optimization process, and consistently achieves superior performance relative to direct optimization.

### B.6 The Effect of Rollout Size

To examine how the number of rollouts per question affects the training of Intuitor , we conduct an additional experiment using a rollout size of 14 and compare the resulting validation accuracy with previous settings. As shown in Table 7 , increasing the rollout size leads to higher accuracy on both GSM8K and MATH500. A larger rollout size reduces the variance of the advantage estimates computed using self-certainty, which in turn improves training stability and generalization.

### B.7 Attempts at Combining Golden Answers and Self-Certainty

We also investigate whether combining self-certainty with golden-answer supervision can yield further performance improvements. Our first attempt uses a simple weighted sum of the advantages from Intuitor and GRPO, forming a combined advantage: A ′ = 1 2 ​ A Intuitor + 1 2 ​ A GRPO A^{\prime}=\tfrac{1}{2}A_{\textsc{Intuitor}}+\tfrac{1}{2}A_{\text{GRPO}} However, as shown in Table 8 , this straightforward combination not only fails to improve accuracy but in fact performs worse than GRPO alone.

Next, we experiment with a two-stage training scheme. We first train using Intuitor on MATH for one epoch, then switch to GRPO for an additional epoch. Interestingly, this alternating approach yields better performance than training with GRPO for two full epochs. One possible explanation is that Intuitor helps the model establish more confident and coherent reasoning trajectories, allowing subsequent GRPO training to better identify and reinforce correct reasoning traces. A more thorough investigation is needed to develop principled methods for combining these two types of signals, which remains a promising direction for future research.

### B.8 Sequential Training Across Domains

We further investigate how training on one domain with Intuitor affects subsequent training on another domain. Specifically, we compare Qwen2.5-3B trained directly on the MATH dataset using Intuitor with a model first trained on Codeforces and then fine-tuned on MATH using Intuitor . As shown in Table 9 , the model that was pretrained on Codeforces achieves higher accuracy on both GSM8K and MATH500 after being trained on MATH. This suggests that Intuitor training on one domain does not hinder later learning on another domain. In fact, pretraining on Codeforces appears to improve downstream mathematical reasoning performance.

### B.9 Standard Deviation of Response Correctness

To assess whether training with Intuitor reduces the variance of model responses, we track during training the average standard deviation of correctness within each rollout group and the standard deviation of correctness across all completions at each step. As shown in Figure 16 , the step-wise standard deviation remains largely stable for both methods, exhibiting minimal fluctuation. Meanwhile, the average within-group standard deviation decreases under both training procedures and converges to a similar level. Overall, we find no strong evidence that Intuitor reduces within-group correctness variance more aggressively than GRPO.

### B.10 Failure Case Analysis

Since Intuitor relies on the model’s own judgment to select better completions, it implicitly assumes that the model has already acquired sufficient domain knowledge during pretraining or supervised fine-tuning. If this prior knowledge is inadequate, training can collapse immediately or after a brief, unstable improvement. For example, applying our training setup to Llama3.2-3B-Base on MATH fails to raise its near-zero accuracy, likely because the base model is not aligned with the chat template. Similarly, training on noisy or confusing data, such as incomplete questions, can push the model toward repetitive or degenerate completions. In such cases the model may become more confident in generating its own problem-like text than in answering questions that are unsolvable for them. Fortunately, this behavior typically causes only minor drops on evaluation benchmarks, and models usually recover quickly once training resumes on cleaner, more suitable data.

In addition, self-certainty is a weaker learning signal because it provides no guarantee that the model’s preference reflects true correctness. As a result, Intuitor can be more sensitive to hyperparameter choices. For instance, using a learning rate of 3 × 10 − 6 3\times 10^{-6} on Qwen2.5-7B leads to performance degradation after a short initial climb. Reducing the learning rate to 1 × 10 − 6 1\times 10^{-6} stabilizes training and prevents this collapse.

### B.11 Training Hyperparameters

Training hyperparameters are listed in Table 10 .

## Appendix C Prompts and Model Completions

This section presents sample prompts and the responses generated by the models. Unless otherwise specified, the default base model used is Qwen2.5-3B, and the default training dataset is MATH.

### C.1 Training Prompts

### C.2 Example from LiveCodeBench Code Generation

Models trained with Intuitor tend to generate reasoning before producing code, whereas GRPO-tuned models typically respond with Python code directly under the same prompt.

### C.3 Example from LiveCodeBench Code Generation

Models trained with Intuitor are typically effective at interpreting instructions and producing correct Python code, while GRPO-tuned models may misinterpret the instructions or decline to provide a response.

### C.4 Example from CRUXEval-O

Models trained with Intuitor usually articulate their reasoning first, then deliver the formatted response as instructed, while GRPO-tuned models often produce the required JSON output immediately under the same prompt.

### C.5 Rollout Example of Intuitor During Training on Code Corpus

The following examples illustrate representative rollouts during the reinforcement‐learning training of Qwen2.5-3B on the CodeContest dataset. As shown in Figure 6 , the model initially produces concise reasoning and brief explanations to “convince” itself of its interim outputs. Over the course of training, it gradually shifts toward more detailed, step-by-step reasoning and richer explanations, which further reinforce its understanding and improve the final responses.

### C.6 Evolution of Qwen2.5-3B’s Responses on LiveCodeBench Code Generation trained with Intuitor on MATH

During training Qwen2.5-3B with Intuitor on the MATH dataset, its code-generation capability steadily improves. By the middle of training, the model learns to produce syntactically valid Python; as training progresses, it refines formatting and clarity. By the end, it arrives at correct solutions.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .