Learning to Reason without External Rewards — part 1/4
Cleaned-text code points [0, 15779). Bibliography omitted here; source.md includes it.
# Learning to Reason without External Rewards
###### Abstract
Training large language models (LLMs) for complex reasoning via Reinforcement Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on costly, domain-specific supervision. We explore Reinforcement Learning from Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic signals without external rewards or labeled data. We propose Intuitor , an RLIF method that uses a model’s own confidence—termed self-certainty —as its sole reward signal. Intuitor replaces external rewards in Group Relative Policy Optimization (GRPO) with self-certainty scores, enabling fully unsupervised learning. Experiments demonstrate that Intuitor matches GRPO’s performance on mathematical benchmarks while achieving better generalization to out-of-domain tasks like code generation, without requiring gold solutions or test cases. Our findings show that intrinsic model signals can drive effective learning across domains, offering a scalable alternative to RLVR for autonomous AI systems where verifiable rewards are unavailable. Code is available at https://github.com/sunblaze-ucb/Intuitor .
## 1 Introduction
Reinforcement learning has become essential for enhancing large language model capabilities. Early work focused on Reinforcement Learning from Human Feedback (RLHF), which aligns model outputs with human values through reward models trained on preference data ( Ouyang et al., 2022 ) . Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) replace learned reward models with automatically verifiable signals, such as exact answer matching in mathematical problem-solving, demonstrating improved reasoning capabilities in models like DeepSeek-R1 ( Guo et al., 2025 ; Lambert et al., 2024 ) .
Despite these successes, both RLHF and RLVR face fundamental limitations that constrain their broader applicability. RLHF requires extensive human annotation, making it expensive and potentially biased ( Gao et al., 2023 ) . RLVR, while avoiding learned reward models, demands domain-specific verifiers and gold-standard solutions. In mathematics, this requires expert annotation of solutions; in code generation, it necessitates comprehensive test suites and execution environments ( Liu et al., 2023 ; Liu and Zhang, 2025 ; Team et al., 2025 ; Xiaomi, 2025 ) . These requirements limit RLVR to carefully curated domains and complicate deployment in open-ended scenarios. Moreover, outcome-oriented verifiable rewards limit transferability to other domains. These challenges motivate exploration of more general and scalable reward paradigms, leading to a critical research question: Can LLMs enhance their reasoning abilities by relying solely on intrinsic, self-generated signals, without recourse to external verifiers or domain-specific ground truth?
In this paper, we introduce and explore such a paradigm: Reinforcement Learning from Internal Feedback (RLIF) , where models optimize intrinsic feedback to improve performance without external rewards or supervision. The motivation for RLIF extends to future scenarios where models develop superhuman capabilities that become difficult for humans to evaluate directly ( Burns et al., 2023 ) , requiring self-improvement through intrinsic mechanisms ( Oudeyer and Kaplan, 2007 ) .
Under the RLIF paradigm, we propose Intuitor , a novel reinforcement learning approach leveraging a model’s own confidence as an intrinsic reward. This builds on observations that LLMs exhibit lower confidence on difficult problems ( Farquhar et al., 2024 ; Kuhn et al., 2023 ; Kang et al., 2024 ; Kang et al., 2025 ) ; optimizing for confidence should improve reasoning capabilities. Specifically, we use self-certainty ( Kang et al., 2025 ) , the average KL divergence between the model’s output distribution and a uniform distribution, as our confidence measure. This metric has proven useful for distinguishing high-quality responses from flawed ones ( Kang et al., 2025 ; Ma et al., 2025 ) . Building on this insight, Intuitor guides learning through self-generated signals, eliminating the need for external supervision or handcrafted rewards. The implementation of Intuitor is simple, efficient, and effective: we replace the verifiable reward signal in existing RLVR frameworks, specifically Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) , with self-certainty scores, using the same policy gradient algorithm.
Our experiments demonstrate promising results. On the MATH dataset ( Hendrycks et al., 2021 ) with Qwen2.5-3B base ( Yang et al., 2024 ) , Intuitor matches the performance of GRPO without relying on any gold answers. As Intuitor rewards the generation trajectory rather than only the end result, it generalizes more effectively: training a Qwen2.5-3B base model on MATH yields a 65 % 65\% relative improvement on LiveCodeBench Code generation task ( Jain et al., 2024 ) versus no improvement for GRPO, and a 76 % 76\% gain on CRUXEval-O ( Gu et al., 2024 ) compared with 44 % 44\% for GRPO. Additionally, when we fine-tune the Qwen2.5-1.5B base model with Intuitor on the MATH corpus, a model that originally produces repetitive content and scores 0 % 0\% on LiveCodeBench learns to emit coherent reasoning chains and well-structured code, reaching 9.9 % 9.9\% accuracy after the tuning. Beyond the Qwen family, experiments with Llama ( Meta AI, 2024 ) and OLMo ( OLMo et al., 2024 ) models also show impressive gains, underscoring the strong generalization capabilities of Intuitor . As Intuitor requires only a clear prompt and no verifiable reward, it is broadly applicable across tasks, providing fresh evidence that pretrained LLMs possess richer latent behavioral priors than previously recognized.
Our contributions can be summarized as follows: • We introduce and explore Reinforcement Learning from Internal Feedback (RLIF), a novel reinforcement learning paradigm enabling LLMs to improve reasoning skills by leveraging intrinsic, self-generated signals, without reliance on external supervision or labeled data.
• We introduce Intuitor , an RLIF-based method that utilizes a model’s own internal confidence measure—termed self-certainty —as the sole intrinsic reward.
• We demonstrate that Intuitor matches supervised RL performance on in-domain tasks and achieves competitive, sometimes better out-of-domain generalization. We uncover emergent structured reasoning and enhanced instruction-following capabilities induced by intrinsic rewards.
## 2 Related Work
#### Reinforcement Learning from Human Feedback (RLHF).
RL has become instrumental in refining LLMs. Early pivotal work centered on Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ) , which aligns LLMs with human values by training a reward model on human preference data. While effective, RLHF is often resource-intensive due to the need for extensive human annotation ( Touvron et al., 2023 ) . Subsequent innovations like Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) aimed to simplify this by directly training models on preferences. The reliance on human-generated or model-approximated human preferences poses scalability challenges and introduces potential biases from the reward model itself ( Gao et al., 2023 ) .
#### Reinforcement Learning with Verifiable Rewards (RLVR).
RLVR emerged as a powerful alternative, particularly for tasks with clear correctness criteria like mathematical reasoning and code generation ( Hu et al., 2025 ; Team et al., 2025 ; Xiaomi, 2025 ) . RLVR utilizes rule-based verification functions, such as exact answer matching ( Guo et al., 2025 ; Team et al., 2025 ; Xiaomi, 2025 ; Jaech et al., 2024 ) , to provide reward signals, thereby avoiding the complexities and potential pitfalls of learned reward models. This approach has sparked significant advances, with models like DeepSeek-R1 ( Guo et al., 2025 ) achieving state-of-the-art reasoning capabilities. The development of robust policy optimization algorithms like GRPO ( Shao et al., 2024 ) and its variants ( Luo et al., 2025 ; Liu et al., 2025 ) has further solidified RLVR’s success. Nevertheless, RLVR’s applicability is largely confined to domains where verifiable gold solutions or exhaustive test cases can be constructed, and its predominant focus on outcome-based rewards can limit generalization to dissimilar tasks or those requiring nuanced, process-oriented feedback.
#### Intrinsic Signals and Self-Play in LLM Optimization.
Self-play and intrinsic rewards enable autonomous model improvement. Methods like SPIN ( Chen et al., 2024 ) and Self-Rewarding LMs ( Yuan et al., 2024 ) use the model itself for feedback. Earlier work like STaR ( Zelikman et al., 2022 ) relies on outcome evaluation, while others explore procedural generalization ( Poesia et al., 2024 ; Cheng et al., 2024 ) . Concurrent works such as Genius, TTRL, SRT, and Absolute Zero ( Xu et al., 2025 ; Zuo et al., 2025 ; Shafayat et al., 2025 ; Zhao et al., 2025 ) leverage unlabeled queries for RL but are often restricted to specific task distributions. Song et al. (2025) examine LLM self-improvement through the generation–verification gap, while Huang et al. (2025) study it through the lens of sharpening dynamics. Intuitor aligns with this direction, offering a lightweight, general-purpose approach using self-certainty as a confidence-based intrinsic reward, enabling single-agent RL across diverse tasks without explicit feedback or gold labels.
## 3 Method
### 3.1 Reinforcement Learning from Internal Feedback (RLIF)
To overcome the limitations of RLHF’s costly human annotation and RLVR’s domain-specific supervision, we propose Reinforcement Learning from Internal Feedback (RLIF). Instead of depending on external evaluation, RLIF uses the model’s own assessment of its outputs as feedback. This offers several advantages: it reduces reliance on supervision infrastructure, provides task-agnostic reward signals, and supports learning in domains where external verification is unavailable. The optimization objective for policy π θ \pi_{\theta} is: max π θ 𝔼 o ∼ π θ ( q ) [ u ( q , o ) − β KL [ π θ ( o | q ) ∥ π ref ( o | q ) ] ] \max_{\pi_{\theta}}\mathbb{E}_{o\sim\pi_{\theta}(q)}\left[u(q,o)-\beta\mathrm{KL}[\pi_{\theta}(o|q)\|\pi_{\text{ref}}(o|q)]\right] (1) where q q is an input query, o o is the generated output, π ref \pi_{\text{ref}} is an initial reference policy, and β \beta controls the KL divergence to prevent excessive deviation from π ref \pi_{\text{ref}} . Here, u ( q , o ) u(q,o) is an intrinsic signal derived from the model’s internal state or computation, rather than external verification. The key challenge lies in identifying intrinsic signals that correlate with output quality and can effectively guide learning.
Concurrent research explores related concepts within the RLIF paradigm. For example, Entropy Minimized Policy Optimization (EMPO) ( Zhang et al., 2025 ) minimizes LLM predictive entropy on unlabeled questions in a latent semantic space. SEED-GRPO ( Chen et al., 2025 ) uses the semantic entropy of generated sequences, combined with ground truth rewards, to modulate policy updates. Reinforcement Learning with a Negative Entropy Reward (EM-RL) ( Agarwal et al., 2025 ) employs a reward signal based solely on the negative sum of token-level entropy, akin to REINFORCE but without labels. These methods highlight the growing interest and potential of leveraging intrinsic signals for LLM training under the RLIF framework.
### 3.2 Intuitor : Policy Optimization with Self-Certainty
We propose Intuitor , a novel RLIF method that utilizes a model’s own confidence as the sole intrinsic reward signal u ( q , o ) u(q,o) . Our choice of model confidence as the intrinsic reward is motivated by observations that LLMs often exhibit lower confidence when encountering unfamiliar tasks or lacking sufficient knowledge ( Kang et al., 2024 ) . Conversely, higher confidence frequently correlates with correctness. By rewarding increased self-confidence, Intuitor encourages to iteratively “practice” and refine its reasoning pathways until it becomes more confident in its outputs.
We adopt the self-certainty metric from Kang et al. (2025) , defined as the average KL divergence between a uniform distribution U U over the vocabulary 𝒱 \mathcal{V} and the model’s next-token distribution: Self-certainty ( o | q ) ≔ 1 | o | ∑ i = 1 | o | KL ( U ∥ p π θ ( ⋅ | q , o < i ) ) = − 1 | o | ⋅ | 𝒱 | ∑ i = 1 | o | ∑ j = 1 | 𝒱 | log ( | 𝒱 | ⋅ p π θ ( j | q , o < i ) ) \displaystyle\textbf{Self-certainty}(o|q)\coloneqq\frac{1}{|o|}\sum_{i=1}^{|o|}\mathrm{KL}(U\parallel p_{\pi_{\theta}}(\cdot|q,o_{<i}))=-\frac{1}{|o|\cdot|\mathcal{V}|}\sum_{i=1}^{|o|}\sum_{j=1}^{|\mathcal{V}|}\log\left(|\mathcal{V}|\cdot p_{\pi_{\theta}}(j|q,o_{<i})\right) where o < i o_{<i} are the previously generated tokens and p ( j | q , o < i ) p(j|q,o_{<i}) is the model’s predicted probability for token j j at step i i . Higher self-certainty values indicate greater confidence.
Self-certainty is related to a KL divergence where the model’s prediction is the second argument, KL ( U ∥ p π θ ) \mathrm{KL}(U\parallel p_{\pi_{\theta}}) . This contrasts with entropy (or reverse KL divergence from uniform). Critically, self-certainty is reported to be less prone to biases towards longer generations, a common issue with perplexity or entropy-based measures ( Fang et al., 2024 ; Kang et al., 2025 ) , making it a potentially more reliable indicator of intrinsic confidence. Kang et al. (2025) demonstrate that self-certainty is effective for selecting high-quality answers from multiple candidates, and uniquely among different confidence measures, its utility improves with more candidates. Optimizing for self-certainty thus encourages the model to generate responses that it deems more convincing. The RL process can achieve this by, for instance, guiding the model to produce more detailed reasoning steps, thereby increasing the model’s conviction in its final answer. This mechanism is more nuanced than simply increasing the probability of the most likely output; it involves modifying the generation process itself to build confidence.
To optimize the objective in Equation 1 , various policy gradient algorithms can be employed. Informed by the recent success in models such as DeepSeek-R1 ( Guo et al., 2025 ) and its widespread adoption of GRPO in open-source projects, we utilize GRPO to optimize for self-certainty. The overall pipeline for this GRPO-based instantiation of Intuitor is illustrated in Figure 2 .
The core idea behind the optimization is to sample multiple candidate outputs for a given query and use their relative rewards to estimate advantages for policy updates. For each query q ∼ P ( Q ) q\sim P(Q) , GRPO samples a group of G G outputs o 1 , … , o G {o_{1},\ldots,o_{G}} using a behavior policy π θ old \pi_{\theta_{\text{old}}} (e.g., a previous iteration or the SFT model). The target policy π θ \pi_{\theta} is then optimized by maximizing: 𝒥 GRPO ( θ ) = 𝔼 q ∼ P ( Q ) , { o i } i = 1 G ∼ π θ old ( ⋅ | q ) [ 1 G ∑ i = 1 G 1 | o i | ∑ t = 1 | o i | ( min [ c i , t ( θ ) A ^ i , t , clip ε ( c i , t ( θ ) ) A ^ i , t ] − β 𝔻 KL ( π θ ∥ π ref ) ) ] \displaystyle\mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}_{\begin{subarray}{c}q\sim P(Q),\\ \{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|q)\end{subarray}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\left(\min\left[c_{i,t}(\theta)\hat{A}_{i,t},\;\operatorname{clip}_{\varepsilon}\left(c_{i,t}(\theta)\right)\hat{A}_{i,t}\right]-\beta\mathbb{D}_{\mathrm{KL}}\bigl(\pi_{\theta}\|\pi_{\mathrm{ref}}\bigr)\right)\right]