Learning to Reason without External Rewards — part 2/4

Cleaned-text code points [15779, 30738). Bibliography omitted here; source.md includes it.

where c i , t ​ ( θ ) = π θ ​ ( o i , t ∣ q , o i , < t ) π θ old ​ ( o i , t ∣ q , o i , < t ) c_{i,t}(\theta)=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})} is the importance weight, clip ε \operatorname{clip}_{\varepsilon} is the function that clips to [ 1 − ε , 1 + ε ] [1-\varepsilon,1+\varepsilon] . Hyperparameters ϵ \epsilon (for clipping) and β \beta (for KL penalty strength) control stability and exploration, and A ^ i , t \hat{A}_{i,t} is the advantage estimate.

Integration of Self-Certainty. The key innovation in Intuitor is replacing external rewards with self-certainty scores in GRPO’s advantage computation. Specifically, each output o i o_{i} is scored by: u i = Self-certainty ​ ( o i | q ) , A ^ i , t = u i − mean ​ ( { u 1 , u 2 , ⋯ , u G } ) std ​ ( { u 1 , u 2 , ⋯ , u G } ) . u_{i}=\text{Self-certainty}(o_{i}|q),\quad\hat{A}_{i,t}=\frac{u_{i}-\text{mean}(\{u_{1},u_{2},\cdots,u_{G}\})}{\text{std}(\{u_{1},u_{2},\cdots,u_{G}\})}. (2) This formulation enables the policy to favor outputs that the model itself considers more confident. The complete Intuitor training pipeline operates by sampling multiple candidate outputs for each query, computing self-certainty scores for each candidate, using these scores to estimate advantages within the group, and updating the policy to increase the likelihood of generating high-confidence outputs. This process requires no external supervision, making it a self-reinforcing learning loop.

## 4 Experimental Setup

Training Setup. Both GRPO and Intuitor are trained with the Open-R1 framework ( Face, 2025 ) on the training split of the MATH dataset ( Hendrycks et al., 2021 ) , which contains 7,500 problems. We use Qwen2.5-1.5B and Qwen2.5-3B ( Yang et al., 2024 ) as backbone models, with a chat-based prompting format throughout. Given the models’ initially weak instruction-following abilities, we do not require them to disentangle intermediate reasoning from final answers. Each update processes 128 problems, generating 7 candidate solutions per problem, with a default KL penalty of β = 0.005 \beta=0.005 . For a fair comparison, GRPO and Intuitor share identical hyperparameters (see Appendix) without additional tuning. We also evaluate a GRPO variant, denoted GRPO-PV in Table 1 , which uses plurality voting 1 1 1 Self-consistency uses a plurality rule, selecting the most frequent answer even without majority support, while majority voting requires > 50 % >50\% support and otherwise yields no winner ( De Condorcet and others, 2014 ) . as a proxy for ground truth. This follows the approach from TTRL ( Zuo et al., 2025 ) , which shows that self-consistency-based rewards can match the performance of golden answers when training on inference data.

Intuitor for Code Generation ( Intuitor -Code). To assess generalization beyond mathematical reasoning, we apply Intuitor to the Codeforces code generation dataset ( Li et al., 2022 ) . This variant, denoted Intuitor -Code in Table 1 , modifies the setup as follows: the number of sampled completions per problem is increased to 14; the learning rate is reduced from 3 × 10 − 5 3\times 10^{-5} to 1 × 10 − 5 1\times 10^{-5} ; and the KL penalty is increased to β = 0.01 \beta=0.01 . For simplicity, we limit the run to 50 steps, utilizing a total of 3,200 problems.

Evaluation. Evaluations generally use the same chat-style prompting format as in training, except for MMLU-Pro ( Wang et al., 2024 ) , where we follow the benchmark’s original prompt format. Greedy decoding is used for all completions. Experiments were conducted on NVIDIA A100 GPUs, each with 40GB of memory. We evaluate performance on the following benchmarks (1) Math reasoning : MATH500 and GSM8K, using the lighteval library ( Habib et al., 2023 ) . (2) Code reasoning : CRUXEval-O ( Gu et al., 2024 ) , using the ZeroEval framework ( Lin, 2024 ) , and LiveCodeBench v6 (LCB) ( Jain et al., 2024 ) . (3) Instruction following : AlpacaEval 2.0 with length-controlled win rates ( Dubois et al., 2024 ) , judged by GPT-4.1 ( OpenAI, 2025 ) .

## 5 Results and Analysis

We evaluate the effectiveness of Intuitor by addressing the following research questions: • (RQ1) How does the overall performance of Intuitor compare to supervised RLVR methods?

• (RQ2) How does intrinsic feedback influence the model’s qualitative behavior?

• (RQ3) How robust is online self-certainty when used as an intrinsic reward signal during training?

Table 1 presents main evaluation results, and Figure 4 illustrates response length evolution during training. On in-domain MATH and GSM8K datasets, Intuitor and GRPO-PV (both golden-answer-free) achieve performance comparable to GRPO (using golden answers). This aligns with TTRL ( Zuo et al., 2025 ) , where plurality voting approximated golden answers without significant performance loss. While Intuitor performs slightly worse than GRPO overall, on MATH it produces longer responses and demonstrates markedly improved code generation, suggesting enhanced reasoning capabilities.

### 5.1 Learning to Follow Instructions

Intuitor significantly enhances instruction-following. Initially, the pretrained Qwen2.5-1.5B struggles with chat-style prompts, scoring < < 10% on all chat-template tasks (Table 1 ) and generating repetitive, nonsensical output, which inflates average response lengths (Figure 4 ). Fine-tuning with Intuitor sharply reduces such gibberish, decreases completion lengths, and enables non-trivial performance across all evaluated benchmarks. Furthermore, on the MATH dataset, Intuitor substantially improves the Length Control Win Rate on AlpacaEval for both Qwen2.5-1.5B and Qwen2.5-3B, surpassing GRPO under identical settings. This demonstrates robust gains in instruction adherence.

### 5.2 Fostering Structured Reasoning

Rapid Initial Learning. Self-certainty, a continuous and inherently process-aware reward derived from the model’s internal assessment across all tokens, contrasts with binary rewards. This internal signal may encourage LLMs to follow more effective learning trajectories. Given comparable final performance between GRPO and Intuitor , we assess early-stage learnability by comparing in-domain accuracy at training step 10. As shown in Table 2 , Intuitor consistently outperforms GRPO on both GSM8K and MATH benchmarks for Qwen2.5-1.5B and Qwen2.5-3B, highlighting its advantage in rapid initial learning.

Cross-Task Generalization. Figure 4 illustrates performance trajectories on MATH500 (in-domain) and LiveCodeBench (transfer task) for models trained on the MATH dataset. For both Intuitor and GRPO, accuracy improvements on LiveCodeBench emerge later in training, following initial gains on MATH500. Notably, LiveCodeBench performance continues to improve even after MATH500 accuracy plateaus. This pattern suggests that initial in-domain learning (on MATH) facilitates subsequent generalization to code generation tasks (LiveCodeBench).

Emergence of Long-Form Reasoning. While large models like Deepseek-R1 achieve long-form reasoning through extensive RL, Intuitor enables smaller models to develop structured reasoning with limited data. On CRUXEval-O (Figure 5 ), models trained with Intuitor often exhibit free-form reasoning before summarizing it within the instructed JSON block, despite prompts requiring reasoning directly in JSON. A similar pattern of pre-code natural language reasoning is observed on LiveCodeBench. This emergent pre-reasoning may contribute to Intuitor ’s strong performance on these benchmarks.

### 5.3 Understanding Emergent Long-Form Reasoning

When LLMs encounter unfamiliar questions, they sample from a distribution of possible answers ( Kang et al., 2024 ) . Self-certainty reflects the model’s internal assessment of its output coherence. By reinforcing high-confidence responses, Intuitor encourages more elaborate reasoning, potentially improving the model’s comprehension of its own outputs. While not explicitly targeting benchmark accuracy, this enhancement in output quality and structure leads to more reliable answers and better generalization.

We analyze models trained with Intuitor on code corpora by examining outputs for ten randomly selected LiveCodeBench questions across different training steps. Figure 6 shows the evolution of output types alongside model accuracy. The results reveal a clear progression: models first learn to generate valid Python code (evidenced by improved accuracy and fewer invalid responses), then develop pre-code reasoning to facilitate self-understanding. Further inspection of generations confirms that models progressively elaborate their reasoning throughout training, supporting our hypothesis that Intuitor encourages traces that the model itself can better understand.

To quantify this effect, we classify outputs from successive checkpoints into three categories: invalid code (”No Answer”), valid code without reasoning (”No Reasoning”), and valid code with explicit reasoning (”Reasoning”). Figure 6 (a) illustrates how these proportions evolve during training alongside LiveCodeBench accuracy. The model first reduces invalid outputs and improves code correctness before incorporating pre-code reasoning, reflecting an emergent emphasis on self-explanatory traces. Figure 6 (b) demonstrates how training with Intuitor leads to structured reasoning before code generation. Additional evidence appears in Figure 8 , where Intuitor -trained models assign significantly higher confidence to their generated responses compared to baseline models, as discussed further in Section 5.4 .

### 5.4 Online Self-Certainty Prevents Reward Exploitation

Over-optimization against static reward models is a known failure mode in reinforcement learning ( Gao et al., 2023 ) . To assess the robustness of self-certainty as a reward, we compare offline self-certainty (rewards from a fixed base model) with online self-certainty (rewards from the evolving policy model), using a reduced batch size of 224 responses per gradient update.

Figure 7 demonstrates that the offline annotator is susceptible to exploitation. Around the 100th update step, the policy model learns to inflate its self-certainty reward by appending an auxiliary, already-solved problem to its answer for the given question. This exploitation manifests as a sharp increase in response length (dashed line) and a concurrent collapse in validation accuracy (solid line). In contrast, the online annotator, whose reward signal co-evolves with the policy, prevents such reward hacking and maintains stable training.

To further evaluate the quality of self-certainty as a reward signal, we analyze the distribution of self-certainty scores from policies trained with Intuitor and GRPO on MATH500 responses (Figure 8 ). We employ Mann–Whitney U tests to determine if correct responses achieve significantly higher self-certainty scores than incorrect ones. Both GRPO and Intuitor models exhibit significantly higher average self-certainty scores, indicating that GRPO also enhances the model’s self-assessment capabilities. Notably, policies trained with online self-certainty (i.e., Intuitor ) show no signs of reward hacking. The Intuitor policy yields the lowest p p -values and largest effect sizes ( r r ) in the Mann-Whitney U tests (Figure 8 , inset). This indicates it is most effective at discriminating its own correct and incorrect answers using self-certainty, even while assigning higher absolute confidence scores overall. These findings underscore the potential of Intuitor for robust training on larger datasets.

### 5.5 Ablation Studies

To comprehensively validate Intuitor ’s design and robustness, we conducted extensive ablation studies, with full details provided in Appendix B due to page limitations. Key findings are: (1) KL term: Varying the KL penalty (Sec. B.1 ) shows a stability–performance trade-off; moderate values yield the best accuracy. (2) Scaling: Intuitor scales to larger backbones (Qwen2.5-7B/14B, Qwen3-14B; Sec. B.2 ), delivering consistent gains in reasoning and generalization. (3) Architecture: On Llama-3.2 and OLMo-2 (Sec. B.3 ), Intuitor remains effective, indicating robustness across model families and sizes. (4) Reward design: Compared to entropy minimization ( Agarwal et al., 2025 ) and random rewards ( Shao et al., 2025 ) , Intuitor yields stable improvements, while the alternatives trigger catastrophic collapse (Sec. B.4 ). (5) Optimization strategy: Directly optimizing self-certainty as a loss function leads to reward hacking and performance collapse; our advantage-weighted policy-gradient formulation avoids this and trains reliably (Sec. B.5 ).

## 6 Discussion and Future Research

Scalability and Generalization. Our experiments, constrained by computational resources, utilize relatively compact models trained on relatively small, unsupervised corpora. We aim to demonstrate the potential of a model’s self-certainty as a reward signal for policy optimization. The results show that this signal consistently promotes more coherent, well-justified, and interpretable explanations, indicating a path towards more autonomous learning. Future work could explore these benefits in larger foundation models (with hundreds of billions of parameters) and on more diverse, real-world datasets. Given that purely offline training with Intuitor led to performance degradation over time, scaling up will likely require periodic online updates to self-certainty estimates or hybrid offline-online schedules to maintain calibration.

Theoretical Analysis of RLIF. While we have empirically demonstrated the superior performance of using self-certainty as a reward for RLIF, the underlying theoretical mechanisms warrant further investigation. Huang et al. (2025) analyze LLM self-improvement as a “sharpening” mechanism, proposing a statistical framework to evaluate algorithm efficiency via sample complexity. Similarly, Yue et al. (2025) question whether RLVR functions primarily by sharpening the base model’s existing distribution. However, determining the theoretically optimal reward signal for RLIF and establishing the fundamental reasoning boundaries of LLMs remain open problems. These challenges highlight the need for future theoretical research to complement empirical findings.

Combining Reward Signals. To enable a direct comparison between self-certainty and golden-answer rewards, this paper focuses exclusively on a single reward signal. However, these signals are not mutually exclusive. Future work could explore combining them, for instance, by summation or by alternating based on the availability of golden answers. Furthermore, other reward signals, such as formatting rewards ( Guo et al., 2025 ) , could be additively combined to enhance performance. Integrating RLIF with methods like RLHF and RLVR may further advance LLM capabilities across various dimensions.

## 7 Conclusion