Learning to Reason without External Rewards — part 4/4

Cleaned-text code points [46369, 50809). Bibliography omitted here; source.md includes it.

Since Intuitor relies on the model’s own judgment to select better completions, it implicitly assumes that the model has already acquired sufficient domain knowledge during pretraining or supervised fine-tuning. If this prior knowledge is inadequate, training can collapse immediately or after a brief, unstable improvement. For example, applying our training setup to Llama3.2-3B-Base on MATH fails to raise its near-zero accuracy, likely because the base model is not aligned with the chat template. Similarly, training on noisy or confusing data, such as incomplete questions, can push the model toward repetitive or degenerate completions. In such cases the model may become more confident in generating its own problem-like text than in answering questions that are unsolvable for them. Fortunately, this behavior typically causes only minor drops on evaluation benchmarks, and models usually recover quickly once training resumes on cleaner, more suitable data.

In addition, self-certainty is a weaker learning signal because it provides no guarantee that the model’s preference reflects true correctness. As a result, Intuitor can be more sensitive to hyperparameter choices. For instance, using a learning rate of 3 × 10 − 6 3\times 10^{-6} on Qwen2.5-7B leads to performance degradation after a short initial climb. Reducing the learning rate to 1 × 10 − 6 1\times 10^{-6} stabilizes training and prevents this collapse.

### B.11 Training Hyperparameters

Training hyperparameters are listed in Table 10 .

## Appendix C Prompts and Model Completions

This section presents sample prompts and the responses generated by the models. Unless otherwise specified, the default base model used is Qwen2.5-3B, and the default training dataset is MATH.

### C.1 Training Prompts

### C.2 Example from LiveCodeBench Code Generation

Models trained with Intuitor tend to generate reasoning before producing code, whereas GRPO-tuned models typically respond with Python code directly under the same prompt.

### C.3 Example from LiveCodeBench Code Generation

Models trained with Intuitor are typically effective at interpreting instructions and producing correct Python code, while GRPO-tuned models may misinterpret the instructions or decline to provide a response.

### C.4 Example from CRUXEval-O

Models trained with Intuitor usually articulate their reasoning first, then deliver the formatted response as instructed, while GRPO-tuned models often produce the required JSON output immediately under the same prompt.

### C.5 Rollout Example of Intuitor During Training on Code Corpus

The following examples illustrate representative rollouts during the reinforcement‐learning training of Qwen2.5-3B on the CodeContest dataset. As shown in Figure 6 , the model initially produces concise reasoning and brief explanations to “convince” itself of its interim outputs. Over the course of training, it gradually shifts toward more detailed, step-by-step reasoning and richer explanations, which further reinforce its understanding and improve the final responses.

### C.6 Evolution of Qwen2.5-3B’s Responses on LiveCodeBench Code Generation trained with Intuitor on MATH

During training Qwen2.5-3B with Intuitor on the MATH dataset, its code-generation capability steadily improves. By the middle of training, the model learns to produce syntactically valid Python; as training progresses, it refines formatting and clarity. By the end, it arrives at correct solutions.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .