Can Large Reasoning Models Self-Train? — part 1/5
Cleaned-text code points [0, 14938). Bibliography omitted here; source.md includes it.
# Can Large Reasoning Models Self-Train?
###### Abstract
Recent successes of reinforcement learning (RL) in training large reasoning models motivate the question of whether self-training — the process where a model learns from its own judgments — can be sustained within RL. In this work, we study this question using majority voting as a simple self-feedback mechanism. On a comprehensive set of experiments on both synthetic and real reasoning tasks, we find that this basic approach improves not only the model’s reasoning performance, but also its capability of generating better quality feedback for the next RL iteration, driving further model improvement. Yet our analysis also reveals a critical limitation of such a self-training paradigm — prolonged RL with self-reward leads to reward hacking where models learn to maximize training (pseudo-)reward, resulting in sudden and complete performance collapse. Together, these results highlight feedback design as the central challenge and call for future research on mechanisms to enable prolonged self-improvement.
## 1 Introduction
Pre-training on human-curated corpora has endowed language models with broad general-purpose capabilities ( Brown et al., 2020 ; Rae et al., 2022 ) , but the supply of such data is becoming a bottleneck as compute scales rapidly ( Hoffmann et al., 2022 ; Sevilla et al., 2022 ) . Reinforcement learning (RL) ( Sutton et al., 1998 ) with verifiable rewards (RLVR) addresses this limitation by using automatic correctness checks, and has already shown success in reasoning and agentic tasks ( DeepSeek-AI et al., 2025 ; OpenAI et al., 2024 ) . Yet, achieving super-intelligence inevitably requires moving beyond domains where humans can provide ground-truth solutions. In these domains, an appealing approach is to have models self-improve — judging the correctness of their own outputs and using this signal to refine future generations ( Zelikman et al., 2022 ; Song et al., 2025b ; Huang et al., 2025 ) . If sustained iteratively, and if teacher models can also self-improve , this process could enable continual progress without human supervision — which is crucial step for AI based automatic scientific discovery.
Prior work on self-improvement has mainly relied on supervised fine-tuning (SFT) ( Zelikman et al., 2022 ; Huang et al., 2023 ) or direct preference optimization (DPO) ( Rafailov et al., 2024 ; Prasad et al., 2024 ) , where the self-labeling rule is updated only a handful of times (e.g., 1–10 rounds) and is generally kept fixed within a training round . These studies show that self-improvement can be effective, but leave open whether it can be sustained over longer horizons. Moreover, they are fundamentally bounded by the verification capabilities of the teacher model used to obtain training supervision, which is kept fixed for extended periods of training. In contrast, RL continuously updates the model on freshly generated (“on-policy”) data ( Xu et al., 2024 ; Tajwar et al., 2024 ; Lanchantin et al., 2025 ) , a property that has been critical to its success in training reasoning models with verifiable rewards. This success raises a natural question: can self-improvement leverage the same continuous-update paradigm? To study this question, we investigate the setting in which the feedback signal is updated at every gradient step —fundamentally altering the dynamics of self-improvement compared to earlier work.
In this RL-based setup, the choice of feedback mechanism is critical. If feedback is always correct (e.g., perfectly verifying mathematical solutions), the procedure reduces to standard RL with ground-truth supervision. However, in practice, self-feedback is imperfect, and its design determines whether self-training is effective or not. As a first step, we study the simplest possible signal: majority vote . Wang et al. (2023a) has empirically demonstrated that majority vote tends to have higher accuracy compared to individual generations. Here, we cast the majority vote mechanism as a reward function — granting positive reward to model outputs that match the most common answer. Previous works ( Huang et al., 2023 ; Prasad et al., 2024 ) have employed majority voting mainly as a mechanism to extract better quality generations from a fixed teacher policy to then distill it into the student model. Zuo et al. (2025) concurrently examined a similar RL procedure with majority voting, but in a different setting, as their focus is on training and testing on the same set of prompts. In contrast, our aim is to use this simple pseudo-label generation mechanism to investigate the validity of RL powered self-training frameworks.
Our key contributions are threefold: (1) Our comprehensive set of experiments demonstrate that this simple mechanism yields measurable gains over the base model on key reasoning metrics such as maj@k and avg@k success rates (Section 4.1 ). Remarkably, we observe clear improvement in the label generating policy after each gradient step, and this translates to gains over employing labels from a fixed teacher. Moreover, self-training achieves comparable performance to RLVR on 4 different base models.
(2) In synthetic tasks where one can control the difficulty of the training dataset, we observe that a simple curriculum-based self-training approach can enable the model to keep climbing on progressively harder tasks without ground-truth labels (Section 4.2 ). Overall, this signals a promising path towards self-improvement of LLMs.
(3) However, prolonged training with this framework consistently teaches models to ignore the prompt entirely and output the same template final answer, which maximizes training reward but leads to complete collapse on test datasets. To our knowledge, we are the first to demonstrate this failure mode of self-improving LLMs by employing self-consistency based training signal. In this work, we analyze the dynamics of model collapse in depth and trace them to the self-reinforcing nature of imperfect feedback.
Overall, these findings identify feedback design as the key challenge that future research should address to sustain self-improvement.
Our code is publicly available here: https://github.com/tajwarfahim/srt . For other project-related assets such as constructed datasets, please see our project website at https://self-rewarding-llm-training.github.io/ .
## 2 Preliminaries
Let π θ \pi_{\theta} denote a language model parameterized by θ \theta . Given a prompt x x , the model produces a response y = ( y 1 , y 2 , … ) y=(y^{1},y^{2},\dots) auto-regressively. Formally, each token in the response sequence is generated according to the conditional probability: y k + 1 ∼ π θ ( ⋅ ∣ x , y ≤ k ) , y^{k+1}\sim\pi_{\theta}(\cdot\mid x,y^{\leq k}), (1) where we use y ≤ k y^{\leq k} to refer to the first k k tokens generated by the model.
For reasoning-based tasks considered here, the model typically produces responses following a step-by-step “chain-of-thought” reasoning approach ( Wei et al., 2022 ) . A verification function r ( y ) r(y) , whose dependence on x x is omitted for brevity, extracts the model’s proposed solution from the generated response and evaluates its correctness against the prompt-specific ground-truth answer: r ( y ) = { 1 if y is correct, 0 if y is incorrect. r(y)=\begin{cases}1&\text{if $y$ is correct,}\\ 0&\text{if $y$ is incorrect.}\end{cases} (2)
Typically, one optimizes the expected pass rate , defined as the average accuracy across a distribution of prompts 𝒳 \mathcal{X} : J ( θ ) = 𝔼 x ∼ 𝒳 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ r ( y ) ] . J(\theta)=\mathbb{E}_{x\sim\mathcal{X}}\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}[r(y)]. (3)
Taking the gradient of the objective function ( 3 ) with respect to θ \theta and employing a baseline for variance reduction ( Sutton et al., 1998 ) leads to the well-known policy gradient formulation: ∇ θ J ( θ ) = 𝔼 x ∼ 𝒳 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ 𝒜 ( y ) ∇ θ log π θ ( y ∣ x ) ] , \nabla_{\theta}J(\theta)=\mathbb{E}_{x\sim\mathcal{X}}\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\left[\mathcal{A}(y)\nabla_{\theta}\log\pi_{\theta}(y\mid x)\right], (4) where the advantage function 𝒜 ( y ) \mathcal{A}(y) is given by: 𝒜 ( y ) = r ( y ) − 𝔼 y ′ ∼ π θ ( ⋅ ∣ x ) [ r ( y ′ ) ] . \mathcal{A}(y)=r(y)-\mathbb{E}_{y^{\prime}\sim\pi_{\theta}(\cdot\mid x)}[r(y^{\prime})]. (5) Here 𝔼 y ′ ∼ π θ ( ⋅ ∣ x ) [ r ( y ′ ) ] \mathbb{E}_{y^{\prime}\sim\pi_{\theta}(\cdot\mid x)}[r(y^{\prime})] is the average pass rate for prompt x x . In practice, the policy gradient in equation 4 is estimated through Monte Carlo samples, yielding the classical REINFORCE algorithm ( Williams, 1992 ) . Recent works have modified this base policy gradient formulation to improve its stability, efficiency, and practicality, resulting in advanced methods such as REINFORCE++ ( Hu et al., 2025 ) , GRPO ( Shao et al., 2024 ) , PPO ( Schulman et al., 2017 ) , RLOO ( Ahmadian et al., 2024 ) , Dr. GRPO ( Liu et al., 2025 ) , GSPO ( Zheng et al., 2025 ) , etc. Implicit to the training method used in our work is the notion of a generation-verification gap ( Song et al., 2025b ) , where generating correct solutions is hard, but verifying them is easy. We present its definition in Appendix A .
## 3 Self-Rewarded Training
Our objective in this work is to investigate whether reliable training supervision for language models can be generated without external labels. The typical practice for online RL involves generating multiple responses to a prompt, then assigning high or low rewards to each generation according to a ground-truth verifier. In the absence of such a verifier, one might develop a mechanism to derive proxy labels. This mechanism then provides a simple recipe for framing self-improvement as an RL problem. At a high level, each iteration proceeds as follows: (1) Sample a mini-batch of prompts, (2) Determine pseudo-labels, y pseudo y_{\text{pseudo}} , using the mechanism for each prompt, (3) Generate n n responses per prompt and use agreement with the derived psuedo-labels as an intrinsic binary reward:
r ( y ) = 𝟏 [ answer ( y ) = y pseudo ] , \displaystyle r(y)=\mathbf{1}[\text{answer}(y)=y_{\text{pseudo}}], (6)
and then (4) Perform a single RL update step on this mini-batch using the reward function r ( ⋅ ) r(\cdot) .
#### Self-supervision via majority voting.
Among several possible choices for determining reasonably accurate pseudo-labels, we explore the simplest one in this work, majority voting . Majority voting has been empirically demonstrated to have higher accuracy compared to individual model generations ( Wang et al., 2023a ) and is thus a suitable choice to exploit an LLM’s inherent generation-verification gap (see Appendix Figure 9 , where majority voting consistently outperforms the model’s average capabilities across 3 test datasets and therefore can act as a verifier). In our setting, models typically produce a step-by-step chain of thought followed by a final answer (in the case of regular RL training, this final answer is extracted and matched with the ground truth to produce rewards), so one can group all responses by their final answer to determine the majority vote. Concretely, assume we want to generate pseudo-labels using policy π label \pi_{\text{label}} (which can be any reasonable policy). This procedure then involves: (1) sampling multiple answers per prompt using policy π label \pi_{\text{label}} , (2) grouping answers according to their parsed final solutions, (3) estimating the ground truth answer with the most common solution.
#### Self-Rewarded Training (SRT).
The general procedure is described in Algorithm 1 , which henceforth shall be called Self-Rewarded Training ( SRT ) in this paper. Since the method prescribes a specific form of the reward function using model self-consistency, it is compatible with all the common RL training algorithms such as PPO, RLOO, REINFORCE, GRPO, etc. We study the quality of generated labels during training by controlling π label \pi_{\text{label}} : setting π label \pi_{\text{label}} to be the base model recovers our familar setting of learning the majority voting decisions of a fixed model (while still using the current policy’s rollouts for RL training), and setting π label \pi_{\text{label}} to be the current policy π θ \pi_{\theta} after each gradient step allows us to study whether the quality of the learning signal can be concurrently improved during RL training. In the case where we use the current policy π θ \pi_{\theta} to generate our pseudo-labels, we can reuse them for performing the RL gradient step as well. As the number of generations per prompt typically falls in the range 16-64 ( Yu et al., 2025 ) , this variant of SRT incurs no additional compute cost compared to the versions of these algorithms employing ground truth labels. In our work, whether the final answer is correctly formatted and parseable is used to filter responses, but given more compute, one can theoretically employ more sophisticated systems like LLM-as-a-judge ( Zheng et al., 2023 ; Gu et al., 2025 ) or generative verifiers ( Zhang et al., 2025a ) to further improve the quality of the training signal. We leave these for future work.
As long as majority voting leads to a positive generation-verification gap at each RL iteration, we expect iterative self-rewarding to provide a useful supervisory signal. We describe our empirical observations in the following section.
## 4 Experiments and Analysis
In this section, we present the results of our empirical study. Our primary aim is to answer the two following research questions: (1) Can SRT improve an LLMs’ reasoning abilities beyond the base model’s capabilities , in terms of both the quality of training reward signal and performance on downstream tasks? Specifically, since simply performing majority voting on the base model can improve performance compared to individual generations, can we improve the quality of the majority votes (which acts as the verifier for SRT ) in addition to pass@1 performance compared to the base model via SRT ? (2) If SRT can go beyond the base model, can this improvement be sustained indefinitely? We systematically design experiments to answer these questions below.
### 4.1 Can SRT go beyond the base model’s capabilities?
There are two potential axes of improvements over the base model for SRT that can be studied: (1) the improvement in accuracy over held-out prompts, and (2) the improvement in the quality of generated labels themselves during the training procedure. Unlike previous works ( Huang et al., 2023 ; Prasad et al., 2024 ) that distill the majority voting decision of a fixed policy (typically the base model) into the current model, we want to study whether the quality of the majority votes of the evolving policy improves as a result of self-improvement.
#### Experiments on synthetic reasoning tasks.