Reinforcement Pre-Training — part 1/2
Cleaned-text code points [0, 15755). Bibliography omitted here; source.md includes it.
# Reinforcement Pre-Training
###### Abstract
In this work, we introduce Reinforcement Pre-Training (RPT) as a new scaling paradigm for large language models and reinforcement learning (RL). Specifically, we reframe next-token prediction as a reasoning task trained using RL, where it receives verifiable rewards for correctly predicting the next token for a given context. RPT offers a scalable method to leverage vast amounts of text data for general-purpose RL , rather than relying on domain-specific annotated answers. By incentivizing the capability of next-token reasoning, RPT significantly improves the language modeling accuracy of predicting the next tokens. Moreover, RPT provides a strong pre-trained foundation for further reinforcement fine-tuning. The scaling curves show that increased training compute consistently improves the next-token prediction accuracy. The results position RPT as an effective and promising scaling paradigm to advance language model pre-training.
## 1 Introduction
Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, largely driven by the scalability of the next-token prediction objective on vast text corpora. This self-supervised paradigm has proven to be an effective general-purpose pre-training approach. Concurrently, reinforcement learning (RL) has emerged as a powerful technique for fine-tuning LLMs, aligning them with human preferences or enhancing specific skills such as complex reasoning OWJ+ (22) ; JKL+ (24) ; GYZ+ (25) .
However, current applications of RL in LLM training face scalability and generality challenges. Reinforcement learning from human feedback OWJ+ (22) , while effective for alignment, relies on costly human preference data, and its learned reward models can be susceptible to reward hacking, limiting scalability. Alternatively, reinforcement learning with verifiable rewards (RLVR) LMP+ (25) utilizes objective, rule-based rewards, often from question-answer pairs. While this mitigates reward hacking, RLVR is typically constrained by the scarcity of annotated data with verifiable answers, restricting its application to domain-specific fine-tuning rather than general-purpose pre-training.
In this work, we introduce reinforcement pre-training ( RPT ), a novel paradigm that bridges the gap between scalable self-supervised pre-training and the power of reinforcement learning. RPT reframes the fundamental next-token prediction task as a next-token reasoning process. For any given context in a pre-training corpus, the model is incentivized to reason about the subsequent token before predicting it. It receives a verifiable, intrinsic reward based on the correctness of its prediction against the ground-truth next token from the corpus itself. This approach transforms the vast, unannotated text data typically used for next-token prediction into a massive dataset for general-purpose RL, without requiring external annotations or domain-specific reward functions.
This approach offers several crucial advantages. First, RPT is inherently scalable and general-purpose: it leverages the same vast, unannotated text data used for standard next-token prediction, transforming it into a massive dataset for general-purpose RL without requiring external annotations. Second, the use of direct, rule-based reward signals (i.e., the correctness of the predicted next token) inherently minimizes the risk of reward hacking often associated with complex, learned reward models. Third, by explicitly encouraging next-token reasoning patterns, RPT promotes deeper understanding and generalization instead of merely memorizing next tokens. The model learns to explore and validate hypotheses about why a certain token should follow, fostering more robust representations. Finally, the internal reasoning process during pre-training effectively allows the model to allocate more “thought” or computational effort to each prediction step, akin to a form of inference-time scaling applied at training time for each token, which directly contributes to improved next-token prediction accuracy.
Our experiments demonstrate that RPT significantly improves the accuracy of predicting next tokens. RPT also provides a more robust pre-trained foundation for subsequent reinforcement fine-tuning, leading to better final task performance. The scaling curves reveal that increased training compute under the RPT framework consistently improves next-token prediction accuracy, indicating its potential as a sustainable scaling strategy. These results position reinforcement pre-training as an effective and promising new paradigm to advance the pre-training of large language models.
Our contributions are summarized as follows: • We introduce reinforcement pre-training ( RPT ), a new scaling paradigm that reframes next-token prediction as a reasoning task trained with reinforcement learning, utilizing intrinsic verifiable rewards derived directly from the pre-training corpus.
• RPT offers a scalable and general-purpose approach to RL pre-training, minimizing reward hacking through rule-based rewards and promoting generalization by encouraging next-token reasoning patterns over rote memorization.
• RPT significantly improves next-token prediction accuracy and exhibits favorable scaling properties, where performance consistently improves with increased training compute.
• RPT yields a stronger pre-trained foundation for subsequent reinforcement fine-tuning and enhances zero-shot performance on various downstream tasks.
## 2 Preliminary
#### Next-Token Prediction (NTP)
Next-token prediction is the fundamental training objective for modern large language models AAA+ (23) . Given an input sequence x 0 ⋯ x T x_{0}\cdots x_{T} from the training corpus, the model is trained to maximize the following objective: 𝒥 NTP ( θ ) = ∑ t = 1 T log P ( x t ∣ x 0 , x 1 , … , x t − 1 ; θ ) , \mathcal{J}_{\text{NTP}}(\theta)=\sum_{t=1}^{T}\log P(x_{t}\mid x_{0},x_{1},\ldots,x_{t-1};\theta), (1) where θ \theta represents the parameters of the language model.
#### Reinforcement Learning with Verifiable Rewards (RLVR)
RLVR employs a reinforcement learning objective to enhance specific skills with verifiable answers LMP+ (25) . RLVR requires a labeled dataset of question-answer pairs 𝒟 = { ( q , a ) } \mathcal{D}=\{(q,a)\} . For a specific pair ( q , a ) ∈ 𝒟 (q,a)\in\mathcal{D} , the LLM π θ \pi_{\theta} generates a response o ∼ π θ ( ⋅ ∣ q ) o\sim\pi_{\theta}(\cdot\mid q) . A deterministic verifier 𝒱 \mathcal{V} calculates a verifiable reward r = 𝒱 ( o , a ) r=\mathcal{V}(o,a) , and the model is trained to maximize the expected reward:
𝒥 RLVR ( θ ) = 𝔼 ( q , a ) ∼ 𝒟 , o ∼ π θ ( ⋅ ∣ q ) [ r ( o , a ) ] . \mathcal{J}_{\text{RLVR}}(\theta)=\mathbb{E}_{(q,a)\sim\mathcal{D},\ o\sim\pi_{\theta}(\cdot\mid q)}\left[r(o,a)\right]. (2)
## 3 Reinforcement Pre-Training
### 3.1 Pre-Training Task: Next-Token Reasoning
We propose the next-token reasoning task for language modeling. Given an input sequence x 0 ⋯ x T x_{0}\cdots x_{T} from the training corpus, for each position t ∈ { 1 , … , T } t\in\{1,\ldots,T\} , the prefix x < t x_{<t} is treated as the context, and ground-truth next token is x t x_{t} . In the next-token reasoning task, the model π θ \pi_{\theta} is required to generate a chain-of-thought reasoning sequence, denoted by c t c_{t} , before generating a prediction y t y_{t} for the next token. The overall model response is o t = ( c t , y t ) o_{t}=(c_{t},y_{t}) , o t ∼ π θ ( ⋅ ∣ x < t ) o_{t}\sim\pi_{\theta}(\cdot\mid x_{<t}) .
As illustrated in Figure 2 , the long chain-of-thought process for next-token reasoning can involve various reasoning patterns such as brainstorming, self-critique and self-correction. The next-token reasoning task reconstructs the pre-training corpus into a vast set of reasoning problems, shifting pre-training beyond learning superficial token-level correlations to understanding the hidden knowledge behind them and making RL scaling possible.
### 3.2 Pre-Training with Reinforcement Learning
Reinforcement pre-training ( RPT ) trains LLMs to perform next-token reasoning via on-policy reinforcement learning, as illustrated in Figure 3 . For the context x < t x_{<t} , we prompt the language model π θ \pi_{\theta} to generate G G responses (thinking trajectories), { o t i } i = 1 G \{o^{i}_{t}\}_{i=1}^{G} . Each response o t i = ( c t i , y t i ) o^{i}_{t}=(c^{i}_{t},y^{i}_{t}) consists of a chain-of-thought reasoning sequence c t i c^{i}_{t} and a final prediction sequence y t i y^{i}_{t} .
To verify the correctness of y t i y^{i}_{t} , we introduce a prefix matching reward , which supports verifying predictions that span multiple tokens or involve out-of-vocabulary tokens. 1 1 1 Additional reward design choices for next-token reasoning are discussed in Appendix A . Let x ¯ ≥ t \overline{x}_{\geq t} and y ¯ t i \overline{y}^{i}_{t} denote the byte sequences of the ground-truth completion sequence x ≥ t x_{\geq t} and the prediction y t i y^{i}_{t} , respectively. Denote the byte length of y ¯ t i \overline{y}^{i}_{t} by l l . We define the cumulative byte lengths of the tokens in the ground-truth completion sequence as valid boundaries, and denote this set by ℒ g t \mathcal{L}_{gt} . Formally, the reward r t i r^{i}_{t} for the i i -th output for x < t x_{<t} is defined as: r i t = { 1 if y ¯ i t = x ¯ ≥ t [ 1 : l ] and l ∈ ℒ g t 0 otherwise , r^{i}_{t}=\begin{cases}1&\text{if }\overline{y}^{i}_{t}=\overline{x}_{\geq t}[1:l]\text{ and }l\in\mathcal{L}_{gt}\\ 0&\text{otherwise}\end{cases}, (3) where the reward is 1 if the byte sequence of the prediction is an exact prefix of the ground-truth completion sequence and its length l l matches any valid token boundary.
Let 𝒟 \mathcal{D} be the set of all { x < t } t = 1 T \{x_{<t}\}_{t=1}^{T} , the model is trained to maximize the expected reward:
𝒥 RPT ( θ ) = 𝔼 ( x < t , x ≥ t ) ∼ 𝒟 , { o i t } i = 1 G ∼ π θ ( ⋅ ∣ x < t ) [ r t i ] . \mathcal{J}_{\text{{RPT}{}}}(\theta)=\mathbb{E}_{(x_{<t},x_{\geq t})\sim\mathcal{D},\ \{o^{i}_{t}\}_{i=1}^{G}\sim\pi_{\theta}(\cdot\mid x_{<t})}\left[r^{i}_{t}\right]. (4)
### 3.3 Pre-Training Setup
We use the OmniMATH dataset GSY+ (24) for reinforcement pre-training. OmniMATH contains 4,428 competition-level mathematical problems and solutions from official websites such as AoPS Wiki 2 2 2 https://artofproblemsolving.com/wiki/index.php and AoPS forum 3 3 3 https://artofproblemsolving.com/community/c13_contests . Since many tokens are easily predictable even without reasoning, we perform token-level data filtering before reinforcement pre-training. Particularly, we use Deepseek-R1-Distill-Qwen-1.5B as a small proxy model. For each token, we calculate the proxy model entropy on the top-16 next tokens. By applying an entropy threshold, we filter out low-entropy positions, prioritizing training on challenging tokens that require greater computational effort to predict.
In all experiments, we use Deepseek-R1-Distill-Qwen-14B GYZ+ (25) as the base model. R1-Distill-Qwen-14B serves as a good starting point for reinforcement learning due to its basic reasoning capabilities. We implement our training framework with the verl library SZY+ (24) and use vllm for inference. We employ the GRPO algorithm GYZ+ (25) , with specific hyperparameters detailed in Appendix B . During training, we adopt an 8k training length, a learning rate of 1 × 10 − 6 1\times 10^{-6} , zero KL penalty, and a batch size of 256 questions. For each question, G=8 responses are sampled, and for the rollout process, we use a temperature of 0.8. From each response, we directly extract the full sequence inside the last \boxed{} following the special token ‘</think>’ as the model prediction for the next token. Starting from 500 steps, we utilize dynamic sampling to boost training efficiency YZZ+ (25) . The total training steps for our main experiment is 1,000. The prompt template and its variants are discussed in Appendix D .
### 3.4 Evaluation of Pretrained Models
Once the model is pretrained, we can directly conduct next-token prediction and reinforcement fine-tuning on downstream tasks. We use the settings to show that reinforcement pre-training improves the language modeling capabilities and reasoning abilities of large language models.
#### Language Modeling
Given the next-token reasoning objective, our models can be naturally used for language modeling. We report the next-token prediction accuracy to evaluate the language modeling performance and scaling properties of RPT .
#### Reinforcement Fine-Tuning on Downstream Tasks
We conduct continual RL fine-tuning with RPT models in a pretrain-then-finetune manner. Since RPT aligns the pre-training process with RL, the objective gap between pre-training and RL during post-training is minimized. We evaluate whether the reinforcement pre-training process further enhances post-training on end tasks.
## 4 Experiments
### 4.1 Language Modeling
We evaluate the language modeling performance on a held-out validation set of 200 samples from OmniMATH. Following the entropy-based data filtering strategy described in our setup ( Section 3.3 ), we categorize token positions in the validation set according to their difficulty. Specifically, we calculate the entropy at each token position using R1-Distill-Qwen-14B. We then designate positions as belonging to easy, medium, or hard splits if their entropy exceeds thresholds of 0.5, 1.0, and 1.5, respectively. For comparison, we report the performance of R1-Distill-Qwen-14B evaluated in two different ways: (1) Standard next-token prediction, selecting the token with the highest probability; and (2) Next-token reasoning, generating a chain-of-thought before the final prediction. We also include the results of Qwen2.5-14B, as it is the base model for R1-Distill-Qwen-14B.
As shown in Figure 4 , RPT -14B achieves consistently higher next-token prediction accuracy across all difficulty levels compared to R1-Distill-Qwen-14B. Remarkably, it matches the performance of a significantly larger model, i.e., R1-Distill-Qwen-32B ( Figure 4 ). These results suggest that reinforcement pre-training is effective at capturing the complex reasoning signals underlying token generation, and holds strong potential for improving the language modeling capability of LLMs.
### 4.2 Scaling Properties of Reinforcement Pre-Training
In this section, we investigate the scaling properties of reinforcement pre-training. The loss achieved by next-token pre-training on natural language corpus empirically follows a power-law decay with respect to model size, number of training tokens, and training compute HBM+22b () ; KMH+ (20) . Below, we analyze the scaling behavior of RPT specifically with respect to training compute C C . We model this relationship using the following power-law form: P ( C ) = A C α + P ∗ P(C)=\frac{A}{C^{\alpha}}+P^{*} (5) where P ( C ) P(C) denotes the next-token prediction accuracy on the validation set. P ∗ P^{*} , α \alpha , and A A are parameters to be estimated.
We evaluate the next-token prediction accuracy of RPT at various training steps (100, 200, 400, 800, 1000, and 1200) and convert them into the corresponding training compute. To assess the impact of data difficulty, we consider validation splits filtered by entropy thresholds 0.5 (easy), 1.0 (medium), and 1.5 (hard). A higher threshold corresponds to more challenging inputs for the LLM. For each difficulty level, we fit the results according to Equation 5 . We measure the goodness of fit using the coefficient of determination R 2 R^{2} , which quantifies how well the scaling curve fits the observed data.