Reinforcement Pre-Training — part 2/2

Cleaned-text code points [15755, 28926). Bibliography omitted here; source.md includes it.

As shown in Figure 5 , the next-token prediction accuracy of RPT improves reliably as the training compute is scaled up. High R 2 R^{2} values across all difficulty levels demonstrate that the fitted curves accurately capture performance trends.

### 4.3 Reinforcement Fine-Tuning with RPT

To investigate whether RPT models can be more effectively fine-tuned with RLVR, we randomly sample questions with verifiable answers from Skywork-OR1 HLL+ (25) for further training. We use 256 examples for training and 200 for testing. Following the data filtering pipeline from Skywork-OR1 HLL+ (25) , we use R1-Distill-Qwen-32B to identify challenging instances for training. We set both the training batch size and the PPO mini-batch size to 64, and train the model for 15 epochs. During evaluation, the maximum number of tokens for validation is set to 32,000, with a temperature of 0.6.

As shown in Table 2 , the reinforcement pre-trained model achieves a higher upper bound when further trained with RLVR. The reasoning ability of the model significantly declines when continually trained on the same data using a next-token prediction objective. Subsequent RLVR yields only slow performance improvements. These results indicate that with limited data, reinforcement pre-training can quickly transfer the strengthened reasoning patterns learned from next-token reasoning to end tasks.

### 4.4 Zero-Shot Performance on End Tasks

We evaluate the zero-shot performance of RPT -14B on end tasks. For comparison, we assess the next-token prediction performance of R1-Distill-Qwen-14B and R1-Distill-Qwen-32B, as well as the the reasoning performance of RPT -14B with R1-Distill-Qwen-14B.

Our evaluation involves two widely acknowledged benchmarks: MMLU-Pro HBB+ (20) , a comprehensive multi-task understanding benchmark evaluating LLM capabilities across various domains; SuperGPQA DYM+ (25) , a large-scale benchmark of graduate-level reasoning questions spanning 285 disciplines. Under the reasoning setting, we set the maximum number of tokens to 12,288 and the temperature to 0.8. Following previous works MLJ+ (25) ; ZLS+25b () , we use a multiple-choice question format for evaluation and report the accuracy.

As shown in Table 3 , RPT -14B consistently outperforms R1-Distill-Qwen-14B (whether using standard next-token prediction or evaluated as a reasoning model) across all benchmarks. Notably, it also surpasses the significantly larger R1-Distill-Qwen-32B (under next-token prediction), with gains of 7 points on SuperGPQA and approximately 22 points on MMLU-Pro. Detailed per-subject results for each benchmark are provided in Appendix C .

### 4.5 Next-Token Reasoning Pattern Analysis

We analyze the differences in reasoning patterns between next-token reasoning and explicit problem solving. Following previous studies WYZ+ (25) ; GCD+ (25) , we statistically measure the proportion of model responses containing reasoning-indicative keywords (e.g., “ break down ”, “ alternatively ’’). 4 4 4 The keywords are listed in Appendix E .

Our analysis compares the thought processes of two models on the OmniMATH datasets, i.e., R1-Distill-Qwen-14B for problem solving, and RPT -14B for next-token reasoning, based on 200 sampled responses from each model. We categorize reasoning patterns into six types: transition (switching strategies), reflection (self-checking), breakdown (decomposing the problem), hypothesis (proposing and verifying assumptions), divergent thinking (exploring possibilities), and deduction (logical inference).

As illustrated in Figure 6 , RPT -14B’s next-token reasoning process is markedly different from the problem-solving of R1-Distill-Qwen-14B, exhibiting a 161.8% greater use of the hypothesis pattern and a 26.2% greater use of the deduction pattern. In contrast, the problem-solving process relies more heavily on the breakdown pattern, highlighting that next-token reasoning elicits an inferential process qualitatively different from structured problem-solving.

We also provide an example of reasoning patterns in Table 4 . The example reveals that the model engages in a deliberative process, not a simple pattern match. It analyzes the broader semantic context (“ calculating vector magnitude ”), identifies pivotal phrases (“ go over some… ”), and then brainstorms and weighs multiple plausible continuations. This involves hypothesis generation (“ the next part is likely going to be… ”), consideration of alternatives (“ Alternatively, it could be… ”), and reflection on structural cues (“ markdown with headers ”) and even fine-grained token-level details (“ could have a space ”). This multi-faceted reasoning, encompassing both high-level semantic understanding and low-level textual features, demonstrates the model’s effort to deduce the next token through a reasoned exploration, aligning with the goals of RPT to cultivate deeper understanding beyond superficial correlations. More examples are provided in Appendix F .

## 5 Related Work

#### Scaling Paradigms of Large Language Models

The advancements of large language models have been driven by two primary scaling dimensions: training-time compute KMH+ (20) ; HBM+22a () and test-time compute ZLS+25a () . Training-time scaling substantially increases model parameters and training data, using next-token prediction as the pre-training task. Meanwhile, test-time scaling JKL+ (24) trades extended inference compute to improve the reasoning capabilities of large language models. Going beyond existing scaling paradigms, RPT uniquely integrates the above principles, framing each next-token prediction as a reasoning task.

#### Reinforcement Learning for Large Language Models

Reinforcement learning (RL) has played a crucial role in the post-training stage of large language models. Reinforcement learning from human feedback OWJ+ (22) fine-tunes pre-trained language models on human preference data to improve alignment. Beyond alignment, large-scale RL has also been adopted to enhance the reasoning capabilities of language models JKL+ (24) ; GYZ+ (25) . ZHS+ (24) is the most relevant work, which encourages language models to generate helpful rationales for next-token prediction. The helpfulness-based reward tends to be hacked by repeating the target token in the generated rationale, where the shortcut potentially harms the model. In contrast, we use next-token prediction correctness as a rule-based reward signal to minimize reward hacking.

## 6 Conclusion and Future Work

We introduce reinforcement pre-training ( RPT ), a novel paradigm for pre-training large language models. By framing next-token prediction as a verifiable reasoning task and applying reinforcement learning with correctness-based rewards, RPT allows LLMs to leverage extended computation during pre-training to build stronger foundational reasoning capabilities. Our experiments demonstrate that RPT improves next-token prediction, enhances performance on mathematical and general reasoning benchmarks in zero-shot settings, and provides a better starting point for further RL fine-tuning. RPT offers a promising new direction for developing more capable and generally intelligent LLMs by fundamentally rethinking the pre-training objective itself.

While promising, this initial exploration of RPT has certain limitations. Our experiments are primarily conducted using a 14B parameter model. Although the RPT methodology is designed to be general, the current pre-training corpus predominantly consists of mathematical documents; future work will explore its efficacy on broader, general-domain text. Furthermore, RPT training is initialized from a reasoning model; investigating RPT training from a standard base language model would provide further insights into its foundational impact.

The work can be advanced from the following perspectives. We would like to scale up the training corpus, including data size, and domain coverage. Large-scale general Internet data can be utilized during reinforcement pre-training. We will also scale up training compute to push the frontier. Moreover, we can establish scaling laws for reinforcement pre-training to guide the scaling of large language models. Additionally, we are interested in integrating hybrid thinking JWH+ (25) with RPT to enable fine-grained adaptive thinking by adaptively triggering next-token reasoning.

## Acknowledgement

We extend our gratitude to Yuting Jiang for maintaining the GPU cluster. We also thank Zewen Chi and Yang Wang for technical support during the development of the RL infrastructure on the MI300 GPUs. We implement training based on verl SZY+ (24) .

## Appendix A Design Choices of Reward

We have also investigated several alternative reward functions to assess their impact on reinforcement pre-training, in addition to the reward mechanism described in Section 3 , i.e., prefix matching reward.

One variation is first-token matching. In this setup, the reward reflects only whether the first token of the model prediction y t i y_{t}^{i} matches the ground-truth next token x t x_{t} , ignoring all tokens after the first in the prediction. Another alternative explores a ‘dense reward’ scheme. Here, correctly predicted next tokens (i.e., y t i ​ [ 0 ] = x t y_{t}^{i}[0]=x_{t} ) receive a full reward (e.g., 1). For incorrect predictions ( y t i ​ [ 0 ] ≠ x t y_{t}^{i}[0]\neq x_{t} ), the reward is a positive, smaller value, specifically the language model probability of generating that particular incorrect token, P ⁡ ( y t i ​ [ 0 ] ∣ x < t ; θ ) P(y_{t}^{i}[0]\mid x_{<t};\theta) . This provides a denser feedback signal than binary rewards. A third design is a conditional application of this dense reward structure. The dense reward (1 for correct, P ⁡ ( y t i ∣ x < t ; θ ) P(y_{t}^{i}\mid x_{<t};\theta) for incorrect) is used as described above, but only for training instances (groups of rollouts for a given prefix x < t x_{<t} ) where at least one of the G G sampled rollouts correctly predicted the next token. If all G G rollouts in a group are incorrect, a different reward scheme (e.g., zero reward for all, or a uniform small penalty) will be applied.

Our experiments indicate that the alternative reward designs generally achieved performance comparable to the prefix matching reward. This suggests that the reinforcement pre-training framework is relatively robust to these particular modifications in the reward signal, and its core benefits may not be overly sensitive to these specific choices, at least within the scope of variations tested.

## Appendix B Hyperparameters Used for Reinforcement Pre-Training

Table 5 presents the detailed hyperparameters for reinforcement pre-training in Section 4 . We follow the setting of exact on-policy reinforcement learning [ 9 ] and set the entropy loss coefficient to 0 0 .

## Appendix C Detailed Results on End Tasks

Table 6 and Table 7 present a detailed per-category performance across the general end task benchmarks. Notably, the performance of R1-Distill-Qwen-14B is evaluated in two different manner: standard next-token prediction and reasoning-based answer prediction (indicated as ‘+ think’). The RPT -14B model demonstrates superior performance compared to R1-Distill-Qwen-14B and R1-Distill-Qwen-32B.

## Appendix D Impact of Prompt Templates

We explore the impact of various prompt templates on the initial next-token reasoning performance. Table 10 shows seven template variants. The templates use different phrasing of instructions and wrap the context in various formats.

As presented in Table 8 , clear prompts significantly improve the correctness of the initial performance. Notice that the reinforcement pre-training experiments in Section 4 used the ‘v0’ prompt template. We leave prompt engineering based on other template variants for future work, which tends to improve the final performance.

## Appendix E Keywords for Reasoning Pattern Analysis

Table 9 presents the pattern groups and keywords applied in reasoning pattern analysis.

## Appendix F Case Studies

To provide qualitative insights into the behavior of models trained with RPT , we present some cases of next-token reasoning in Table 11 .

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .