Self-Improving Pretraining: using post-trained models to pretrain better models — part 3/5

Cleaned-text code points [31470, 47375). Bibliography omitted here; source.md includes it.

We show further detailed results in Table 24 which highlight that, for example, optimizing for safety does not optimize for factuality, or vice-versa. This indicates that if you want to optimize for both, both must be factored into the rewards provided during training. We also report the performance of the larger Llama-3.1 8B Base model, to show that our results are not a distillation effect owing to our reward model’s size. Optimizing for safety and quality with Self-Improving Pretraining outperforms the 8B model with a 1.4B model.

##### Pretraining from-scratch results

The previous results are from continued pretraining from the initial Llama baseline model. Potentially, our Self-Improving Pretraining could provide much larger improvements if used earlier in pretraining, for example by making the model learn safety measures earlier on in training.

We compare 4 training setups in the safety pretraining setting:

• Pretrain Baseline (model trained on RedPajama suffixes);

• Pretrain on Rewrites;

• Self-Improving Pretraining: RF-NLL (suffix vs. rewrite);

• Self-Improving Pretraining: RF-NLL (rollout vs. rewrite).

In these experiments, we only use 1 rollout for training.

Table 2 summarizes quality and safety evaluation results. NLL pretraining on rewritten suffixes outperforms baseline training on safety evaluations, but does not improve on overall quality. Using the fine-tuned Llama3.1-8B-Instruct suffix judge promotes generations that are better in both quality and in safety, resulting in improved performance for our models. Self-Improving Pretraining using RF-NLL (rollout vs. rewrite) has a generation quality win rate of 32.4, compared to the next-token prediction baseline win rate of only 1.3 – a huge improvement. Simultaneously, safety evaluations improve from 85.2 to 97.5.

#### 1.2.5 Analysis & ablations

##### Training objective

Table 6 provides ablation results on variants of the Self-Improving Pretraining training objective in the safety optimization case. First, we find that continued pretraining using standard next token prediction on RedPajama lowers the performance compared to the initial baseline on safety evaluations slightly (from 76.9 to 75.5), while standard evaluations are similar or slightly improved (47.6 vs. 47.9). As RedPajama contains unsafe contexts this is not unexpected. Continued pretraining on the cleaner SlimPajama keeps the safety evaluations more or less unchanged (76.9 vs. 77.0), although standard evaluations drop.

Next, training with SFT on rewrites or a single rollout without a judge gives little improvement in quality for the former (52.7 of safe and 50.6 on unsafe prefixes), and large deterioration for the latter (dropping to 2.0 and 0.2 on safe and unsafe prefixes), which is expected (i.e., model collapse). Upon inspection of the model generations, we found that the model trained on a single rollout collapsed to generating meaningless - but safe - sequences of words or symbols. In contrast online DPO with the rewrite as chosen and current rollout as rejected gives slightly improved standard and safety evaluations (48.8 and 77.7 respectively).

Overall, however, with our full Self-Improving Pretraining method using a post-trained suffix judge, we find much larger gains – particularly in the online DPO case, and for larger numbers of rollouts. We find applying RF-NLL improves safety evaluations over the baseline (85.0 vs. 76.9) but is only on par with the improvement found using SFT on rewrites, which does not use a judge, while both do not give significant gains in generation quality. For online DPO however, we see major boosts in generation quality. Online DPO using rewrites and a single rollout improves generation quality from 50.0 to 60.0 on standard prefixes, and from 50.0 to 87.2 on unsafe prefixes. Increasing to 16 rollouts gives even larger gains on standard prefixes (from 50.0 to 73.6), and on overall safety evaluations (from 76.9 to 91.1).

##### Suffix & rewrite vs. rollouts

In both the continual and from-scratch pretraining settings, we find that early in training the model relies on the original and rewritten suffixes more often for supervision. As the model improves the judge picks rollouts more and more frequently, see Figure 8 . Later in training RL rewards high-quality rollouts, resulting in a higher rollout chosen rate.

##### Number of rollouts

We report ablation results on the number of rollouts used in online DPO for quality, factuality, and safety training in Figure 9 . We generally find improved performance across all benchmarks with an increasing number of rollouts, where we experimented with between 1 and 16 rollouts. We did not experiment past 16 rollouts due to the increased compute required, but we expect further gains.

Furthermore, similar trends can be seen in generation quality and standard evaluations, as shown in Appendix Table 17 , where more rollouts lead to better final performance across all benchmarks tested. Detailed standard task results are also given in Appendix Table 18 and Table 19 .

##### Judge choice

As mentioned in subsubsection 1.2.1 , we experiment with two types of judges: one fine-tuned specifically for a target task such as quality, and another used directly via prompting without training. In Table 7 we compare these two judges when they are used for quality training. We find that the prompted GPT-OSS-120B model generally performs better, but the finetuned Llama judge is not far behind, demonstrating that we can purpose-train a smaller model for this goal. A detailed breakdown of results across standard tasks can be found in Appendix Table 20 .

##### Pivots in pairwise comparison judgments

We also experiment with speeding up pairwise quality judgments by instead using a pivot. That is, one generation is selected and then all generations in the training batch are compared only against this pivot generation to produce rewards. Results are given in Appendix Table 21 , Table 22 and Table 23 for various settings. Overall we find deterioration in performance from using pivots, leaving how to make judgments faster while maintaining quality an open question.

### 1.3 Related Work

Pretraining of neural language models stretches back to the work of Bengio et al. (2003) , and language modeling itself stretches back to at least Shannon (1948) . Subsequent work then built both masked language modeling ( Collobert et al., 2011 ; Peters et al., 2018 ; Devlin et al., 2019 ) and next token prediction systems ( Dai and Le, 2015 ; Raffel et al., 2020 ; Radford et al., 2018 ) . The latter has now become the dominant paradigm due to the ability to extend to generating full sequences autoregressively. Despite rapid progress, particularly by scaling ( Brown et al., 2020 ; Achiam et al., 2023 ) , there remain unanswered questions in key areas of generalization, for example safety, factuality and reasoning.

##### Safety.

Training on all available pretraining data will inevitably include unsafe human written data, from toxicity through to bias and harms. Simply filtering the pretraining data of unsafe content can impoverish the model, and will make it unable to handle unsafe inputs ( Xu et al., 2020 ) . As with other issues, one approach is to attempt to fix these problems in post-training ( Dinan et al., 2019 ; Xu et al., 2021 ; Bai et al., 2022 ) . However, due to poor generalization issues still remain typically when considering out-of-distribution inputs, as is shown by jailbreak attacks ( Zou et al., 2023 ) . It should also be noted that fine-grained control of safety is likely a better choice than simply removing capabilities ( Yi et al., 2025 ) . Korbak et al. (2023) is an early work incorporating safety into pretraining, which reported success with control tokens which incorporate human preferences. More recently, Min et al. (2023) also use a combination of rewriting and special tokens, and report encouraging results. Shilov et al. (2025) proposes a different approach, whereby they alter the training scheme altogether. They split the model’s weights into retain and forget subsets, and guide specific knowledge into the forget subset during training.

##### Factuality.

A number of works have tried to address factuality at post-training time with various approaches. Tian et al. (2023) ; Lin et al. (2024) ; Zhang et al. (2024b) mostly focused on supervised fine-tuning (SFT) and offline RL approaches such as DPO ( Rafailov et al., 2023 ) . Chen et al. (2025b) and Chen et al. (2025a) built specific rewards using retrieval tools to provide measures of factuality for RL training.

##### Reasoning and RL.

Standard pretraining already gives reasoning capabilities, including chain-of-thought emergence ( Kojima et al., 2022 ) . These traits are further amplified via post-training, particularly through reinforcement learning on verifiable rewards (RLVR) ( DeepSeek-AI, 2025 ) . The success of improving reasoning at post-training time has encouraged researchers to try to move post-training techniques further upstream to either mid-training or pretraining. Recent works have augmented pretraining with thinking tokens ( Wang et al., 2025 ; Fujii et al., 2025 ) , and incorporated RL for optimizing thoughts for the next token ( Dong et al., 2025 ; Hatamizadeh et al., 2025 ) or the next set of tokens ( Yu et al., 2024 ; Li et al., 2025 ; Team et al., 2025 ) .

### 1.4 Conclusion

Our work re-envisions pretraining by using a strong post-trained model to provide superior supervision signals. This works in two ways: (i) by providing rewrites on the original streaming pretrain data; and (ii) by acting as a judge. We showed that such a self-improving setup can improve the factuality, safety and overall generation quality of pretrained models.

### 1.5 Discussion

Here we discuss some common questions about our approach.

##### Isn’t this slower than next token prediction pretraining?

Self-Improving Pretraining is indeed slower than standard next token prediction, especially when using rollouts. However, using rewrites and suffixes only, which can work at the start of pretraining, might not be that much slower. Nevertheless, our thinking follows that of Chung (2023) : training methods should be designed to exploit future increases in compute, favoring incentive-based objectives over explicit skill instruction. Hence, using strong post-trained models as judges may prove to be a winner in the long run, especially as pretraining hits a “data wall” where increased compute with next token prediction does not offer gains, in the case that we have “run out of data”.

##### Is making models safe always a good idea?

We showed how our approach can make models safer, but indeed there may be cases where safe generations are not the goal. An example is generating a movie script with dialogue from bad actors, which would necessitate the ability to generate unsafe text. During training, one way to get around this is the use of control tokens, or some other method of fine-grained control of safety, i.e. to train for both safe and unsafe cases, given the control token which can be switched on/off at inference time. We believe this might actually be a better choice than simply removing capabilities ( Yi et al., 2025 ) . As mentioned earlier, Korbak et al. (2023) is an early work incorporating safety into pretraining, which reported success with control tokens which incorporate human preferences.

##### What else can this framework do? How do you generalize it?

We showed that safety, factuality and general quality can be optimized in our framework, e.g. simply by providing different LLM-as-judge prompts. An obvious approach to combine all three methods at the same time is to sum the rewards from the prompts, or potentially combine them into a single prompt. We already showed that combining quality and safety or quality and factuality works, so we believe this should not be difficult. Ideally we would prefer a more generic judge prompt that can capture all these skills well at the same time. Going further, there are other aspects of a powerful model one may wish for pretraining to also capture, i.e. other skills! – an obvious one being stronger reasoning ability. Training chain-of-thought can also fit fairly well into our framework, i.e. switching between rewrites from a strong post-trained model earlier in pretraining (in this case, to rewrite the original suffix to contain chain-of-thought), and then switching to improving rollouts later in training. See subsection 1.3 for existing related work in the area of chain-of-thought augmentation and reinforcement learning. We will address this topic in the next part of the paper.

## 2 Thinking Mid-training: Reinforcement Learning of Interleaved Reasoning

Large language models are typically trained in two stages: pretraining on raw text followed by post-training for instruction-following and reasoning. This creates a fundamental gap where reasoning capabilities must be acquired almost entirely during post-training, as pretraining data lacks explicit reasoning traces. We introduce thinking mid-training , an intermediate training phase that bridges this gap by teaching models to reason on augmented pretraining corpora. Our approach consists of three components: (1) a data augmentation strategy that uses a teacher model to enrich pretraining text with interleaved “thoughts”, or intermediate reasoning steps inserted at semantically appropriate positions, (2) supervised fine-tuning on the augmented corpus to teach model how to interleave thoughts, and (3) reinforcement learning with an LLM judge to optimize the utility of thoughts for predicting subsequent text. Experiments on Llama-3-8B demonstrate that thinking mid-training substantially improves post-training effectiveness: our full pipeline achieves an average accuracy of 0.38 across challenging reasoning benchmarks (GSM8K, MATH-500, AMC23, Olympiad, GPQA-Diamond), compared to 0.12 for direct RL post-training on the base model, a 3.2 × \times improvement, and more than doubled the existing practices of mid-training with raw data. Our results suggest that introducing reasoning earlier in the training pipeline results in models that are not only initially better at reasoning, but also better prepared for reasoning-intensive post-training.

Large language models (LLMs) have achieved remarkable capabilities through a two-stage training paradigm: pretraining on vast corpora of unstructured text, followed by post-training on curated instruction-response pairs ( Ouyang et al., 2022 ; Touvron et al., 2023a ) . pretraining imbues models with foundational knowledge of language, world facts, and basic patterns, while post-training, through supervised fine-tuning (SFT) and reinforcement learning (RL), teaches models to follow instructions, engage in dialogue, and perform complex reasoning ( Wei et al., 2022 ; Zelikman et al., 2022 ) . This clearly defined multi-stage process has proven remarkably effective, yet introduces a fundamental tension: reasoning capabilities are not prioritized during pretraining and must be optimized primarily during post-training.

This gap between pretraining and post-training creates several challenges. First, post-training must simultaneously teach both task-specific formats and general reasoning skills, limiting its efficiency. Second, the raw text consumed during pretraining is presented without explicit reasoning traces, leaving models to learn only surface-level patterns rather than the underlying thought processes. Lastly, recent work on reinforcement learning with verifiable rewards (RLVR) has demonstrated that models can acquire substantial reasoning capabilities through post-training alone ( Guo et al., 2025 ) , but this approach may be fundamentally limited by the reasoning foundations established during earlier training phases.