Self-Improving Pretraining: using post-trained models to pretrain better models — part 1/5

Cleaned-text code points [0, 15595). Bibliography omitted here; source.md includes it.

# Self-Improving Pretraining: using post-trained models to pretrain better models

###### Abstract

Large language models are classically trained in stages: pretraining on raw text followed by post-training for instruction following and reasoning. However, this separation creates a fundamental limitation: many desirable behaviors such as safety, factuality, overall generation quality, and reasoning ability are only added at a late stage, even though the patterns learned earlier strongly shape a model’s capabilities. To tackle this issue, we introduce a new way to pretrain and mid-train models that incorporates these behaviors earlier. We utilize an existing strong, post-trained model to both rewrite pretraining data and to judge policy model rollouts, thus using reinforcement earlier in training. In our experiments, we show this can give strong gains in quality, safety, factuality and reasoning.

## Introduction

Large-scale pretraining on raw text followed by extensive fine-tuning on curated data is by now the classical paradigm to train large language models. This design has a core weakness: properties like safety, factuality, quality, and reasoning are typically layered on after pretraining. Even though the patterns acquired during pretraining largely dictate what the model can ultimately do, no guidance is provided at this early stage concerning these desirable properties that effort should be concentrated on developing them. Current models still exhibit various weaknesses, e.g. exhibiting factuality, safety and reasoning flaws. We hypothesize the current training pipeline is a fundamental cause of this issues, and fixing it could rectify some of these failings.

In this work, we introduce self-improving pretraining: using stronger post-trained models to improve earlier stages of the training pipeline. At a high-level, our approach consists of using this existing strong model to inform earlier stage training of the current policy in two ways: by rewriting pretraining data to encourage desirable behaviors, and as a judge to reward desirable behaviors. Thus, in Section 1 we first introduce a reinforcement-learning–based pretraining approach that rewrites pretraining data, and evaluates candidate continuations using a strong judge model to improve safety, factuality, and overall generation quality directly during pretraining. In Section 2, we propose thinking mid-training, an intermediate training stage that rewrites (augments) pretraining data with interleaved reasoning traces and uses supervised learning and reinforcement learning with a judge to optimize the usefulness of these thoughts. Together, these methods show that incorporating post-trained model rewrites and judgments to guide the policy model early in the training process can substantially improve downstream capabilities. Across safety, factuality, quality, and challenging reasoning benchmarks, our approaches demonstrate large improvements over standard training pipelines, suggesting that leveraging stronger models to shape pretraining data and objectives is a promising direction for building more capable and reliable language models.

## 1 Self-Improving Pretraining for Safety, Factuality and Quality

Ensuring safety, factuality and overall quality in the generations of large language models is a critical challenge, especially as these models are increasingly deployed in real-world applications. The prevailing approach to addressing these issues involves collecting expensive, carefully curated datasets and applying multiple stages of fine-tuning and alignment. However, even this complex pipeline cannot guarantee the correction of patterns learned during pretraining. Therefore, addressing these issues during pretraining is crucial, as it shapes a model’s core behaviors and prevents unsafe or hallucinated outputs from becoming deeply embedded. To tackle this issue, we introduce a new pretraining method that streams documents and uses reinforcement learning (RL) to improve the next K generated tokens at each step. A strong, post-trained model judges candidate generations—including model rollouts, the original suffix, and a rewritten suffix—for quality, safety, and factuality. Early in training, the process relies on the original and rewritten suffixes; as the model improves, RL rewards high-quality rollouts. This approach builds higher quality, safer, and more factual models from the ground up. In experiments, our method gives 36.2% and 18.5% relative improvements over standard pretraining in terms of factuality and safety, and up to 86.3% win rate improvements in overall generation quality.

Standard pretraining works by predicting the next token on large, usually human-written, corpora. Human-written documents vary widely in quality, safety – and to a degree factuality as well. A standard approach is to curate the training data by identifying and removing low quality documents, but issues likely remain ( Nguyen et al., 2025 ) . Typically, final pretrained models can still produce toxic, biased or otherwise unsafe responses. Further, no matter how factual the original training data is, trained models can still hallucinate due to high next token probabilities being seemingly plausible, but not being grounded in reality. In any case, simply removing all low-quality, unsafe or nonfactual data from pretraining contexts will also mean the model does not learn to steer towards quality, safety and factuality given these inputs, for example in dialogue with a human or given such low-quality documents as context at inference time. The standard approach tries to course correct for these issues during post-training, but this cannot guarantee to fix these patterns, which are inherently core behaviors of the pretrained model ( Itzhak et al., 2025 ) .

In this work we propose a new scheme for pretraining, quite different from the next token prediction paradigm, termed Self-Improving Pretraining . Our overall setup is depicted in Figure 1 . First, we assume we have access to an existing strong post-trained model, typically trained from a previous iteration of the self-improving cycle. We use this strong model to help pretrain our policy model. Second, during pretraining, we apply and learn from sequence generation rather than next token prediction, so that the model can more accurately learn how to generate sequences , which is the goal during deployment. We argue that only addressing high quality, safe and factual sequence generation at post-training time may already be too late. Our method thus streams the pretraining data and at each step splits it into the most recent N N tokens (termed suffix), conditioned on the remaining earlier context (prefix). The existing post-trained model at this point is prompted to rewrite the suffix to steer away from potential unsafe or otherwise low-quality prefixes towards a high quality suffix, which can be used to pretrain our policy model. Third, the existing post-trained model is used as a judge to evaluate the original suffix, the rewrite, and rollouts from the current policy model. This is used to assign rewards and pretrain the policy via reinforcement learning (RL). Early in pretraining, the process relies on the original and rewritten suffixes; as the model improves, RL rewards high-quality rollouts.

In our experiments, we find strong gains in performance across a broad set of different evaluations compared to standard next token prediction, in both from-scratch and continual pretraining settings. For example in the latter continual pretraining setting, we obtain win rates in generation quality of up to 86.3% over the standard pretraining baseline, and relative improvements of 36.2% and 18.5% in terms of factuality and safety. Similarly in the from-scratch setting we observe absolute gains in generation quality win rate of 31.1% and a 14.4% relative improvement in safety. We provide a detailed analysis and ablation studies of the optimization strategies that contribute to these wins.

### 1.1 Method

#### 1.1.1 The sequence pretraining task: prefix-conditioned suffix generation

We re-envision pretraining as a sequence learning task, rather than next token prediction. To this end, we segment the stream of pretraining data into chunks of size N N , where the current chunk x j x_{j} is termed the suffix, and contiguous chunks in the context are termed the prefix, denoted x 1 , … , j − 1 x_{1,\dots,j-1} .

The sequence pretraining task is thus to generate a high quality sequence of length N N given the prefix : x ¯ j ∼ π ( ∗ | x 1 , … , j − 1 ) , \bar{x}_{j}\sim\pi(*|x_{1,\dots,j-1}), where π \pi is our policy model, to be trained. We limit this generation x ¯ j \bar{x}_{j} to be N N tokens, and can compare it to the known suffix x j x_{j} present in the pretraining data for judgment purposes. However, crucially, we should not expect or always desire an exact match with the suffix, and in fact in many cases will not want one, e.g. supposing the suffix is low-quality, unsafe or nonfactual. However, in the case of high-quality suffixes in the pretraining data, they can act directly as references that we would like our policy model to mimic. Our proposed sequence pretraining will thus makes use of an existing teacher model that can differentiate between these cases, as described in the next section.

#### 1.1.2 Self-Improving pretraining using post-trained models

Our self-improvement framework assumes we already have access to a fully trained (i.e., first pre- and then post-trained) model. This model has effectively absorbed information from across the entire pretrain and post-train datasets already – and this expertise can now be brought to bear on individual examples in the pretraining datasets to train a new model using an effectively superior training signal than the one from which it was trained itself.

We consider using this fixed teacher model in two ways: as a rewriter and as a judge .

##### Suffix Rewriter

Given a prefix x 1 , … , j − 1 x_{1,\dots,j-1} and a suffix x j x_{j} , the task of the rewriter is to produce a rewrite of the suffix x ^ j \hat{x}_{j} that is superior to x j x_{j} for policy training . Policy training would proceeed using the same suffix x 1 , … , j − 1 x_{1,\dots,j-1} but with the rewrite x ^ j \hat{x}_{j} as the target.

There are various ways that the rewrite x ^ j \hat{x}_{j} can be superior to the suffix x j x_{j} during training:

• Overall quality: if the suffix is low quality, e.g. comes from a low quality part of the pretraining corpus, the rewriter can improve it, making the training target higher quality.

• Safety: if the prefix and suffix are unsafe, the rewriter can steer the model towards a safe suffix given an unsafe prefix. Note this is quite different to simply rewriting the whole original document, which would mean the model is no longer exposed to unsafe inputs.

• Augmentation: rewriting the data in various ways can improve performance, as has been shown in the offline setting of rewriting entire documents. This has been shown to improve diversity and knowledge ( Hao et al., 2025 ; Allen-Zhu and Li, 2023 ) , quality ( Nguyen et al., 2025 ) , and reasoning ability ( Wang et al., 2025 ; Ishibashi et al., 2025 ) . Our setting allows the model to steer from natural input (prefix) data towards new augmentations (via a rewritten suffix).

To build such a rewriter we can either directly prompt an existing post-trained model or fine-tune it further especially for this task. We detail our approach in subsection 1.2 .

##### Suffix Judge

Given a prefix x 1 , … , j − 1 x_{1,\dots,j-1} and possible completions x ¯ j \bar{x}_{j} , the task of our judge is to discern which completion is superior as a target for policy training .

There are thus various ways that a judge can provide signal to improve the policy model, including: • Overall quality: if the suffix, rewrite or certain rollouts are low quality, they will receive low reward. At the start of training, rollouts are likely to be poor and the suffix or rewrite may receive higher reward. After sufficient training, rollouts are more likely to receive high reward.

• Safety: if the prefix and suffix are unsafe, the rewrite or rollouts can steer the model towards a safe suffix given an unsafe prefix. Among the multiple policy rollouts the judge can choose between them to encourage safety amongst model generations.

• Factuality: similarly, after sufficient training, selecting the most factual generations among the rollouts can improve the factuality of the policy model.

Similarly to building the rewriter, to build a judge we can either directly prompt an existing post-trained model, or further fine-tune it especially for this task. In our experiments, we consider both settings. We also consider judging each of the above — quality, safety and factuality — by prompting the post-trained judge model for each individually. The prompts we employ are given in Figure 3 , Figure 3 , and Figure 4 . We detail our full approach in subsection 1.2 .

##### Policy Model Training

Putting it all together, we train our policy model using the sequence pretraining task described in subsection 1.1 . We assume we have access to a post-trained model that can act as a suffix judge and a suffix rewriter, as described above.

For each prefix, we consider several candidate completions during online training. We can consider (i) the original suffix, (ii) a rewritten suffix; and (iii) K K rollouts x ¯ j k \bar{x}_{j}^{k} , k = 1 , … , K k=1,\dots,K from the current policy π \pi .

The suffix judge is used to provide rewards for online RL by scoring the provided completions. In our experiments we consider both online DPO ( Qi et al., 2024 ; Lanchantin et al., 2025 ) and reward-filtered negative log-likelihood training (RF-NLL) ( Christiano et al., 2017 ) , but other update algorithms are possible. Online DPO has shown performance comparable to GRPO ( Lanchantin et al., 2025 ) . Unlike GRPO, however, DPO is an off-policy algorithm that allows learning from sequences not generated from the current policy, such as the original suffix or rewrites, making it suitable for our approach. For online DPO we take the chosen completion as the highest scoring, and the rejected as the lowest scoring. For RF-NLL we simply take the highest scoring to conduct an NLL update.

At the beginning of training, we expect the rollouts from the policy to be low quality. Hence, the original suffix and rewrite are most important at this stage. We thus expect that rewarding rollouts should be introduced after sufficient examples have already been seen. Using a rewriter, however, can improve training starting from the initial updates. In our experiments we consider various ablations of including candidate completions of type (i) original, (ii) rewrite and (iii) rollouts, as well as the number of rollouts K K .

### 1.2 Experiments

#### 1.2.1 Models and data

Models. We primarily use the pretrained Llama2 1.4 billion parameter model as a baseline policy model ( Touvron et al., 2023b ) , and conduct continual sequence pretraining from that checkpoint. Additionally, we conduct pretraining experiments where we train the same model from scratch by first re-initializing the weights. For the sequence pretraining task, we use chunk size N = 128 N=128 . Both suffix judge and rewriter need to have strong instruction-following capabilities, as such we compare two models: (1) fine-tuned Llama3.1-8B-Instruct ( Dubey et al., 2024 ) ; and (2) prompted GPT-OSS-120B ( OpenAI, 2025 ) .