Self-Improving Pretraining: using post-trained models to pretrain better models — part 4/5

Cleaned-text code points [47375, 63353). Bibliography omitted here; source.md includes it.

We hypothesize that closing this gap by introducing reasoning earlier in the training pipeline can yield models that are not only better at reasoning out of the box, but also better suited for post-training and ultimately achieve stronger reasoning capabilities. Our key insight is that pretraining data, while lacking explicit reasoning traces, contains rich opportunities for intermediate thinking which can be trained by RL: mathematical derivations benefit from step-by-step explanations, factual passages invite reflection on causes and implications, and narrative text contains implicit logical progressions that can be made explicit.

In this work, we introduce a comprehensive framework for thinking mid-training, a novel training phase that bridges pretraining and post-training by teaching models to perform general reasoning on pretraining corpora. Our approach consists of three key components. The first is mid-training data thinking augmentation, where we leverage a teacher language model to augment pretraining chunks with interleaved “thoughts”, intermediate reasoning steps inserted at semantically appropriate positions within the original text. This creates a corpus where reasoning is explicitly woven into natural text. The second component is thinking SFT mid-training, in which we perform supervised fine-tuning on the augmented corpus, training a student model to produce both the original content and the inserted thoughts. This “cold-start” phase teaches the model the mechanics of interleaved reasoning. The third component is thinking RL mid-training, where we further refine the model’s reasoning through reinforcement learning. Here, the model must generate useful thoughts that help predict subsequent text, with an LLM judge providing rewards based on the quality of predictions. This encourages thoughts that are genuinely beneficial rather than merely imitative.

Following thinking mid-training, we apply standard RL post-training with verifiable rewards on mathematical reasoning tasks. Our experiments demonstrate that thinking mid-training substantially improves the effectiveness of post-training: on Llama-3.1-8B, our full pipeline achieves an average score of 0.3785 across challenging mathem and reasoning benchmarks including GSM8K, MATH-500, AMC23, Olympiad, and GPQA-Diamond compared to 0.1197 for direct RL post-training on the base model. Notably, even the SFT mid-training phase alone yields significant gains, with the RL mid-training phase providing additional improvements particularly on the most challenging competition-level problems.

Our contributions can be summarized as follows. First, we identify and address the reasoning gap between pretraining and post-training, proposing thinking mid-training as an intermediate phase that prepares models for reasoning-intensive post-training. Second, we introduce a data augmentation strategy that enriches pretraining corpora with interleaved thoughts, enabling models to learn reasoning patterns from naturally-occurring text. Third, we propose a two-phase mid-training procedure combining supervised learning for reasoning pattern acquisition with reinforcement learning for reasoning quality optimization. Finally, we demonstrate substantial improvements on mathematical reasoning benchmarks, showing that our approach effectively closes the gap between pretraining and post-training.

### 2.1 Method

We introduce a multi-step procedure for teaching models to reason throughout mid-training and post-training.

#### 2.1.1 Mid-training Data Thinking Augmentation

We introduce a data augmentation strategy that enriches pretraining corpora with intermediate “thoughts”. Given a pretraining corpus 𝒟 \mathcal{D} , we first partition it into chunks of length L L : 𝒟 = { c 1 , c 2 , … , c N } \mathcal{D}=\{c^{1},c^{2},\ldots,c^{N}\} , where each chunk c i c^{i} represents a contiguous segment of text with | c i | ≤ L |c^{i}|\leq L tokens.

For each chunk c i c^{i} , we employ an annotator language model 𝒜 \mathcal{A} to generate an augmented version c ~ i \tilde{c}^{i} that interleaves the original content with generated thoughts: c ~ i = ℳ teacher ​ ( c i , p t ) \tilde{c}^{i}=\mathcal{M}_{\text{teacher}}(c^{i};p_{t})

where p t p_{t} represents the prompt ( Figure 19 ) that instructs the teacher model to insert thoughts at semantically appropriate positions within c i c^{i} . The resulting augmented chunk c ~ i \tilde{c}^{i} takes the form: c ~ i = [ x 1 , τ 1 , x 2 , τ 2 , … , x K , τ K ] \tilde{c}^{i}=[x_{1},\tau_{1},x_{2},\tau_{2},\ldots,x_{K},\tau_{K}] , where x j x_{j} represents segments of the original text and τ j \tau_{j} denotes the generated thoughts, such that concat ​ ( x 1 , … , x K ) = c i \text{concat}(x_{1},\ldots,x_{K})=c^{i} . The final augmented pretraining corpus is constructed as: 𝒟 ~ = { c ~ 1 , c ~ 2 , … , c ~ N } \tilde{\mathcal{D}}=\{\tilde{c}^{1},\tilde{c}^{2},\ldots,\tilde{c}^{N}\} .

#### 2.1.2 Thinking Mid-training

We introduce a two-step mid-training phase. The first is a “cold-start” supervised fine-tuning phase which learns how to think on pretraining data. The second is a reinforcement learning phase which learns how to optimally think before predicting the next sequence.

##### Thinking SFT Mid-training

We perform supervised fine-tuning (SFT) mid-training on half of the augmented corpus, which we call 𝒟 ~ S ​ F ​ T \tilde{\mathcal{D}}_{SFT} using standard next-token prediction. Given a base model ℳ 0 \mathcal{M}_{\text{0}} parameterized by θ \theta , we optimize the following objective: ℒ SFT ​ ( θ ) = − 𝔼 c ~ i ∼ 𝒟 ~ ​ [ ∑ j = 1 | c ~ i | log ⁡ P θ ​ ( c ~ j i ∣ c ~ < j i ) ] \mathcal{L}_{\text{SFT}}(\theta)=-\mathbb{E}_{\tilde{c}^{i}\sim\tilde{\mathcal{D}}}\left[\sum_{j=1}^{|\tilde{c}^{i}|}\log P_{\theta}(\tilde{c}^{i}_{j}\mid\tilde{c}^{i}_{<j})\right]

where c ~ j i \tilde{c}^{i}_{j} denotes the j j -th token in the augmented chunk c ~ i \tilde{c}^{i} , and c ~ < j i \tilde{c}^{i}_{<j} represents all preceding tokens. Importantly, the loss is computed over the entire augmented sequence, including both the original content tokens x j x_{j} and the generated thought tokens τ j \tau_{j} . This allows the model to learn to produce intermediate reasoning steps alongside the original content.

This SFT mid-training phase serves as an intermediate step between initial pretraining and final task-specific fine-tuning, enabling the model to internalize the reasoning patterns demonstrated by the teacher model.

##### Thinking RL Mid-training

While SFT mid-training encourages the model to imitate the teacher’s reasoning patterns, it does not directly optimize for the utility of the generated thoughts. To address this, we introduce a reinforcement learning mid-training phase to further refine the model’s reasoning capabilities on pretraining data.

Given the second half of the augmented pretraining corpus 𝒟 ~ R ​ L \tilde{\mathcal{D}}_{RL} , we process each chunk c ~ i \tilde{c}^{i} by splitting it into a prefix p i p^{i} and a suffix s i s^{i} : c ~ i = [ p i , s i ] \tilde{c}^{i}=[p^{i},s^{i}] where p i p^{i} consists of the initial l l tokens and s i s^{i} contains the remaining tokens, with l < | c ~ i | l<|\tilde{c}^{i}| . For each prefix p i p^{i} , the model ℳ 1 \mathcal{M}_{\text{1}} is tasked with generating a sequence of “thinking” tokens τ ^ i \hat{\tau}^{i} followed by a predicted suffix s ^ i \hat{s}^{i} : [ τ ^ i , s ^ i ] = ℳ 1 ​ ( p i ) [\hat{\tau}^{i},\hat{s}^{i}]=\mathcal{M}_{\text{1}}(p^{i}) , where τ ^ i \hat{\tau}^{i} represents the model’s intermediate reasoning steps and s ^ i \hat{s}^{i} is its prediction of the ground truth suffix s i s^{i} .

To evaluate the quality of the generated suffix, we employ a LLM as a judge. The judge, ℳ judge \mathcal{M}_{\text{judge}} receives both the generated suffix s ^ i \hat{s}^{i} and the ground truth s i s^{i} , and outputs a binary reward r i ∈ { 0 , 1 } r^{i}\in\{0,1\} indicating whether s ^ i \hat{s}^{i} matches s i s^{i} sufficiently well according to predefined criteria (e.g., semantic similarity, factual correctness, or task completion): r i = ℳ judge ​ ( s ^ i , s i ) r^{i}=\mathcal{M}_{\text{judge}}(\hat{s}^{i},s^{i}) .

The RL objective is then to maximize the expected reward over the augmented corpus: ℒ RL ( θ ) = − 𝔼 p i ∼ 𝒟 ~ [ 𝔼 [ τ ^ i , s ^ i ] ∼ ℳ 1 ( ⋅ ∣ p i ) [ r i ] ] \mathcal{L}_{\text{RL}}(\theta)=-\mathbb{E}_{p^{i}\sim\tilde{\mathcal{D}}}\left[\mathbb{E}_{[\hat{\tau}^{i},\hat{s}^{i}]\sim\mathcal{M}_{\text{1}}(\cdot\mid p^{i})}[r^{i}]\right] where θ \theta are the parameters of the model. We optimize this objective using DrGRPO ( Liu et al., 2025b ) .

By incorporating RL mid-training, our method encourages the model not only to imitate the teacher’s reasoning steps, but also to generate thoughts that lead to high-quality, goal-directed completions. This approach leverages the strengths of both supervised and reinforcement learning, resulting in models that reason more effectively and produce more reliable outputs during pretraining.

#### 2.1.3 RL Post-Training

The final stage of the pipeline is to run standard post-training. Given a set of questions 𝒬 \mathcal{Q} from a post-training dataset, the model ℳ 2 \mathcal{M}_{\text{2}} generates thoughts τ \tau and answer y ^ i \hat{y}^{i} for each question Q i ∈ 𝒬 Q^{i}\in\mathcal{Q} . We employ a rule-based reward model, ℳ RLVR \mathcal{M}_{\text{RLVR}} to score the responses compare to the ground truth y i y^{i} : r i = ℳ RLVR ​ ( y ^ i , y i ) r^{i}=\mathcal{M}_{\text{RLVR}}(\hat{y}^{i},y^{i}) .

ℒ RLVR ( θ ) = − 𝔼 p i ∼ 𝒫 [ 𝔼 y ^ i ∼ ℳ 2 ( ⋅ ∣ Q i ) [ r i ] ] \mathcal{L}_{\text{RLVR}}(\theta)=-\mathbb{E}_{p^{i}\sim\mathcal{P}}\left[\mathbb{E}_{\hat{y}^{i}\sim\mathcal{M}_{\text{2}}(\cdot\mid Q^{i})}[r^{i}]\right]

where θ \theta are the parameters of the model. We optimize this using DrGRPO.

### 2.2 Experiments

#### 2.2.1 Experimental Setup

##### Mid-train Data and Models.

We use pretraining corpora containing general reasoning such as DCLM ( Li et al., 2024 ) , FineMath ( Allal et al., 2025 ) as sources, and gpt-oss-120b as the annotator model to augment the raw data with interleaving thoughts. For data used in SFT, the teacher model generates both positions to insert thoughts and the thought tokens. We use an non-overlapping split of the data for RL training, where the teacher model only generates positions to insert thought, while the thought tokens and continuations are generated by the policy model. We also use gpt-oss-120b as the judge to compare the continuation generated by the policy model, after conditioning on the thought tokens, against the original continuation in the raw data.

##### Post-train Data and Models.

We use the DAPO-Math-14k dataset from Yu et al. (2025) . This dataset contains mathematical questions with integer answers. We use math-verify 1 1 1 https://github.com/huggingface/Math-Verify to verify the generated answers against the ground truth answers.

##### Training Framework.

We use fairseq2 ( Balioglu et al., 2023 ) for both SFT and RL training. We run main experiments with Llama-3-8B given that it has not gone through mid-training, and thus provides clean comparisons of different approaches. We also verify the effectiveness RL mid-training on Qwen3-8B, which has shown to be a stronger base model ( Yang et al., 2025 ) .

#### 2.2.2 Evaluations

We compare different approaches of mid-training in terms of pass@k performance after mid-training, as well as final performance after RL post-training. We evaluate on both general-domain and mathematical reasoning tasks.

For mathematical reasoning, we evaluate on GSM8k ( Cobbe et al., 2021 ) , MATH-500 ( Hendrycks et al., 2021 ) , Olympiad ( He et al., 2024 ) , AMC23 ( MAA, n.d. ) . In addition, we evaluate on GPQA-Diamond ( Rein et al., 2024 ) to assess general reasoning.

We use pass@1, averaged of n n sampled responses. We sample n = 16 n=16 responses with temperature 0.6 0.6 and top- p p 0.95 0.95 . The maximum generation length is set to 4096 tokens. Correctness in mathematical reasoning is evaluated using Math-Verify. We use lighteval ( Habib et al., 2023 ) as the standardized implementation.

#### 2.2.3 Results

##### Mid-training Performance

First, we evaluate whether the proposed approach improves reasoning capabilities without further finetuning on downstream tasks.

We show the Llama3-8b-Base results in Table 8 , where we found that simply training on 10B tokens from raw data brings doubles the average performance, although further scaling up data sizes yields slower increase in overall performance. However, SFT on context-augmented data drastically improves average performance from 0.0264 to 0.1249. RL mid-training brings the largest improvement to 0.1896 ( 9 × \times ) despite using much less data.

Figure 11 shows the RL-Midtraining rewards alongside the generated thinking length for the LLama3-8b-Base model. We observe a steady increase of rewards, correlated with a steady increase in thinking length.

(a) RL Mid-training Reward

(b) RL Mid-training Thinking Length

##### Post-training Performance.

We next evaluate how well each mid-training approach prepares the model for downstream RL post-training. We apply standard RLVR post-training to each mid-trained checkpoint using mathematical reasoning tasks with verifiable rewards. Table 9 summarizes the results. The base Llama-3.1-8B model, when directly post-trained with RLVR without any mid-training, achieves an average score of 0.1197. In contrast, our full pipeline SFT mid-training on thinking-augmented data followed by RL mid-training achieves an average of 0.3837 after post-training, representing a 3.2 × \times improvement. Notably, the gains from thinking mid-training compound with post-training: models that undergo SFT mid-training on thinking-augmented data alone achieve substantially higher post-training performance than those trained on raw data, confirming that reasoning patterns learned during mid-training transfer effectively to downstream tasks. These results demonstrate that thinking mid-training not only improves zero-shot reasoning capabilities but also fundamentally enhances the model’s capacity to benefit from subsequent post-training.

Figure 12 shows the RL post-training rewards for different Llama3-8b checkpoints. We see that the models which were RL-mid-trained not only start with higher rewards than the SFT models, but sustain the higher average reward over the course of the 1,000 post-training steps. Furthermore, we observe that as we increase the number of RL mid-training steps, the higher the resulting post-training rewards are.

##### Data Efficiency of RL Mid-training.

We further compare the effects of allocating token budgets in SFT vs. in RL. As is shown in Table 9 , increasing SFT token budget from 7.8B (SFT think 7k steps) to 10.5B (SFT think 10k steps) improves average accuracy from 0.3346 to 0.3480. On the other hand, scaling up RL Mid-train achieves 0.3785 average accuracy with less tokens (8.7B). As pretraining is shifting from compute-bound to data-bound, our approach demonstrates consistent improvement by effectively leveraging compute while less affected by the “data wall”.

#### 2.2.4 Ablations

We find similar results for RL-midtraining on Qwen3-8B-Base models, shown in Table 10 . Surprisingly, we find that for Qwen models, and amount of SFT mid-training (raw or thinking) reduces performance. However, even starting at a worse performing model compared to the Base average accuracy of 0.3572, RL mid-training increases the average accuracy to 0.3660.

### 2.3 Related Work

Wei et al. (2022) introduced chain-of-thought (CoT) prompting to elicit thinking before answering a question. Similarly, Nye et al. (2021) proposed “scratchpads” of post-question thoughts and trained models to generate them. Hao et al. (2024) extend CoT/Scratchpads to use continuous vectors instead of natural language tokens.