Self-Improving Pretraining: using post-trained models to pretrain better models — part 5/5
Cleaned-text code points [63353, 75500). Bibliography omitted here; source.md includes it.
Lanchantin et al. (2023) introduce Self-Notes, enabling models to interleave reasoning steps with text at any point, including during the question or context, but it is done by supervising the thoughts on toy tasks. Lyu et al. (2025) show that a simple retrieval-augmented generation pipeline, powered by a diverse and compact web-scale datastore, yields strong improvements on reasoning-intensive benchmarks. Ruan et al. (2025) propose inferring latent thoughts underlying text to improve pretraining data efficiency, demonstrating gains via synthetic data and EM-based bootstrapping. Ishibashi et al. (2025) evaluate continual pretraining with synthetic hidden thoughts, finding that reasoning skills transfer across domains and adapt to problem difficulty. Fernando et al. (2023) present Promptbreeder, a self-referential prompt evolution framework that automatically improves prompt strategies and outperforms hand-crafted baselines. Zelikman et al. (2024) generalize rationale generation to arbitrary text, showing that tokenwise rationales improve zero-shot reasoning and prediction accuracy. Liu et al. (2025a) show that perplexity-based rewards work reasonably well when the task is non-verfiable. Li et al. (2025) propose RLPT, a reinforcement learning paradigm that derives rewards from pretraining data, enabling scalable reasoning improvements without human annotation. Dong et al. (2025) introduce Reinforcement pretraining (RPT) which uses a binary reward on each next token’s prediction after reasoning. Our work differs from these in that we propose a 2-stage process to augment the entire context with reasoning traces at once, and then augment segments autoregressively to refine the reasoning. ### 2.4 Conclusion We have presented thinking mid-training, an intermediate training phase that bridges the gap between pretraining and post-training by explicitly teaching models to reason on augmented pretraining corpora. Our approach addresses a fundamental limitation of current LLM training paradigms: the absence of explicit reasoning traces during pretraining leaves models ill-prepared for the reasoning demands of post-training. Our framework consists of three key contributions: (1) a data augmentation strategy that enriches pretraining text with interleaved thoughts generated by a teacher model, (2) a supervised fine-tuning phase that teaches models the mechanics of interleaved reasoning, and (3) a reinforcement learning phase that optimizes the utility of generated thoughts for predicting subsequent text. Together, these components create a smooth transition from raw text compression to elaborative reasoning. Our experiments demonstrate the effectiveness of this approach. On Llama-3-8B, thinking mid-training combined with RL post-training achieves a 3.2 × \times improvement in average performance across mathematical reasoning benchmarks compared to RL post-training starting from a base model using existing approach. Notably, each component of our pipeline contributes meaningfully: SFT mid-training with thought-augmented data yields a 6 × \times improvement over the base model, and RL mid-training provides additional gains, particularly on the most challenging competition-level problems. These results suggest that reasoning capabilities benefit from being trained as native behavior earlier in the training pipeline. Thinking mid-training offers a principled approach to closing the reasoning gap between pretraining and post-training, ultimately enabling models that are better prepared for complex reasoning tasks. ## Appendix A Additional Judge experiments ### A.1 Suffix judge comparisons for quality We compared several medium-sized post-trained models on the quality task we use for judge training. We used our synthetic data from the SP validation subset and asked Llama3.1-8B-Instruct, Llama3.3-70B-Instruct, DeepSeek-R1-Distill-Llama-8B, and DeepSeek-R1-Distill-Llama-70B. Results are summarized in Table 11 . We found that all models underperform on this task and thus cannot be used as Judge without further fine-tuning. The main problem we found is that the models tend to favor suffixes that feel more complete , rather than those more coherent with respect to the context . Training helps to fix this problem. ### A.2 Suffix judge comparisons for factuality To evaluate different strong post-trained models as judges to measure factuality, here we conduct experiments by prompting GPT-4o, GPT-OSS-120B, and Llama3.1-70B-instruct with a test set of 200 SlimPajama instances. We try 5 versions (v1-v5) of a so called “with-reference” prompt with the (typically human-written) original suffix used as a reference to judge the factuality of a model completion given the prefix, and 4 versions (v1-v4) of a so-called “no reference” prompt, where the original suffix is not given as reference. The prompts are described in subsubsection A.2.1 and subsubsection A.2.1 . A summary of different versions of the with-reference prompts we tried according to various aspects can also be found in Table 12 . We present evaluation results in Tables 13 , 14 and 15 . Overall, we find that by providing the original (typically human-written) suffix as a reference, the post-trained models perform better at the factuality judgment task when judging model generations. Through manual annotation, we find GPT-4o tends to provide the best evaluation results. Then, considering GPT-4o’s prediction as a reference label, we calculate the agreement ratio between GPT-OSS-120B or Llama3.1-70B-instruct with GPT-4o. The results are given in Table 16 . We find that combining both our manual inspection and the overall metrics results, that GPT-OSS-120B performs better as a factuality judge than Llama3.1-70B-instruct, and is thus used in subsequent experiments with the v4 with-reference prompt. #### A.2.1 Factuality Prompts: with reference ##### Prompt Variants v1-v5 We now describe for each prompt version we tried how it differs from the base prompt provided above. V1 Differences from Base: • Explicitly includes the human continuation as ground truth and instructs the evaluator to treat it as the primary reference. • Emphasizes step-by-step reasoning about whether the model continuation logically follows from both the original text and the human continuation. • Focuses on hallucinations, internal inconsistencies, or statements implausible given both references. V2 Differences from Base: • De-emphasizes coherence with the original text; focuses on factual correctness only. • Allows use of general world knowledge as valid ground truth. • Only statements that are false, self-contradictory, or implausible count as hallucinations. • Minor logical or coherence issues with the original text should not be considered hallucinations. • Provides explicit label definitions for “No Hallucination”, “Possible Hallucination”, and “Definite Hallucination”. V3 Differences from Base: • Allows minor invented/unverifiable terms if plausible and not contradicting world knowledge. • De-emphasizes off-topic or loosely connected content. • Only clear factual errors or implausible claims are considered hallucinations. • Plausible but invented terms or creative liberties are treated as “Possible Hallucination” unless they contradict the human continuation or known facts. V4 Differences from Base: • Removes ambiguity around coherence, style, and narrative oddities. • Explicitly tolerates creative, loosely grounded content while still catching true factual errors. • Instructs not to penalize for minor semantic or logical quirks in story continuations. • Narrative oddities, off-topic content, or unusual story events are not hallucinations if plausible or creatively reasonable. V5 Differences from Base: • Most lenient: only clear, unambiguously false, self-contradictory, or impossible statements count as hallucinations. • If uncertain, lean toward “No Hallucination”. • Unusual, speculative, or imaginative content is not penalized. • Plausible inventions or mild factual stretching are at most “Possible Hallucination”. • Ignore coherence gaps, logical quirks, or off-topic continuations unless they make the text factually impossible. #### A.2.2 Factuality Prompts: without reference ##### Prompt Variants v1-v4 We now describe for each prompt version we tried how it differs from the base prompt provided above. V1 Differences from Base: • Focuses on whether the continuation logically follows from the original text. • No reference to human continuation or world knowledge. • Hallucinations include internal inconsistencies or implausible statements given the original text. V2 Differences from Base: • De-emphasizes coherence with the original text; focuses on factual correctness only. • Allows use of general world knowledge as valid ground truth. • Only statements that are false, self-contradictory, or implausible count as hallucinations. • Minor logical or coherence issues with the original text should not be considered hallucinations. • Provides explicit label definitions for “No Hallucination”, “Possible Hallucination”, and “Definite Hallucination”. V3 Differences from Base: • Allows minor invented/unverifiable terms if plausible and not contradicting world knowledge. • De-emphasizes off-topic or loosely connected content. • Only clear factual errors or implausible claims are considered hallucinations. • Plausible but invented terms or creative liberties are treated as “Possible Hallucination” unless they contradict facts. V4 Differences from Base: • Removes ambiguity around coherence, style, and narrative oddities. • Explicitly tolerates creative, loosely grounded content while still catching true factual errors. • Instructs not to penalize for minor semantic or logical quirks in story continuations. • Narrative oddities, off-topic content, or unusual story events are not hallucinations if plausible or creatively reasonable. ## Appendix B Synthetic data generation ### B.1 Unsafe test set To extract unsafe data, we applied two-staged filtering to the RedPajama dataset: first, we used existing tags to extract unsafe content. Specifically, we modify recommended quality filtering rules 2 2 2 https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2 to add a rule that searches for curse words or blocklist content ( Figure 15 ). Filtered data is then split into train, validation and test data. Since we further extract a prefix and suffix from each sample randomly, it might happen that the extracted prefix is safe. To limit testing on purely unsafe prefixes, we then used a strong model – GPT-OSS-120B – to further filter validation and test splits. In particular, we prompt the model to evaluate safety of the prefixes with 8 random seeds, and only use data where all 8 responses judged prefixes as unsafe. We use the same safety prompt we used for judging safety during training ( Figure 3 ). ## Appendix C Evaluation results ### C.1 Evaluation prompts The judge prompts for coherence, FActScore, HaluEval can be found in Figure 16 , Figure 17 , and Figure 18 , respectively. ### C.2 Finegrained evaluation results ## Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" ( ) button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .