Self-Improving Pretraining: using post-trained models to pretrain better models

arxiv:2601.21343

Original source · Paper JSON

Data, training, optimization · 2026-01-29

Cites (8) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• Self-Improving Pretraining: using post-trained models to pretrain better models
• Abstract
• Introduction
• 1 Self-Improving Pretraining for Safety, Factuality and Quality
• 1.1 Method
• 1.1.1 The sequence pretraining task: prefix-conditioned suffix generation
• 1.1.2 Self-Improving pretraining using post-trained models
• Suffix Rewriter
• Suffix Judge
• Policy Model Training
• 1.2 Experiments
• 1.2.1 Models and data
• 1.2.2 Experimental Setup
• Safety Experimental Setup
• Factuality Experimental Setup
• Quality Experimental Setup
• Policy training variants and ablations
• Continual pretraining experiments
• From-scratch pretraining
• Ablations
• 1.2.3 Evaluations
• 1.2.4 Results
• Main results
• Pretraining from-scratch results
• 1.2.5 Analysis & ablations
• Training objective
• Suffix & rewrite vs. rollouts
• Number of rollouts
• Judge choice
• Pivots in pairwise comparison judgments
• 1.3 Related Work
• Safety.
• Factuality.
• Reasoning and RL.
• 1.4 Conclusion
• 1.5 Discussion
• Isn’t this slower than next token prediction pretraining?
• Is making models safe always a good idea?
• What else can this framework do? How do you generalize it?
• 2 Thinking Mid-training: Reinforcement Learning of Interleaved Reasoning
• 2.1 Method
• 2.1.1 Mid-training Data Thinking Augmentation
• 2.1.2 Thinking Mid-training
• Thinking SFT Mid-training
• Thinking RL Mid-training
• 2.1.3 RL Post-Training
• 2.2 Experiments
• 2.2.1 Experimental Setup
• Mid-train Data and Models.
• Post-train Data and Models.
• Training Framework.
• 2.2.2 Evaluations
• 2.2.3 Results
• Mid-training Performance
• Post-training Performance.
• Data Efficiency of RL Mid-training.
• 2.2.4 Ablations
• 2.3 Related Work
• 2.4 Conclusion
• Appendix A Additional Judge experiments
• A.1 Suffix judge comparisons for quality
• A.2 Suffix judge comparisons for factuality
• A.2.1 Factuality Prompts: with reference
• Prompt Variants v1-v5
• A.2.2 Factuality Prompts: without reference
• Prompt Variants v1-v4
• Appendix B Synthetic data generation
• B.1 Unsafe test set
• Appendix C Evaluation results
• C.1 Evaluation prompts
• C.2 Finegrained evaluation results
• Instructions for reporting errors