The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models

arxiv:2503.02875

Original source · Paper JSON

Post-training · 2025-03-04

Cites (4) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models
• Abstract
• 1 Introduction
• 2 Prefix Self-Consistency
• Early reasoning steps are highly consistent across reasoning trajectories.
• Errors predominantly occur in later reasoning steps.
• 3 Methodology
• 3.1 Preliminary
• 3.2 Modeling Coverage and Accuracy
• 3.3 Unsupervised Prefix Fine-Tuning
• Prefix Coverage and Accuracy
• Sampling Only First Few Tokens
• Structure Tuning
• 4 Experiment
• 4.1 Experiments Setup
• Backbone LLMs
• Fine-Tuning
• Benchmarks
• Tuning Scenarios
• 4.2 Ablation Study
• Impact of Prefix Length
• Impact of Structure Tuning Ratio
• 4.3 Unsupervised Fine-Tuning
• UPFT demonstrates superior performance compared to SFT in unsupervised fine-tuning.
• The benefits of UPFT are more pronounced on complex reasoning tasks.
• The U-Hard dataset maximizes UPFT’s potential through difficulty-focused curation.
• UPFT achieves dramatic efficiency gains through minimal token training.
• 4.4 Supervised Fine-Tuning
• UPFT achieves competitive performance with supervised methods while requiring significantly fewer tokens in both sampling and tuning.
• UPFT offers seamless integration with label verification for enhanced accuracy.
• 5 Related Work
• Unsupervised Learning
• Self-Training and Self-Improvement.
• 6 Conclusion
• Appendix A Proof of Prefix Lower bound
• Appendix B Case study of Prefix Data
• Instructions for reporting errors