The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models — part 3/3
Cleaned-text code points [31633, 38794). Bibliography omitted here; source.md includes it.
However, a significant limitation of many existing approaches, particularly when considering complex tasks like mathematical reasoning, lies in their explicit reliance on sampling complete correct reasoning traces from LLMs. Although more recent studies such as s1 ( Muennighoff et al., 2025 ) and LIMO ( Ye et al., 2025 ) show that a small number of data can unlock strong generalization in LLMs for reasoning tasks, these approaches require external strong LLMs, which naturally are limited by external LLMs’ ability and cannot broaden the knowledge boundary by itself. Consequently, a critical question arises: How can we design an efficient and scalable method to enhance LLM reasoning without external supervision?
Our approach answers this question by leveraging the latent reasoning structures LLMs acquire during pre-training on large corpora ( Brown et al., 2020 ) , which can be aligned using a small subset of high-quality data ( Muennighoff et al., 2025 ; Ye et al., 2025 ) . We aim to enhance reasoning capabilities without annotated data by exploiting these inherent structures. Specifically, we observe that reasoning trajectories sampled by LLMs for the same question often share common prefix substrings, indicating a locally correct supervision signal, which we term Prefix Self-Consistency. Errors typically occur in later steps of the trajectory (see Appendix B for details).
#### Unsupervised Learning
Unsupervised learning encompasses diverse methodologies, including pseudo-labeling ( Ye et al., 2020 ) , pivot-based approaches ( Pan et al., 2010 ) , and adversarial networks ( Ganin et al., 2016 ) . Self-supervised learning has since emerged as a dominant paradigm, particularly for language models, leading to the success of pre-trained models like BERT ( Kenton & Toutanova, 2019 ) and others ( Han et al., 2021 ; Radford et al., 2019 ) . More recently, self-improvement techniques, such as self-rewarding ( Chen et al., 2024 ; Yuan et al., 2024 ) , further advance unsupervised learning by enabling models to iteratively enhance performance without external supervision. Building upon self-improvement, AL ( Ji et al., 2024 ) extends this paradigm to unsupervised domain adaptation. In contrast to these general approaches, this paper focuses on discovering self-supervised signals specifically for mathematical reasoning. We introduce UPFT, a simple and effective unsupervised post-training method requiring only questions and the LLM itself.
#### Self-Training and Self-Improvement.
A family of methods, starting with STaR ( Zelikman et al., 2022 ) , reinforced self-training ( Gulcehre et al., 2023 ) , and rejection fine-tuning ( Yuan et al., 2023 ) , leverages the solutions generated by large language models (LLMs) to iteratively update and improve the models themselves. These techniques involve fine-tuning the model on the generated solutions that produce correct answers. ReST EM ( Singh et al., 2023 ) views this fine-tuning process as expectation-maximization-based reinforcement learning (RL) for a solution-generating agent. Wang et al. (2023a) propose using a contrastive loss to enhance the likelihood of correct solutions over incorrect ones. The discovery of successful solutions presents a significant exploration challenge. Luong et al. (2024) demonstrated that RL-based fine-tuning of an LLM is particularly difficult unless preceded by several steps of supervised fine-tuning. In An et al. (2023) , a more powerful LLM was employed to edit the incorrect rationales generated by a smaller model, thereby providing positive data for its fine-tuning. However, Huang et al. (2023) argued that LLMs have limited capacity to correct their own reasoning flaws.
## 6 Conclusion
In this work, we presented an unsupervised fine-tuning method that enhances the reasoning capabilities of large language models using only prefix substrings as minimal guidance. Our approach leverages the inherent reasoning structures within pretrained models, exploiting the phenomenon of Prefix Self-Consistency where different reasoning trajectories share common prefixes. Extensive experiments demonstrated that our method outperforms traditional full-token fine-tuning and achieves performance comparable to supervised approaches like RFT, with significantly reduced training and inference times. This work highlights the potential of minimal unsupervised fine-tuning in improving the reasoning abilities of LLMs without relying on external supervision or extensive computational resources. Future work will explore the application of this method to other challenging tasks and investigate the theoretical underpinnings of Prefix Self-Consistency in more depth.
## Appendix A Proof of Prefix Lower bound
Following previous Equation 4 , we have: log p ( y | x ) \displaystyle\log p(y|x) ≥ ∑ r p ( r | x ) log p ( y | r , x ) \displaystyle\geq\sum_{r}p(r|x)\log p(y|r,x) = ∑ r p ( r < t | x ) p ( r ≥ t | r < t , x ) log p ( y | r < t , r ≥ t , x ) \displaystyle=\sum_{r}p(r_{<t}|x)p(r_{\geq t}|r_{<t},x)\log p(y|r_{<t},r_{\geq t},x) = ∑ r < t [ p ( r < t | x ) ∑ r ≥ t p ( r ≥ t | r < t , x ) log p ( y | r < t , r ≥ t , x ) ] \displaystyle=\sum_{r_{<t}}\left[p(r_{<t}|x)\sum_{r_{\geq t}}p(r_{\geq t}|r_{<t},x)\log p(y|r_{<t},r_{\geq t},x)\right] = ∑ r < t p ( r < t | x ) [ ∑ r ≥ t p ( r ≥ t | r < t , x ) log p ( y | r , x ) ] \displaystyle=\sum_{r_{<t}}p(r_{<t}|x)\left[\sum_{r_{\geq t}}p(r_{\geq t}|r_{<t},x)\log p(y|r,x)\right] = ∑ r < t p ( r < t | x ) L ( r < t , x ) \displaystyle=\sum_{r_{<t}}p(r_{<t}|x)L(r_{<t},x) = 𝔼 r < t ∼ p ( ⋅ | x ) [ L ( r < t , x ) ] \displaystyle=\mathbb{E}_{r_{<t}\sim p(\cdot|x)}[L(r_{<t},x)] (9) where, L ( r < t , x ) \displaystyle L(r_{<t},x) = ∑ r ≥ t p ( r ≥ t | r < t , x ) log p ( y | r < t , r ≥ t , x ) \displaystyle=\sum_{r_{\geq t}}p(r_{\geq t}|r_{<t},x)\log p(y|r_{<t},r_{\geq t},x) = 𝔼 r ≥ t ∼ p ( ⋅ | r < t , x ) [ log p ( y | r < t , r ≥ t , x ) ] \displaystyle=\mathbb{E}_{r_{\geq t}\sim p(\cdot|r_{<t},x)}[\log p(y|r_{<t},r_{\geq t},x)] (10)
## Appendix B Case study of Prefix Data
We provide a case study of prefix data shown in Figure 5 .
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .