The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models — part 1/3

Cleaned-text code points [0, 15679). Bibliography omitted here; source.md includes it.

# The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models

###### Abstract

Improving the reasoning capabilities of large language models (LLMs) typically requires supervised fine-tuning with labeled data or computationally expensive sampling. We introduce Unsupervised Prefix Fine-Tuning (UPFT), which leverages the observation of Prefix Self-Consistency – the shared initial reasoning steps across diverse solution trajectories – to enhance LLM reasoning efficiency. By training exclusively on the initial prefix substrings (as few as 8 tokens), UPFT removes the need for labeled data or exhaustive sampling. Experiments on reasoning benchmarks show that UPFT matches the performance of supervised methods such as Rejection Sampling Fine-Tuning, while reducing training time by 75% and sampling cost by 99%. Further analysis reveals that errors tend to appear in later stages of the reasoning process and that prefix-based training preserves the model’s structural knowledge. This work demonstrates how minimal unsupervised fine-tuning can unlock substantial reasoning gains in LLMs, offering a scalable and resource-efficient alternative to conventional approaches.

## 1 Introduction

Large language models (LLMs) have demonstrated remarkable performance across a wide range of natural language understanding and generation tasks, primarily due to large-scale pre-training and subsequent instruction fine-tuning on high-quality datasets ( Longpre et al., 2023 ; Touvron et al., 2023 ) . Despite these successes, enabling LLMs to exhibit systematic reasoning capabilities remains a challenging endeavor. In multiple domains—from mathematical problem solving to logical and commonsense reasoning—models often rely on large amounts of human-annotated data or extensive sampling-and-filtering pipelines to achieve high accuracy.

Recent inquiry has introduced approaches such as Rejection Sampling Fine-Tuning (RFT; Yuan et al. 2023 ) and Self-Taught Reasoner (STaR; Zelikman et al. 2022 ) to leverage model-generated solutions for iterative self-improvement , often requiring multiple candidate responses and subsequent filtering or verification steps. While these methods can yield impressive gains, they are time-consuming, resource-intensive, and assume ready access to correct targets or verification mechanisms — particularly challenging when no reliable ground-truth is available.

In this paper, we propose an unsupervised fine-tuning method that requires only a single pass of model-generated responses per question, coupled with prefix-based fine-tuning. Our key insight is that different solution paths often share a common initial reasoning phase, which we call “Prefix Self-Consistency”. By fine-tuning on these minimal prefixes (as few as 8 tokens), we effectively guide the model’s inherent reasoning structures toward more systematic solution strategies while avoiding the complexity and cost of large-scale or iterative filtering. Moreover, we preserve the model’s overall problem-solving format through a small proportion of full-token fine-tuning experiments, ensuring the model does not lose its length generalization and instruction-following abilities.

We conduct comprehensive experiments across four training corpora and evaluate our method on four widely used reasoning benchmarks. UPFT demonstrates exceptional data efficiency and flexibility, outperforming conventional full-token fine-tuning in an unsupervised sampling setting. Furthermore, UPFT achieves performance competitive with the widely used RFT method, which relies on labeled verification or large-scale rejection sampling, while requiring only 25% of the training time and 1% of the sampling time. Our approach can be easily adapted to various datasets, tasks, and LLM architectures, highlighting its flexibility and universality for developing robust problem-solving capabilities in large language models.

Our main contributions are as follows: 1. We identify Prefix Self-Consistency as a critical phenomenon, showing that early reasoning steps are highly consistent across trajectories, enabling efficient self-improvement learning.

2. We propose an unsupervised fine-tuning method UPFT that leverages only prefix substrings (i.e., minimal initial tokens of model-generated responses), eliminating the need for labeled data or rejection sampling.

3. We conduct comprehensive empirical validation to demonstrate that UPFT exhibits exceptional data efficiency and versatility, outperforming vanilla full-token fine-tuning in unsupervised settings and achieving competitive performance with RFT while drastically reducing sampling and tuning overhead.

## 2 Prefix Self-Consistency

A central observation of this work is that different solution trajectories for the same question often share a common initial reasoning phase, even if their later steps diverge substantially. We term this phenomenon prefix self-consistency . In other words, when an LLM generates multiple solutions for a single question, the opening tokens – typically restatements of the problem or the setup of its initial logical steps – tend to be highly consistent across these separate responses. In this section, we identify two characteristics of reasoning prefixes to demonstrate the existence and plausibility of prefix self-consistency. We report results on 500 questions randomly sampled from the PRM training dataset.

#### Early reasoning steps are highly consistent across reasoning trajectories.

We sampled 1,000 trajectories for each question instance (using a temperature of 0.6) and calculated the average trajectories per prefix of different lengths, as shown in Figure 2(a) . The results confirm that reasoning processes exhibit strong prefix self-consistency, with a remarkably high degree of prefix overlap among multiple trajectories for both models. As we increase t t (i.e., consider longer prefixes), the average number of samples per prefix pattern decreases, yet both models maintain consistent patterns well beyond the very first tokens. Notably, the math-specialized Qwen-Math-7B-Instruct preserves shared prefixes more consistently than the general-purpose Llama-3.1-8B-Instruct, as evidenced by the higher values in all columns. This suggests that Qwen-Math-7B-Instruct’s generation process is more tightly anchored in its initial reasoning steps, whereas Llama-3.1-8B-Instruct introduces more prefix variability.

#### Errors predominantly occur in later reasoning steps.

To empirically validate the plausibility of the reasoning prefix, we conducted 32 rollout samplings for each token in both a correct and an incorrect trajectory for each question. Figure 2(b) illustrates the success rate of these rollout samplings across both correct and incorrect trajectories. Two key observations can be made: • Incorrect trajectories diverge significantly from correct ones in later reasoning steps. The rollout success rate for correct trajectories steadily rises as t t grows for both models. For example, with Llama, the success rate starts at 54.2% at t = 0 t=0 and reaches 96.2% by t = 512 t=512 . This trend is expected: sampling from the later tokens of a correct trajectory leverages more accurate contextual information, increasing the likelihood of staying on a correct path. In contrast, the rollout success rate for incorrect trajectories declines substantially as t t increases, indicating that once an incorrect path is taken, recovering through rollout sampling becomes far less likely. This contrast highlights that errors tend to occur in the later stages of generation.

• Initial reasoning prefixes exhibit self-consistency. At the earliest token positions ( t ≤ 16 t\leq 16 ), the rollout success rates for both correct and incorrect trajectories are strikingly similar. For Qwen-Math-7B-Instruct at t = 4 t=4 , the success rates are 85.7% and 86.1% for correct and incorrect trajectories, respectively. This near-identical performance in early steps suggests that the initial tokens generated – whether they lead to a correct or incorrect final answer – are statistically indistinguishable in terms of leading to a correct result when used as a starting point for rollout sampling.

Overall, these results indicate that while mistakes in reasoning are more prone to appear and accumulate in later steps, the initial reasoning prefixes generated by LLMs exhibit a considerable degree of self-consistency. This consistency is reflected in the similar early-stage rollout success rates across both correct and incorrect trajectories. Consequently, enhancing the accuracy and robustness of these early reasoning steps may be a key factor in improving the overall reliability of complex reasoning tasks.

## 3 Methodology

Based on the aforementioned observation, we firstly introduce the conventional widely-used reject sampling strategy in Section 3.1 , we then provide a Bayesian explanation and identify the coverage and accuracy of Prefix Self-Consistency in Section 3.2 . Finally, based on this observation, we propose our UPFT in Section 3.3 .

### 3.1 Preliminary

A prevalent strategy for enhancing reasoning in Large Language Models (LLMs) involves Supervised Fine-Tuning (SFT) using demonstrations of correct reasoning processes, often in the form of chain-of-thought (CoT) trajectories. This approach typically entails generating multiple reasoning traces from a base model and subsequently selecting only those traces where the extracted final answer aligns with the ground truth. These selected, correct reasoning trajectories are then used to fine-tune the base model via SFT. Given the inherent resemblance of this selection process to reject sampling, this methodology is also entitled with Rejected Fine-Tuning (RFT).

Formally, consider a dataset ( x , y ) ∼ 𝒟 (x,y)\sim\mathcal{D} , where each x x represents an input and y y is the corresponding ground-truth answer. For each input x x , we first generate a set of reasoning traces { r ( k ) } k = 1 K \{r^{(k)}\}_{k=1}^{K} from the base model p ( ⋅ | x ; θ ) p(\cdot|x;\theta) parameterized by θ \theta . Then, for each input x x , we filter these generated traces and uniformly sample a correct reasoning trace r r from the set of traces whose final answer matches the ground truth y y : r ( k ) ∼ p ( ⋅ | x ; θ ) \displaystyle r^{(k)}\sim p(\cdot|x;\theta) r ∈ R 𝟙 ( r ( k ) , y ) \displaystyle r\in_{R}\mathbb{1}(r^{(k)},y) Function 𝟙 ​ ( r , y ) \mathbb{1}(r,y) denotes the rejection sampling, and it is a selection function that only picks the reasoning trace r r whose final answer is y y , and ∈ R \in_{R} denotes that the final selected reasoning trace r r is uniformly selected from { r ( k ) } k = 1 K \{r^{(k)}\}_{k=1}^{K} . Consequently, the overall RFT objective is to maximize the following objective: 𝔼 ( x , y ) ∼ 𝒟 , r ( k ) ∼ p ( ⋅ | x ; θ ) r ∈ R 𝟙 ( r ( k ) , y ) ​ log ⁡ p ⁡ ( r | x ) \mathbb{E}_{\begin{subarray}{c}(x,y)\sim\mathcal{D},~r^{(k)}\sim p(\cdot|x;\theta)\\ r\in_{R}\mathbb{1}(r^{(k)},y)\end{subarray}}\log p(r|x) (1) where the ground-truth answer y y internally exists in the correct reasoning trace r r .

### 3.2 Modeling Coverage and Accuracy

We demonstrate that the previous RFT process can be naturally interpreted within a Bayesian framework. Consider objective in Equation 1 , the prediction of an answer y y given an input x x . From a probabilistic perspective, we can decompose the conditional probability p ⁡ ( y | x ) p(y|x) by marginalizing over all possible reasoning traces r r that could lead to the answer, which is: log ⁡ p ⁡ ( y | x ) \displaystyle\log p(y|x) = log ∑ r p ( y , r | x ) \displaystyle=\log\sum_{r}p(y,r|x) (2) = log ∑ r p ( y | r , x ) p ( r | x ) \displaystyle=\log\sum_{r}p(y|r,x)p(r|x) (3) ≥ ∑ r p ⁡ ( r | x ) ​ log ⁡ p ⁡ ( y | r , x ) \displaystyle\geq\sum_{r}p(r|x)\log p(y|r,x) (4) = 𝔼 r ∼ p ( ⋅ | x ) log p ( y | r , x ) \displaystyle=\mathbb{E}_{r\sim p(\cdot|x)}\log p(y|r,x) (5) where we leverage the total probability rule in Eq. 2 , Bayes’s Theorem in Eq. 3 , and Jensen’s inequality in Eq. 4 , which finally yields an expectation term in Eq. 5 . It reveals that the log-likelihood of predicting the correct answer log ⁡ p ⁡ ( y | x ) \log p(y|x) is lower-bounded by two factors: • Coverage : The expectation 𝔼 r ∼ p ( ⋅ | x ) [ ⋅ ] \mathbb{E}_{r\sim p(\cdot|x)}[\cdot] is with respect to the distribution of reasoning trace p ⁡ ( r | x ) p(r|x) . The term p ⁡ ( r | x ) p(r|x) denotes a prior probability of reasoning trace r r given input x x , which denotes the prefix coverage of the entire reasoning trace space. This corresponds to the average trajectory covered by prefixes in Section 2 indicated by Figure 2(a) .

• Accuracy : Term p ⁡ ( y | r , x ) p(y|r,x) within the expectation can be viewed as the likelihood of the answer y y being correct given a specific reasoning trace r r , which is the accuracy. This is also observed by our previous observation in Section 2 in Figure 2(b) .

Intuitively, a higher p ⁡ ( r | x ) p(r|x) for relevant reasoning traces that cover a broad range of problem-solving approaches indicates a broader exploration of potential solution paths. Simultaneously, a larger log ⁡ p ⁡ ( y | r , x ) \log p(y|r,x) implies that given a reasoning trace r r , the model is more likely to arrive at the correct answer y y .

Therefore, maximizing this lower bound necessitates enhancing both the coverage of a wide range of effective reasoning traces and the accuracy of each trace in reaching the desired solution. In terms of RFT, it is a specific example of trading off these two key factors: RFT maximizes the accuracy of the reasoning trace by reject sampling while it neglects the coverage of the reasoning trace by only selecting one reasoning trace.

### 3.3 Unsupervised Prefix Fine-Tuning

Given the observed dynamic shift from coverage to accuracy in previous content, in this subsection, we are motivated to investigate the existence of an optimal prefix length that balances both objectives in Equation 5 .

#### Prefix Coverage and Accuracy

To incorporate the concept of prefix spans, we utilize the chain rule for conditional probabilities. We can decompose the probability of the full reasoning trace r r given input x x as: p ⁡ ( r | x ) \displaystyle p(r|x) = p ⁡ ( r < t , r ≥ t | x ) \displaystyle=p(r_{<t},r_{\geq t}|x) = p ⁡ ( r < t | x ) ​ p ​ ( r ≥ t | r < t , x ) \displaystyle=p(r_{<t}|x)p(r_{\geq t}|r_{<t},x)

where r < t r_{<t} denotes the prefix of r r before time step t t . Substituting this decomposition of p ⁡ ( r | x ) p(r|x) into the original lower bound from Equation 4 , we get (see Appendix A for the complete proof): log p ( y | x ) ≥ 𝔼 r < t ∼ p ( ⋅ | x ) [ L ( r < t , x ) ] \displaystyle\log p(y|x)\geq\mathbb{E}_{r_{<t}\sim p(\cdot|x)}[L(r_{<t},x)] (6) where L ⁡ ( r < t , x ) L(r_{<t},x) is: 𝔼 r ≥ t ∼ p ( ⋅ | r < t , x ) [ log p ( y | r < t , r ≥ t , x ) ] \displaystyle\mathbb{E}_{r_{\geq t}\sim p(\cdot|r_{<t},x)}[\log p(y|r_{<t},r_{\geq t},x)] (7) Equation 6 presents the original Jensen’s inequality lower bound in terms of prefix spans of the reasoning trace. It also states the importance of coverage and accuracy of the prefix span r < t r_{<t} :

• Prefix Coverage : The outer expectation in Equation 6 𝔼 r < t ∼ p ( ⋅ | x ) [ ⋅ ] \mathbb{E}_{r_{<t}\sim p(\cdot|x)}[\cdot] is with respect to the distribution of prefixes p ⁡ ( r < t | x ) p(r_{<t}|x) . The term p ⁡ ( r < t | x ) p(r_{<t}|x) represents the prior probability of the model generating a prefix reasoning trace r < t r_{<t} given the input x x . This denotes the coverage of prefix reasoning traces.