The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models — part 2/3

Cleaned-text code points [15679, 31633). Bibliography omitted here; source.md includes it.

• Prefix Accuracy : For each prefix r < t r_{<t} , the term L ⁡ ( r < t , x ) L(r_{<t},x) in Equation 7 is the conditional lower bound given the prefix r < t r_{<t} . It is the expected log-likelihood of the answer y y given that the reasoning process starts with prefix r < t r_{<t} , averaging over all possible suffixes r ≥ t r_{\geq t} that can follow r < t r_{<t} . This can be viewed as the expected accuracy when reasoning is conditioned to start with prefix r < t r_{<t} .

We have mathematically derived a representation of Jensen’s inequality lower bound in terms of prefix spans for both prefix coverage and prefix accuracy. To achieve superior performance through prefix fine-tuning, a specific prefix length t t is required to trade off both prefix coverage and prefix accuracy terms.

#### Sampling Only First Few Tokens

Motivated by this observation, we develop our proposed UPFT, which is apt to maximize the coverage of the reasoning trace while maintaining a relatively high accuracy by only learning from the proper prefix of the reasoning trace. This strategy shares several advantages. First, UPFT exhibits enhanced coverage of all possible correct reasoning traces. As demonstrated in Section 2 and Section 3.2 , the prefix of the reasoning traces represents a set of reasoning traces with the identical prefix, which increases the coverage term of Equation 5 ’s lower bound. Second, by solely decoding a certain length of the prefix, UPFT inherently enjoys improved computational efficiency. In contrast, conventional Reasoning From Traces (RFT) methods necessitate rejecting sampling over entire reasoning traces. This approach incurs a significant inference burden, particularly in scenarios where generating valid full reasoning traces from the base model proves challenging.

Our strategy consists of the following steps: • Given a training set ( x , y ) ∈ 𝒟 (x,y)\in\mathcal{D} , we only decode a prefix span r < t r_{<t} of reasoning trace from the base model r < t ∼ p ( ⋅ | x ; θ ) r_{<t}\sim p(\cdot|x;\theta) ;

• We conduct the SFT learning on the prefix spans of the reasoning trace r < t r_{<t} with NLL objective;

#### Structure Tuning

To avoid the catastrophic forgetting ( Muennighoff et al., 2025 ) of reasoning structures brought by prefix tuning, we adopt a simple yet effective multi-task learning approach, to jointly conduct the prefix tuning with conventional SFT process task. We use a probability ratio p % p\% to randomly split out a subset 𝒟 f ⊂ 𝒟 \mathcal{D}_{f}\subset\mathcal{D} to generate the full reasoning trace r r for each data pair ( x , y ) ∈ 𝒟 f (x,y)\in\mathcal{D}_{f} , without justifying the correctness of this reasoning trace r r or reject sampling process. Consequently, the data of the whole trace SFT is totally unsupervised, which also avoids the inference overhead of reject sampling. The rest of dataset 𝒟 p \mathcal{D}_{p} is then used for prefix decoding r < t r_{<t} . Moreover, to clearly demonstrate the base model that the learning process is solely based on prefix spans, we utilize a specialized task template, as shown in Figure 3 , for prefix tuning to enhance the model’s ability to learn effectively from prefix-specific patterns.

In summary, the overall objective is a combination of UPFT and unsupervised SFT by maximizing the following objective:

𝔼 x ∼ 𝒟 p r < t ∼ p ( ⋅ | x ; θ ) ​ [ log ⁡ p ⁡ ( r < t | x ; θ ) ] + 𝔼 x ∼ 𝒟 f r ∼ p ( ⋅ | x ; θ ) ​ [ log ⁡ p ⁡ ( r | x ; θ ) ] \displaystyle\mathbb{E}_{\begin{subarray}{c}x\sim\mathcal{D}_{p}\\ r_{<t}\sim p(\cdot|x;\theta)\end{subarray}}[\log p(r_{<t}|x;\theta)]+\mathbb{E}_{\begin{subarray}{c}x\sim\mathcal{D}_{f}\\ r\sim p(\cdot|x;\theta)\end{subarray}}[\log p(r|x;\theta)] (8) Notably, compared to conventional RFT objective, our method does not require any rejection sampling process denoted by 𝟙 ​ ( r ( k ) , y ) \mathbb{1}(r^{(k)},y) in Equation 1 . Consequently, our method is naturally suitable for learning from unproven questions with extreme data efficiency.

## 4 Experiment

In this experiment, we compare UPFT with traditional supervised methods such as SFT and RFT in both supervised and unsupervised sampling settings. We also evaluate its scalability by varying the self-training corpora and backbone models.

### 4.1 Experiments Setup

#### Backbone LLMs

For our experiments, we selected three representative open-source backbone models: the general-purpose Llama-3.1-8B-Instruct ( Dubey et al., 2024 ) , the math-specialized Qwen2.5-Math-7B-Instruct ( Yang et al., 2024 ) optimized for mathematical tasks, and the long-reasoning DeepSeek-R1-Distill-Qwen-7B ( Guo et al., 2025 ) , distilled from DeepSeek-R1.

#### Fine-Tuning

We used four datasets to generate self-training data for fine-tuning: 1. PRM (12K instances; Lightman et al., 2024 ): This dataset includes 4.5K MATH ( Hendrycks et al., 2021 ) test problems, which are drawn from the PRM800K training set.

2. OMI2 (600K instances; Toshniwal et al., 2024 ): This is a subset of 600K unique questions extracted from the OpenMathInstruct2 math-instruction tuning dataset.

3. LIMO (819 instances; Ye et al., 2025 ): A high-quality training dataset specifically focused on challenging problems.

4. U-Hard (100K questions): This dataset, introduced in this work, is designed to explore the potential of our method in an unsupervised setting.

U-Hard was curated through an extensive collection of questions from publicly available online sources. Adhering to the Omni-Math approach ( Gao et al., 2024 ) , we labeled the collected data by difficulty and subsequently filtered out lower-difficulty questions. This process resulted in a dataset composed exclusively of challenging questions, thereby ensuring U-Hard’s practical relevance to real-world scenarios.

To ensure a fair comparison, UPFT employs the same hyperparameters as conventional SFT when fine-tuning models, as listed in Table 1 . During the inference stage, we adopt a prompted zero-shot setup and use standard greedy decoding, wherein models are directed to answer each question using natural language instructions without any accompanying contextual demonstrations.

#### Benchmarks

We evaluated the performance of our method and the baseline methods on four widely used benchmarks: GSM8K ( Cobbe et al., 2021 ) , MATH500 ( Hendrycks et al., 2021 ) , AIME24 ( MAA Committees, ) , and GPQA Diamond ( Rein et al., 2023 ) .

#### Tuning Scenarios

We employed two experimental settings: • Unsupervised sampling : we took one sample per question without any posterior filtering.

• Supervised sampling : Following RFT ( Yuan et al., 2023 ) , we sampled the responses for each question 16 times and used the ground truth answers to select a correct response.

### 4.2 Ablation Study

In this section, we evaluated the impact of two hyperparameters of UPFT on the PRM-12K dataset using Llama-3.1-8B-Instruct and Qwen2.5-Math-7B-Instruct in the unsupervised setting and report the results on the MATH500 test set.

#### Impact of Prefix Length

Figure 4(a) illustrates how prefix length affects reasoning accuracy. Each model has its own optimal prefix length for peak performance: Llama-3.1-8B-Instruct achieves its highest average accuracy (52.0%) with an 8-token prefix, whereas Qwen2.5-Math-7B-Instruct remains stable over prefix lengths from 8 to 32 tokens. The math-specialized Qwen2.5-Math-7B-Instruct is less sensitive to prefix length, likely because it was trained on the high-quality large-scale math corpus, which reinforces its stability on the MATH500 test set. For subsequent experiments, we set Llama-3.1-8B-Instruct to 8 tokens and Qwen2.5-Math-7B-Instruct to 32 tokens. In the following experiments, we set the prefix length to 8 for Llama-3.1-8B-Instruct and 32 for Qwen2.5-Math-7B-Instruct. Because the responses from the long-reasoning DeepSeek-R1-Distill-Qwen-7B are generally more than four times longer than those of Qwen2.5-Math-7B-Instruct, we use a prefix length of 128 tokens for the former.

#### Impact of Structure Tuning Ratio

Figure 4(b) plots the effect of the structure tuning ratio ( p p ). The performance of both models generally improves with increasing p p , peaking at p = 10 % p=10\% . This result supports our hypothesis that even minimal structural supervision effectively guides models towards generating complete responses. However, exceeding this ratio leads to a performance decrease. Consequently, we set p p to 0.1 in UPFT for all datasets, except LIMO. For the smaller LIMO dataset, we increased p p to 0.3 for structure tuning.

### 4.3 Unsupervised Fine-Tuning

Table 2 presents the results for the unsupervised sampling setting, where the data is not filtered based on the correctness of the extracted answer. We compare these outcomes to those obtained using conventional SFT ( Ouyang et al., 2022 ) , which fine-tunes models on training data without any self-improvement or test-time verification.

#### UPFT demonstrates superior performance compared to SFT in unsupervised fine-tuning.

UPFT consistently outperforms conventional SFT across various self-training datasets and backbone models under the unsupervised sampling setting. For instance, using the U-Hard dataset with Qwen2.5-Math-7B-Instruct, UPFT achieves an average accuracy of 54.5% across all benchmarks, exceeding the 51.3% attained by conventional SFT. Likewise, with DeepSeek-R1-Distill-Qwen-7B and U-Hard, UPFT attains an average of 61.6%, whereas SFT achieves 56.4%. These results highlight UPFT’s effectiveness in leveraging unsupervised data to enhance reasoning, thereby reducing the reliance on labeled data. They also indicate the versatility and broad applicability of UPFT across different LLM architectures, supporting its claim of being a flexible, universal method for improving reasoning capabilities.

#### The benefits of UPFT are more pronounced on complex reasoning tasks.

The performance gains of UPFT over conventional SFT are especially evident on more challenging benchmarks such as AIME2024 and GPQA. For example, on AIME2024 with the U-Hard dataset, UPFT achieves 26.6% accuracy with Qwen2.5-Math-7B-Instruct, a notable improvement over the 16.7% achieved by conventional SFT. A similar trend is observed with DeepSeek-R1-Distill-Qwen-7B on AIME2024, where UPFT reaches 50.0% compared with SFT’s 36.7%, showing UPFT’s capability to improve reasoning on difficult problems, aligning with our claim of enhancing reasoning capabilities effectively and efficiently.

#### The U-Hard dataset maximizes UPFT’s potential through difficulty-focused curation.

Training with U-Hard yields the strongest performance improvements across models, particularly for complex benchmarks. Qwen2.5-Math-7B achieves 26.6% on AIME2024 with U-Hard - 10 points higher than with PRM data. This demonstrates that challenging questions provide richer signals for prefix-based self-improvement, as they demand more sophisticated initial reasoning setups. Using U-Hard, UPFT achieves the highest average accuracy with both Qwen2.5-Math-7B-Instruct and DeepSeek-R1-Distill-Qwen-7B, surpassing other datasets such as PRM-12K and OMI2-600K. These findings demonstrate that UPFT effectively leverages diverse and challenging questions to refine the model’s reasoning skills in an unsupervised manner, showcasing its data efficiency and versatility.

#### UPFT achieves dramatic efficiency gains through minimal token training.

UPFT reduces training sequence length by 82.6-94.7% compared to SFT across datasets, with U-Hard samples averaging 68.2 tokens vs. SFT’s 393.3. This directly translates to 6.3-16.7x faster training iterations and reduced memory consumption. Notably, DeepSeek-R1-Distill-Qwen-7B attains better performance with UPFT’s 561-token samples than SFT’s 3,440-token sequences, proving that strategic prefix selection preserves critical learning signals. These efficiency characteristics validate UPFT’s practical value for resource-constrained applications while maintaining or exceeding baseline performance.

### 4.4 Supervised Fine-Tuning

In the supervised sampling setting, we compare our approach with two SFT variants that require labeled data in Table 3 : • RFT ( Yuan et al., 2023 ) samples 16 candidate responses, then applies a label filter to identify a correct one.

• V-STaR ( Hosseini et al., 2024 ) produces 16 responses for each tuning instance, trains a verifier on them for one iteration, and uses it at test time to select the answer from 4 completions.

#### UPFT achieves competitive performance with supervised methods while requiring significantly fewer tokens in both sampling and tuning.

Across multiple model architectures, UPFT matches or exceeds the performance of supervised baselines such as RFT and V-STaR. On the Llama-3.1-8B-Instruct backbone, UPFT attains an average reasoning benchmark score of 38.3%, matching V-STaR (38.3%) and approaching RFT (38.5%). For Qwen2.5-Math-7B-Instruct, UPFT achieves identical average accuracy to RFT (52.6%), while using only 1.2% of the sampling tokens (0.6M vs. 51.7M). Most notably, UPFT enables DeepSeek-R1 to achieve 58.7% average accuracy, which is 1.5 points higher than RFT, while requiring 16× fewer tuning tokens than RFT. This underscores how prefix-based fine-tuning effectively captures critical reasoning patterns without incurring the computational overhead of sampling 16 responses per question, as required by supervised baselines. Moreover, UPFT maintains its strong performance even when ground-truth answers are available. By focusing on the shared reasoning prefixes, it does not sacrifice accuracy. These results reinforce our core claim: prefix-based training captures the essential reasoning signals available in full trajectories, validating our Prefix Self-Consistency hypothesis. This hypothesis posits that the essential signals for effective reasoning are largely contained within the initial prefixes of successful solution trajectories, allowing for efficient learning and high performance.

#### UPFT offers seamless integration with label verification for enhanced accuracy.

An additional benefit of UPFT is its inherent flexibility and compatibility with label verification techniques when such information is accessible. As demonstrated in the table’s last rows for each model backbone, when UPFT is enhanced with label filtering, it attains 58.8% average accuracy on DeepSeek-R1-Distill-Qwen-7B, surpassing all baselines while maintaining a 4.4× tuning token advantage over RFT. For Qwen2.5-Math-7B-Instruct, we match V-STaR’s 52.9% peak performance without requiring compute-intensive verifier training or best-of-N inference. This dual capability demonstrates UPFT’s unique adaptability - it functions as a standalone unsupervised learner while seamlessly integrating supervision when available, making it practical for real-world deployment across the supervision spectrum.

## 5 Related Work

Large language models (LLMs) have achieved significant progress in natural language processing tasks ( Longpre et al., 2023 ; Touvron et al., 2023 ) . However, enabling LLMs to perform complex reasoning remains challenging.

Recent work has demonstrated strong reasoning capabilities in LLMs ( Yang et al., 2024 ; Dubey et al., 2024 ; Guo et al., 2025 ; OpenAI, 2023 ) . One research direction varies the input query of LLMs to obtain consistent responses through prompting techniques, such as chain-of-thought ( Wei et al., 2022 ) and tree-of-thought ( Yao et al., 2023 ) . Another line of research leverages the verification of output trajectories for LLMs, for instance, ensembling and reranking inference paths, verification ( Cobbe et al., 2021 ; Uesato et al., 2022 ) and self-consistency ( Wang et al., 2023b ) . Trainable approaches like ReST ( Zelikman et al., 2022 ) , Rejection Sampling Fine-Tuning (RFT) ( Yuan et al., 2023 ) , and Self-Taught Reasoner (STaR) ( Zelikman et al., 2022 ) also exemplify this paradigm.