Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning — part 2/2

Cleaned-text code points [15661, 30722). Bibliography omitted here; source.md includes it.

Multi-task datasets. We use four high-quality datasets to assess the efficacy of our approach within multi-task fine-tuning scenarios: Alpaca ( Taori et al., 2023 ) , Dolly ( Conover et al., 2023 ) and LIMA ( Zhou et al., 2023 ) . The Alpaca dataset encompasses a variety of tasks, including arithmetic, coding, and question-answering. It was generated using the Self-Instruct method Wang et al. (2022) via the text-davinci-003 model. The Dolly dataset is composed of seven distinct tasks, such as open question & answer, information extraction, and summarization. The LIMA dataset covers a broad range of topics and was curated from multiple sources. The OpenHermes dataset consists of primarily GPT-4 generated data from a variety of public datasets, with filtering to remove refusals.

Safety evaluation. We utilize the harmful behavior instructions from the Advbench dataset Zou et al. (2023) for evaluation, assessing the safety of models’ outputs through keyword matching following Qi et al. (2024) . We define the proportion of safe responses as Raw Safe Rate . Additionally, we simulate jailbreaking attempts by appending adversarial suffixes to instructions as illustrated in Zou et al. (2023) . The safe rate under this condition is referred to as Jailbreak Safe Rate .

Helpfulness evaluation. We employ AlpacaEval Li et al. (2023) to evaluate the helpfulness of various models. This tool includes a dataset and associated evaluation metrics that facilitate the comparison of generated outputs with the responses from Text-Davinci-003, across a diverse set of 805 detailed instructions sourced from multiple datasets. We report the win rate, which is the proportion of instances where the responses are favored over those produced by Text-Davinci-003, as judged by GPT-4.

Knowledge evaluation. LMs’ general knowledge was assessed through evaluations using benchmarks from the OpenLLM Leaderboard, specifically MMLU Hendrycks et al. (2021) , TruthfulQA Lin et al. (2021) , ARC Clark et al. (2018) , HellaSwag Zellers et al. (2019) , and Winogrande Sakaguchi et al. (2021) . These datasets provide a measure of the models’ factual and commonsense knowledge spanning a variety of domains.

### 4.3 SDFT Achieves Better Results on Downstream Tasks

Table 1 presents the results of fine-tuning on three downstream tasks. The results indicate that while vanilla fine-tuning can enhance the model’s efficacy on target tasks, it also leads to a significant decline in performance on other tasks. For example, as depicted in the table’s first row, fine-tuning with the OpenFunctions dataset results in a diminished coding capability of the model, decreasing from 13.41 13.41 to 9.76 9.76 . A similar decline is observed in mathematical reasoning abilities, where accuracy on the GSM8K dataset drops from 29.42 29.42 to 21.53 21.53 .

Furthermore, the proposed SDFT can effectively mitigate this performance degradation. In the cited instance, the model retains its mathematical reasoning proficiency, achieving an accuracy of 29.11 29.11 , closely aligned with the seed model’s performance ( 29.42 29.42 ). For coding performance evaluated on HumanEval, there is a marginal improvement, with the performance rising to 15.24 15.24 from the seed model’s 13.41 13.41 .When focusing on the target task, SDFT also outperforms vanilla fine-tuning, delivering an accuracy of 36.61 36.61 compared to 34.82 34.82 .

### 4.4 SDFT Preserves Alignment

Fine-tuning on the majority of datasets has been demonstrated to lead to a significant decrease in both safety alignment and general helpfulness, as highlighted by the findings in Table 2 . For instance, following fine-tuning on the GSM8K dataset, the safe rate decreases from 99.81 99.81 to 82.12 82.12 , the jailbreak safe rate drops from 88.85 88.85 to 54.81 54.81 , and the win rate on AlpacaEval diminishes from 66.04 66.04 to 23.38 23.38 . In contrast, our proposed SDFT approach effectively mitigates this decline, improving the raw safe rate and jailbreak safe rate by 5 5 and 11 11 , respectively. Notably, there is a slight increase in the win rate compared to the seed model, with a score of 66.73 66.73 versus 66.03 66.03 .

Table 3 presents evaluation results after fine-tuning on instruction following datasets that contain multiple tasks. As the target tasks of these datasets are unspecified, we focus our evaluation on safety and general helpfulness after fine-tuning. In line with the patterns noted in Table 2 , fine-tuning on Alpaca, Dolly and LIMA typically leads to a marked reduction in both safety and helpfulness metrics. We observe a pronounced decline in all three metrics, with each declining by roughly 20 20 . In contrast, our proposed SDFT method effectively mitigates this reduction, limiting the decrease to under 10 10 . Similarly, vanilla fine-tuning on the OpenHermes Teknium (2023) dataset results in diminished safety alignment. In contrast, SDFT effectively mitigates this degradation, enhancing the jailbreak safe rate from 61.54 61.54 to 87.50 87.50 .

### 4.5 General Knowledge Remains Intact

Figure 4 presents results on general knowledge. Although vanilla fine-tuning compromises downstream performance and alignment, models’ capabilities in general knowledge are relatively unaffected. For instance, after fine-tuning on the OpenFunctions dataset, the disparity in performance between fine-tuned model and seed LM is less than 1 1 . This is also observed after fine-tuning with SDFT.

## 5 Analysis

In this section, we conduct a detailed analysis to understand the impact of distribution shift on catastrophic forgetting. In addition to the evaluation metrics outlined in Section 4 , we incorporate four supplementary metrics to assess the degree of distribution shift. We utilize both the seed model and fine-tuned models to generate responses on the Advbench Zou et al. (2023) dataset and engage in a comparative analysis of these responses.

In particular, we calculate the BLEU-4 and ROUGE-L scores for the fine-tuned models, using the outputs from the seed model as references to evaluate the extent of distribution shift. We also utilize Sentence-BERT Reimers and Gurevych (2019) to derive sentence embeddings and use the cosine similarity between these embeddings following Zhang et al. (2023) . Lastly, we quantify the extent of parameter shift by comparing the updated parameters with those of the seed model, considering their distance as a measure of the parameter shift magnitude. The lower the BLEU-4, ROUGE-L, and embedding similarity scores, the greater the distribution shift. Conversely, the parameter shift is directly proportional to the norm of the parameter changes.

### 5.1 Distribution Shift Correlates with Catastrophic Forgetting

We induce varying degrees of distribution shift too investigate its impact through two approaches: (1) By sampling a diverse quantity of examples for fine-tuning, where an increased number of data points for fine-tuning corresponds to a greater distribution shift. (2) By mixing vanilla fine-tuning with SDFT, which involves substituting distilled samples with original ones. We define mix ratio to represent the proportion of distilled samples employed. A mix ratio of 1 1 signifies exclusive use of our SDFT and 0 0 denotes vanilla fine-tuning.

Figures 6 and 6 illustrate the results with varying sample sizes. As the sample size grows, we observe a notable decrease in the BLEU-4, ROUGE-L, and embedding similarity scores, along with an elevation in parameter shift magnitude. This trend implies a heightened degree of distribution shift. Consequently, there is an observable decline in model performance on benchmarks such as GSM8K, MultiArith, Advbench, and AlpacaEval, suggesting intensified catastrophic forgetting.

In a similar vein, Figures 8 and 8 present results corresponding to an ascending mix ratio. As this ratio increases, there is an upward trend in the BLEU-4, ROUGE-L, and embedding similarity scores, whereas the scale of parameter shift diminishes, denoting a mitigation in distribution shift. Accordingly, benchmark performance exhibits improvement across the board, signaling a reduction in the severity of catastrophic forgetting.

Figure 9 illustrates the similarity distribution obtained through both vanilla fine-tuning and our SDFT. Notably, with SDFT model has higher similarity between the fine-tuned model and the seed model, signifying reduced distribution shift.

### 5.2 Robustness among Distillation Templates

We have constructed two templates to investigate the robustness of SDFT. The template illustrated in Figure 3 is labeled “Using”, where the phrase “Using the reference answer as a guide” is replaced by “Refer to the reference answer”, with the latter template being termed “Refer”. Results after fine-tuning with both templates are detailed in Table 4 . The performance across diverse benchmarks remains consistent across the templates, demonstrating the robustness of SDFT.

### 5.3 Efficacy of SDFT Across Model Scales and Architectures

The SDFT approach is not constrained by any specific fine-tuning technique (such as LoRA) or model architecture, enabling its application across both comprehensive fine-tuning processes and other model architectures. To substantiate this claim, we conducted supplementary experiments that included full fine-tuning on Llama-2-7b-chat and LoRA fine-tuning on Llama-2-7b-chat. Additionally, we explored the fine-tuning of the recently unveiled SOTA model, Llama3 Meta AI (2024) on the OpenFunctions dataset. The results in Table 5 reveal that in all scenarios, SDFT not only consistently outperforms vanilla fine-tuning in the target task but also reduces forgetting across all other tasks, demonstrating its effectiveness.

## 6 Conclusions and Limitations

In this paper, we perform a systematic evaluation of catastrophic forgetting during the fine-tuning of language models for downstream tasks. Our findings indicate that the distribution shift during fine-tuning can lead to performance degradation in general task capabilities, as well as models’ safety alignment and helpfulness. To enhance performance on target task while maintaining LMs’ broad capabilities, we propose a plug-and-play strategy, SDFT, to reduce distribution shift and mitigate catastrophic forgetting. Extensive experiments show that SDFT effectively diminishes forgetting and delivers comparable or superior performance to vanilla fine-tuning on targeted tasks.

Our study is subject to certain limitations. Owing to constraints in computational resources, most of our experiments are based on the Llama-2-7b-chat model with LoRA. Further investigations involving larger models and full fine-tuning remain to be explored. Furthermore, our safety evaluations are limited to the Advbench dataset and fixed adversarial suffixes, leaving the robustness against other jailbreaking strategies for future work.

## Ethics Statement

Our proposed method SDFT effectively mitigates the issue of catastrophic forgetting during the fine-tuning of language models, including the degradation of safety alignment. Therefore, this process does not entail additional risks.

We utilize a variety of open-source English datasets for training, including Alpaca, Dolly, LIMA, OpenHermes, GSM8K, OpenFunctions, and MagiCoder. The Llama-2-chat model serves as our seed model for training. We acknowledge that there may be inherent biases present within these datasets and the model.

## Acknowledgements

This paper is supported by the National Science Foundation of China (62132017, 62302435), Zhejiang Provincial Natural Science Foundation of China (LD24F020011, LQ24F020006) and “Pioneer and Leading Goose” R&D Program of Zhejiang (2024C01167).

## Appendix A Experimental Details

Throughout most experiments, we applied fine-tuning to Llama-2-7b-chat with the Low-Rank Adaptation (LoRA) technique ( Hu et al., 2022 ) . The query and value matrices of the LoRA were tuned with a rank of r = 8 r=8 . We adhered to the default configuration settings of Llama2. The learning rate was initiated at 1 × 10 − 4 1\times 10^{-4} and progressively decayed to zero following a cosine annealing schedule. and the batch size was set to 8 8 .

We randomly sampled a subset of 2,000 examples and conducted fine-tuning for 2 epochs for the Alpaca Taori et al. (2023) , Dolly Conover et al. (2023) , and MagiCoder Wei et al. (2023) datasets. We sampled 20,000 examples for the OpenHermes dataset and train 2 epochs. For GSM8K Cobbe et al. (2021) , LIMA Zhou et al. (2023) and OpenFunctions Patil et al. (2023) datasets, we fine-tune on the entire train set. We train LIMA for 2 epochs and the other two datasets for 5 epochs.

To assess the general helpfulness of models, we employ the AlpacaEval framework † † https://github.com/tatsu-lab/alpaca_eval , with GPT-4 serving as the evaluator. The evaluations of benchmarks in OpenLLM leaderboard are implemented via the lm-evaluation-harness † † https://github.com/EleutherAI/lm-evaluation-harness . Furthermore, we evaluate the coding capabilities using the HumanEval dataset Chen et al. (2021) , leveraging the bigcode-evaluation-harness project † † https://github.com/bigcode-project/bigcode-evaluation-harness .

## Appendix B Templates and Examples

This section provides templates used in our experiments and some illustrative examples of distillation on each dataset.

In most of our experiments, we use the standard alpaca Taori et al. (2023) template for both fine-tuning and prediction, as presented in Figure 10 .

To enhance the reasoning abilities, we slightly modify the standard Alpaca template for datasets regarding reasoning, namely GSM8K and MultiArith. The template used for training and distillation are presented in Figure 11 and Figure 12 respectively.

To make the extraction of final answer easier for mathematical reasoning datasets, we specify the format of final answer during evaluation. The template used for evaluation is presented in Figure 13 .

Figures 14 , 15 , 16 , 17 , 18 present examples of distilled data point on each dataset.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .