Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs

arxiv:2609.16454

Original source · Paper JSON

Post-training · 2026-09-15

Cites (0) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7

Contents

• Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs
• Abstract
• 1 Introduction
• 2 Related work
• 3 Theory
• 3.1 Sign and magnitude of the difference in collision probabilities
• Theorem 1 (Kernel-collision stability) .
• 4 Experimental results
• 4.1 Experiment 1: synthetic languages with known target distribution
• 4.2 Experiment 2: sociological surveys (GSS, WVS, and ANES)
• 4.3 Experiment 3: code generation using CodeNet dataset
• 5 Conclusion
• AI use statement
• Appendix A Proofs and alignment conditions
• A.1 Kernel-collision stability
• A.2 A sufficient condition for nonnegative alignment
• Assumption 1 (Target–bias comonotonicity) .
• A.3 Relative collision bounds and population cross-entropy
• Proposition 1 (Relative bound on the collision ratio) .
• Appendix B Synthetic alignment before and after fitting
• Appendix C Complete survey results
• Appendix D Mitigation methods and finite-sample estimation
• Appendix E Sequence-level bounds and response length
• Appendix F KL-regularized preference optimization
• Appendix G Empirical check of the bound
• Appendix H Experimental details
• H.1 Synthetic languages
• H.2 Surveys
• H.3 CodeNet
• Appendix I Tabular intuition for finite-sample collision
• I.1 Expected collision inflation
• Example 1 (Why any finite sample inflates collision (Jensen)) .
• Remark 1 (This is tabular, not neural) .
• I.2 Amplification over sequences
• I.3 Entropy analogue
• Appendix J Limits of temperature scaling
• Proposition 2 (Global temperature monotonically increases order-2 diversity) .
• J.1 The temperature-reachable diversity interval
• Lemma 1 (Temperature-reachable diversity interval) .
• Theorem 2 (Temperature only redistributes existing support) .
• Lemma 2 (Temperature scaling is an order-preserving one-parameter family) .
• J.2 Effective support
• Proposition 3 (Effective absence of valid missing mass under sampling) .
• Example 2 (A two-peaked target temperature cannot reach) .
• Instructions for reporting errors