Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs — part 2/7
Cleaned-text code points [15499, 31021). Bibliography omitted here; source.md includes it.
The kernel function k ( y , y ′ ) k(y,y^{\prime}) must satisfy k ( y , y ′ ) ∈ [ 0 , 1 ] k(y,y^{\prime})\in[0,1] , k ( y , y ) = 1 k(y,y)=1 , and C p , k ( x ) > 0 C_{p,k}(x)>0 . A kernel representing exact collision is defined as k ( y , y ′ ) = 𝟏 { y = y ′ } k(y,y^{\prime})=\mathbf{1}\{y=y^{\prime}\} . Finally, task-specific kernels may measure embedding cosine similarity, shared cluster membership, normalized Zhang–Shasha ( Zhang and Shasha, 1989 ) similarity between code abstract syntax trees (ASTs), or similarity of survey responses, with all values shifted to the range [ 0 , 1 ] [0,1] .
### 3.1 Sign and magnitude of the difference in collision probabilities
The collision decomposition. For a fixed context h h , consider fitted conditional probability q θ ^ , h = q θ ^ ( ⋅ ∣ h ) q_{\hat{\theta},h}=q_{\hat{\theta}}(\cdot\mid h) and true conditional probability p h p_{h} . Applying Eq. ( 4 ) to next-token outputs at context h h , with the exact-match kernel k ( a , b ) = 𝟏 { a = b } k(a,b)=\mathbf{1}\{a=b\} , gives C h ( q ) := ∑ a ∈ V q ( a ∣ h ) 2 = c ( q ( ⋅ ∣ h ) ) C_{h}(q):=\sum_{a\in V}q(a\mid h)^{2}=c(q(\cdot\mid h)) . Thus C h ( q θ ^ ) = ‖ q θ ^ , h ‖ 2 2 C_{h}(q_{\hat{\theta}})=\|q_{\hat{\theta},h}\|_{2}^{2} and C h ( p ) = ‖ p h ‖ 2 2 C_{h}(p)=\|p_{h}\|_{2}^{2} . We call the difference 𝔼 [ C h ( q θ ^ ) ] − C h ( p ) \mathbb{E}[C_{h}(q_{\hat{\theta}})]-C_{h}(p) the expected collision gap: it is positive when the model collides more often than the target (mode collapse) and negative when the model is more diverse. For an arbitrary random vector X X , we use the standard variance decomposition identity: 𝔼 ‖ X ‖ 2 2 = ‖ 𝔼 X ‖ 2 2 + Tr Cov ( X ) \mathbb{E}\|X\|_{2}^{2}=\|\mathbb{E}X\|_{2}^{2}+\Tr\Cov(X) . Here Tr Cov ( X ) \Tr\Cov(X) is the sum of the component variances. Taking X = q θ ^ , h X=q_{\hat{\theta},h} , 𝔼 [ X ] \mathbb{E}[X] is the mean fitted distribution across independent fits. Next, define its bias relative to the target as b h := 𝔼 [ q θ ^ , h ] − p h . b_{h}:=\mathbb{E}[q_{\hat{\theta},h}]-p_{h}. (5) Thus 𝔼 [ q θ ^ , h ] = p h + b h \mathbb{E}[q_{\hat{\theta},h}]=p_{h}+b_{h} . Substituting this into the variance decomposition identity, expanding ‖ p h + b h ‖ 2 2 \|p_{h}+b_{h}\|_{2}^{2} , and dividing by C h ( p ) C_{h}(p) gives 𝔼 [ R ] = 1 + 1 C h ( p ) [ Tr Cov ( q θ ^ , h ) ⏟ variance ≥ 0 + ‖ b h ‖ 2 2 ⏟ squared bias ≥ 0 + 2 p h ⊤ b h ⏟ target–bias alignment ] . \boxed{\textstyle\;\mathbb{E}[R]=1+\frac{1}{C_{h}(p)}\Big[\underbrace{\Tr\Cov(q_{\hat{\theta},h})}_{\text{variance}\,\geq 0}\;+\;\underbrace{\|b_{h}\|_{2}^{2}}_{\text{squared bias}\,\geq 0}\;+\;\underbrace{2\,p_{h}^{\top}b_{h}}_{\text{target--bias alignment}}\Big].\;} (6) The covariance trace and squared bias are nonnegative, and thus contribute to increasing the expected collision ratio. The target–bias alignment term 2 p h ⊤ b h 2p_{h}^{\top}b_{h} (referred to as the “cross term” in what follows) can be either positive or negative and may dominate these terms, so finite samples alone do not imply mode collapse. Thus the collision ratio R R can be above 1 1 (mode collapse) or below 1 1 (over-dispersion). If b h = 0 b_{h}=0 , the expected gap is Tr Cov ( q θ ^ , h ) ≥ 0 \Tr\Cov(q_{\hat{\theta},h})\geq 0 : the model is calibrated or under-dispersed in expectation, with under-dispersion whenever this variance is positive.
The sign of the cross term. Let q 0 , h q_{0,h} denote the starting distribution at context h h . For a pre-trained LLM, q 0 , h q_{0,h} is its output distribution given context h h before any fine-tuning. At this starting point, using the definitions of bias ( 5 ) and collision ( 1 ) we get 2 p h ⊤ b h = 2 p h ⊤ ( q 0 , h − p h ) = 2 ( p h ⊤ q 0 , h − C ( p h ) ) . 2\,p_{h}^{\top}b_{h}=2\,p_{h}^{\top}(q_{0,h}-p_{h})\;=\;2\big(p_{h}^{\top}q_{0,h}-C(p_{h})\big). (7) The alignment term ( 7 ) is positive when the starting distribution overlaps the target’s modes more than the target overlaps itself. Moreover, the alignment term can determine whether the expected collision gap is positive or negative.
Using this setup, we can prove the following result bounding the collision gap between distributions p p and q q :
###### Theorem 1 (Kernel-collision stability) .
Consider the similarity kernel k : V ∗ × V ∗ → [ 0 , 1 ] k:V^{*}\times V^{*}\to[0,1] , where V ∗ V^{*} is the set of finite token sequences. Write the collision metric C k ( π ) = ∑ a , b π ( a ) π ( b ) k ( a , b ) C_{k}(\pi)=\sum_{a,b}\pi(a)\pi(b)k(a,b) , D k ( π ) = 1 / C k ( π ) D_{k}(\pi)=1/C_{k}(\pi) , where π \pi is a probability distribution on V ∗ V^{*} and D k ( π ) D_{k}(\pi) is its kernel effective diversity. Write TV \TV for total variation distance, ⊗ \otimes for the product measure operator, and KL ( p ∥ q ) = 𝔼 a ∼ p [ − log q ( a ) ] − 𝔼 a ∼ p [ − log p ( a ) ] \mathrm{KL}(p\|q)=\mathbb{E}_{a\sim p}[-\log q(a)]-\mathbb{E}_{a\sim p}[-\log p(a)] for excess population cross-entropy, using natural logarithms. For all distributions p , q p,q on V ∗ V^{*} , | C k ( p ) − C k ( q ) | ≤ TV ( p ⊗ p , q ⊗ q ) ≤ KL ( p ∥ q ) . \boxed{\;|C_{k}(p)-C_{k}(q)|\;\leq\;\TV(p\otimes p,\,q\otimes q)\;\leq\;\sqrt{\mathrm{KL}(p\,\|\,q)}\;.} (8) Consequently, if KL ( p ∥ q ) ≤ ε \mathrm{KL}(p\,\|\,q)\leq\varepsilon then max { 0 , C k ( p ) − ε } ≤ C k ( q ) ≤ min { 1 , C k ( p ) + ε } \max\{0,C_{k}(p)-\sqrt{\varepsilon}\}\leq C_{k}(q)\leq\min\{1,C_{k}(p)+\sqrt{\varepsilon}\} , and the kernel effective diversity obeys D k ( q ) ≥ 1 C k ( p ) + ε = D k ( p ) 1 + D k ( p ) ε . \textstyle D_{k}(q)\;\geq\;\frac{1}{C_{k}(p)+\sqrt{\varepsilon}}\;=\;\frac{D_{k}(p)}{1+D_{k}(p)\sqrt{\varepsilon}}\;. (9)
The proof, via total variation on product measures and Pinsker’s inequality, is in Appendix A , together with a sharper collision-specific bound (Proposition 1 ).
## 4 Experimental results
To test the bias-variance decomposition and convergence results in the above theory, we now run three experiments. The first uses synthetic languages with known ground truth for each decomposition term, allowing direct measurement of every term. The second and third use human data—survey responses from the GSS ( Davern et al., 2025 ) , WVS ( Haerpfer et al., 2022 ) , and ANES ( American National Election Studies, 2022 ) , and human solutions to programming problems from Project CodeNet ( Puri et al., 2021 ) . We report the collision ratio R R defined in Eq. ( 3 ); for example, R = 2 R=2 means the model offers half as many effective choices as the data. Experiment 1 additionally decomposes the expected normalized collision gap 𝔼 [ R ] − 1 \mathbb{E}[R]-1 into the three terms of Eq. ( 6 ).
### 4.1 Experiment 1: synthetic languages with known target distribution
We sample data from a synthetic language, fit a series of small GPT models on that data, and compare the diversity of the fitted models against the diversity of the language itself. Unlike experiments using human-generated data, in this setting, target probabilities are known. That allows us to repeat model fits to estimate variance, squared-bias, and target–bias alignment terms in Eq. ( 6 ). As we have established in the previous section, the latter term determines whether LLMs mode-collapses or is over-diverse. In order to control this term, we initialize the GPTs using two settings: (a) pre-training from random initialization, and (b) fine-tuning a model pre-trained to favor a small set of tokens.
In setting (a), the language has a vocabulary of 1024 1024 tokens, and each next token depends only on the preceding 16 16 tokens. The target probabilities are known exactly at every context, and a small GPT can represent the generating distribution exactly, so model capacity does not limit the fit. In setting (b), the language combines fixed token frequencies with adjustments based on the preceding 16 16 tokens. Appendix H.1 gives the language construction and sampling details. The two settings differ in the model’s starting next-token distribution, model size, and target language. They are not intended as a controlled ablation of the starting distribution alone. Figure 2 shows pre-training from random initialization in panel (a) and fine-tuning from the pre-trained model in panel (b).
Decomposing the expected collision gap as in Section 3.1 requires independent fits: a single fitted model does not show the variance across runs. We therefore fit 100 100 models at each of 100 100 log-spaced sample sizes of N N sampled sequences, giving 10,000 10{,}000 GPTs per language. Each seed draws its own dataset, initialization, and minibatch stream, ensuring that the variance term Tr Cov ( q θ ^ , h ) \Tr\Cov(q_{\hat{\theta},h}) in Eq. ( 6 ) reflects total across-run fluctuation. Details of model sizes, data construction, and fitting configurations are provided in Appendix H.1 .
We consider two settings that sit on opposite sides of the initial-alignment condition in ( 7 ). In setting (a), a GPT starts from random initialization. Its initial next-token distribution q 0 , h q_{0,h} at context h h has high entropy; we call this the diffuse prior . Its probabilities depend on the context, but the network is randomly initialized, so it does not favor or suppress particular tokens based on the target distribution p h p_{h} . In setting (b), we pre-train a GPT to concentrate its probability on a fixed set of common target-language tokens. We call this concentrated starting distribution the mode-aligned prior . Appendix H.1 describes the construction of both priors. This mode-aligned prior is arguably not a realistic model of pre-training from scratch, however, it could showcase the behavior of an existing LLM pre-trained on data from a distribution similar but not identical to the target.
Under the diffuse prior (Figure 2 (a)), the normalized cross term 2 p h ⊤ b h / C ( p h ) 2p_{h}^{\top}b_{h}/C(p_{h}) , averaged over contexts, is negative at every sample size, and is largest in magnitude at small N N . The normalized variance and squared-bias terms stay positive. The sign of 𝔼 [ R ] − 1 \mathbb{E}[R]-1 depends on N N : at the smallest N N the cross term dominates and the measured 𝔼 [ R ] − 1 \mathbb{E}[R]-1 is negative (over-dispersion); however, over most models, the two nonnegative terms dominate and 𝔼 [ R ] − 1 \mathbb{E}[R]-1 is positive (mode-collapse).
Under the mode-aligned prior, the cross term increases the expected collision ratio at intermediate sample sizes (Figure 2 (b)). Even where positive, it accounts for at most 31 % 31\% of the measured 𝔼 [ R ] − 1 \mathbb{E}[R]-1 , so variance and squared bias still account for most of the expected normalized collision gap. At the smallest N N , however, the cross term is negative and mode collapse is driven by the two nonnegative terms.
To relate these outcomes to the starting distributions, Table 1 in Appendix B checks the initial-alignment condition in Eq. ( 7 ) using the mean initial distribution q ¯ 0 , h := 𝔼 [ q 0 , h ] \bar{q}_{0,h}:=\mathbb{E}[q_{0,h}] , estimated across the 100 100 seeds. The table reports averages over contexts after the indicated normalization. The normalized cross term is initially negative under the diffuse prior and positive under the mode-aligned prior, but fine-tuning does not preserve these signs at every N N . In the next two experiments (Sections 4.2 and 4.3 ), we use human data to test whether fine-tuning moves R R toward 1 1 (as Theorem 1 predicts when excess population cross-entropy approaches zero).
### 4.2 Experiment 2: sociological surveys (GSS, WVS, and ANES)
In this experiment, we LoRA fine-tune four LLMs on human responses to three large opinion surveys, to test whether supervised fine-tuning moves a model’s answer diversity toward the diversity of the population it is asked to imitate through fine-tuning.
We consider three social surveys in this section. First, the General Social Survey (GSS) ( Davern et al., 2025 ) , a nationally representative survey of U.S. adults since 1972. We use the cumulative dataset spanning 1972–2024, including data from 74,485 respondents with complete demographic information. Second, we use a random subset of one-third of the countries and one-third of the questions from wave 7 of the World Values Survey (WVS) ( Haerpfer et al., 2022 ) , a cross-national survey of values (the sub-samples are used in the interest of saving compute). Third, we use data from the American National Election Studies (ANES) ( American National Election Studies, 2022 ) , a long-running U.S. political survey.
Respondents are partitioned into demographic groups based on characteristics such as country, age, sex, race/urban-rural status, region, and education. For each fixed question and demographic group X X , let Y Y denote the answer and let m a m_{a} of the m m respondents choose answer a a . We estimate human collision probability as the proportion of pairs of distinct respondents who give the same answer: C ^ U := ∑ a m a ( m a − 1 ) / [ m ( m − 1 ) ] \hat{C}_{\mathrm{U}}:=\sum_{a}m_{a}(m_{a}-1)/[m(m-1)] . The prompt presented to the model includes the demographic group as a persona, the survey question, and its answer options, and asks the model to select a single answer option. Applying softmax over the answer-option logits gives q ( a ∣ X ) q(a\mid X) , the model’s probability of answer a a for the fixed question and demographic group X X . The model’s collision probability is C ( q ) := ∑ a q ( a ∣ X ) 2 C(q):=\sum_{a}q(a\mid X)^{2} . We report R U := C ( q ) / C ^ U R_{\mathrm{U}}:=C(q)/\hat{C}_{\mathrm{U}} , which estimates the collision ratio in Eq. ( 3 ) using C ^ U \hat{C}_{\mathrm{U}} as the human collision estimate.
We repeat this experiment using 376 376 GSS questions ( Davern et al., 2025 ) , 53 53 WVS questions ( Haerpfer et al., 2022 ) , and 44 44 ANES questions ( American National Election Studies, 2022 ) —and four models— gemma-2-2b-it ( Gemma Team, 2024 ) , Qwen3.5-2B and Qwen3.5-4B ( Qwen Team, 2026 ) , and gemma-3-4b-it ( Gemma Team, 2025 ) —using the split defined in Table 3 of Appendix H.2 . For each survey, model, and fine-tuning set of n n respondent–question examples, we evaluate R U R_{\mathrm{U}} for every question–group pair with at least 60 60 respondents. We include the zero-shot model as n = 0 n=0 . This gives a fixed set of 18,270 18{,}270 pairs on the GSS, 8,015 8{,}015 on the WVS, and 8,211 8{,}211 on the ANES. Figure 3 plots, at each n n , median pair-level ratios for early-stopped adapters, with shaded bands for the 25 25 – 75 % 75\% and 5 5 – 95 % 95\% spread across pairs. Movement of the median ratio toward 1 1 means that model answer diversity is approaching the human level for a typical question–group pair.
Three of the four base models mode-collapse across all surveys; the fourth starts closest to median calibration. Zero-shot, gemma-2-2b-it , Qwen3.5-4B , and gemma-3-4b-it have median collision ratios R U R_{\mathrm{U}} between 1.61 1.61 and 2.92 2.92 , corresponding to roughly one-third to two-thirds of the human effective diversity. The collapse is most pronounced on the WVS—the sparsest and only cross-national survey. The three collapsed models all have higher median collision ratios on the WVS than on the U.S. surveys (Table 2 in Appendix C ). The severity of mode collapse therefore varies with both the model and the survey population.