Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs — part 3/7
Cleaned-text code points [31021, 45785). Bibliography omitted here; source.md includes it.
At the smallest n n , the three collapsed models stay near their zero-shot collision ratio. Qwen3.5-2B , the model closest to median calibration at n = 0 n=0 , is instead driven above its zero-shot collision ratio during early fine-tuning on all three surveys—to 2.22 2.22 on the GSS, leaving less than half its starting effective diversity—before recovering (Table 2 in Appendix C ). Beyond roughly 3,000 3{,}000 fine-tuning examples, all twelve survey–model combinations maintain median collision ratios within 10 % 10\% of the calibrated value of 1 1 . Qwen3.5-2B ’s early increase in collision ratio moves it away from 1 1 , showing that small fine-tuning datasets can worsen diversity calibration. The later approach to 1 1 across all models and surveys shows that larger human datasets improve calibration.
### 4.3 Experiment 3: code generation using CodeNet dataset
In contrast to the previous section, exact sequence collisions are too rare to estimate for complete programs. So on CodeNet ( Puri et al., 2021 ) we compare model generations with accepted Python submissions using structural similarity. We fine-tune the same four LLMs as in Experiment 2 on accepted Python submissions. Before fine-tuning and at each fine-tuning-set size, each model generates 32 32 programs per problem. We map each program y y to an ordered labeled canonical Python abstract syntax tree T ( y ) T(y) , removing comments and formatting and canonicalizing identifiers and literals.
Let d ZS d_{\mathrm{ZS}} be the Zhang–Shasha tree edit distance ( Zhang and Shasha, 1989 ) : the minimum number of node insertions, deletions, and relabelings needed to transform one tree into the other, with each operation costing one. We define k ZS ( y , y ′ ) := 1 − d ZS ( T ( y ) , T ( y ′ ) ) | T ( y ) | + | T ( y ′ ) | . \textstyle k_{\mathrm{ZS}}(y,y^{\prime}):=1-\frac{d_{\mathrm{ZS}}\!\left(T(y),T(y^{\prime})\right)}{|T(y)|+|T(y^{\prime})|}. (10) where | T ( y ) | |T(y)| is the number of nodes in T ( y ) T(y) . This similarity kernel k ZS ( y , y ′ ) k_{\mathrm{ZS}}(y,y^{\prime}) lies in [ 0 , 1 ] [0,1] and equals 1 1 for identical trees. For each problem, p p denotes the distribution of accepted human submissions and q q the model’s output distribution. The kernel collisions C ZS ( p ) C_{\mathrm{ZS}}(p) and C ZS ( q ) C_{\mathrm{ZS}}(q) in Eq. ( 4 ) measure expected similarity within each source using Eq. ( 10 ), conditional on both programs parsing as Python. For each source separately, we estimate its collision by averaging k ZS ( y , y ′ ) k_{\mathrm{ZS}}(y,y^{\prime}) over up to 32 32 distinct unordered pairs of parseable programs sampled without replacement. Figure 4 reports the median finite ratio C ZS ( q ) / C ZS ( p ) C_{\mathrm{ZS}}(q)/C_{\mathrm{ZS}}(p) over a fixed panel of 2,647 2{,}647 candidate problems. The base models’ median collision ratios lie on both sides of the accepted-program target: both Qwen models are over-diverse, while both Gemma models are under-diverse. After fine-tuning on the full dataset, each model’s median uses ratios from all candidate problems. To check whether changes in which problems enter the median explain the shift toward calibration, we compare each model before and after fine-tuning on the full dataset using only problems included at both stages (Appendix H.3 ). All four medians still move closer to 1 1 , so the improvement persists when the set of problems is held fixed. Appendix H.3 gives the complete construction, prompt, fine-tuning configuration, and program examples.
## 5 Conclusion
LLMs can exhibit either mode collapse or over-dispersion relative to a target population, even though recent literature has mostly emphasized mode-collapse. Finite-sample fitting has no built-in direction: variance and squared bias contribute to increasing the expected collision ratio, while target–bias alignment can offset them. The absolute difference between collision probabilities of the model and the target population is bounded by the square root of KL divergence, so approaching the target distribution forces diversity calibration. Across four models, our social survey and coding experiments show that increasing fine-tuning data moves diversity toward the target from both directions. These results support evaluating diversity relative to a specified target alongside standard performance benchmarks, and motivate fine-tuning methods that use the diversity-calibrating properties of SFT. Obtaining sufficient coverage of the target distribution remains a challenge when data are limited.
## AI use statement
In this work, we used generative AI tools to: formulate mathematical claims, provide critical ingredients for proving mathematical claims, assist in the writing of proofs, support qualitative and thematic data analysis, implement methods, polish writing. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.
## Appendix A Proofs and alignment conditions
### A.1 Kernel-collision stability
Kernel collision is an expectation of k k under product measures P = p ⊗ p P=p\otimes p and Q = q ⊗ q Q=q\otimes q . Because k ∈ [ 0 , 1 ] k\in[0,1] , the variational characterization of total variation gives | C k ( p ) − C k ( q ) | = | 𝔼 P [ k ] − 𝔼 Q [ k ] | ≤ TV ( P , Q ) . |C_{k}(p)-C_{k}(q)|=|\mathbb{E}_{P}[k]-\mathbb{E}_{Q}[k]|\leq\TV(P,Q). Pinsker’s inequality ( Pinsker, 1964 ) and the product identity KL ( p ⊗ p ∥ q ⊗ q ) = 2 KL ( p ∥ q ) \mathrm{KL}(p\otimes p\|q\otimes q)=2\mathrm{KL}(p\|q) yield TV ( P , Q ) ≤ 1 2 KL ( P ∥ Q ) = KL ( p ∥ q ) . \TV(P,Q)\leq\sqrt{\tfrac{1}{2}\mathrm{KL}(P\|Q)}=\sqrt{\mathrm{KL}(p\|q)}. Clipping collision to [ 0 , 1 ] [0,1] and inverting positive collision gives the remaining statements. ∎
For a random fitted model q θ ^ q_{\hat{\theta}} , the fixed-model bound also controls the expected kernel-collision gap: | 𝔼 θ ^ [ C k ( q θ ^ ) ] − C k ( p ) | ≤ 𝔼 θ ^ [ KL ( p ∥ q θ ^ ) ] ≤ 𝔼 θ ^ [ KL ( p ∥ q θ ^ ) ] . \left|\mathbb{E}_{\hat{\theta}}[C_{k}(q_{\hat{\theta}})]-C_{k}(p)\right|\leq\mathbb{E}_{\hat{\theta}}\!\left[\sqrt{\mathrm{KL}(p\|q_{\hat{\theta}})}\right]\leq\sqrt{\mathbb{E}_{\hat{\theta}}[\mathrm{KL}(p\|q_{\hat{\theta}})]}. (11) The first inequality uses the triangle inequality and Theorem 1 ; the second uses Jensen’s inequality.
### A.2 A sufficient condition for nonnegative alignment
Only the target–bias alignment in Eq. ( 6 ) can be negative. The following condition on the fitted model is sufficient: tokens more likely under the target have no smaller bias. A concentrated initialization does not by itself imply this condition.
###### Assumption 1 (Target–bias comonotonicity) .
For every pair a , b ∈ V a,b\in V , ( p h ( a ) − p h ( b ) ) ( b h ( a ) − b h ( b ) ) ≥ 0 , b h = 𝔼 [ q θ ^ , h ] − p h . \big(p_{h}(a)-p_{h}(b)\big)\big(b_{h}(a)-b_{h}(b)\big)\geq 0,\qquad b_{h}=\mathbb{E}[q_{\hat{\theta},h}]-p_{h}. (12)
Chebyshev’s sum inequality then gives nonnegative covariance under the uniform measure on V V . Since ∑ a b h ( a ) = 0 \sum_{a}b_{h}(a)=0 , p h ⊤ b h = | V | Cov a ∼ Unif ( V ) ( p h ( a ) , b h ( a ) ) ≥ 0 . p_{h}^{\top}b_{h}=|V|\,\Cov_{a\sim\mathrm{Unif}(V)}\!\big(p_{h}(a),b_{h}(a)\big)\geq 0. (13) All three terms inside the brackets in Eq. ( 6 ) are therefore nonnegative, and 𝔼 [ C h ( q θ ^ ) ] − C h ( p ) ≥ 0 . \mathbb{E}[C_{h}(q_{\hat{\theta}})]-C_{h}(p)\geq 0. (14) For a uniform target, the alignment is exactly zero because ∑ a b h ( a ) = 0 \sum_{a}b_{h}(a)=0 . In the synthetic experiments, alignment after fitting is negative in the diffuse setting and positive only over an intermediate N N range in the mode-aligned setting. Positive aggregate alignment does not by itself establish the pairwise comonotonicity assumption, and neither observation shows that initialization determines the later sign.
### A.3 Relative collision bounds and population cross-entropy
Dividing Theorem 1 by exact target collision gives a relative, exact-collision bound:
| C ( q h ) C ( p h ) − 1 | = | C ( q h ) − C ( p h ) | C ( p h ) ≤ ( 8 ) KL ( p h ∥ q h ) C ( p h ) = N 2 true ( h ) KL ( p h ∥ q h ) . \left|\,\frac{C(q_{h})}{C(p_{h})}-1\,\right|\;=\;\frac{\big|C(q_{h})-C(p_{h})\big|}{C(p_{h})}\;\overset{\eqref{eq:cebound}}{\leq}\;\frac{\sqrt{\mathrm{KL}(p_{h}\,\|\,q_{h})}}{C(p_{h})}\;=\;N_{2}^{\mathrm{true}}(h)\,\sqrt{\mathrm{KL}(p_{h}\,\|\,q_{h})}. (15)
###### Proposition 1 (Relative bound on the collision ratio) .
Fix a context h h with true conditional p h p_{h} and model conditional q h q_{h} , write C ( r ) = ‖ r ‖ 2 2 C(r)=\|r\|_{2}^{2} , N 2 ( r ) = 1 / C ( r ) N_{2}(r)=1/C(r) , N 2 true ( h ) = N 2 ( p h ) N_{2}^{\mathrm{true}}(h)=N_{2}(p_{h}) . The local collision ratio is R = C ( q h ) / C ( p h ) R=C(q_{h})/C(p_{h}) , which equals 1 1 when the model’s diversity is calibrated and exceeds 1 1 when the model is more repetitive than the data. With δ = q h − p h \delta=q_{h}-p_{h} (so ∑ a δ a = 0 \sum_{a}\delta_{a}=0 ), | C ( q h ) C ( p h ) − 1 | ≤ 2 N 2 true ( h ) ‖ q h − p h ‖ 2 + N 2 true ( h ) ‖ q h − p h ‖ 2 2 , \boxed{\;\left|\,\frac{C(q_{h})}{C(p_{h})}-1\,\right|\;\leq\;2\sqrt{N_{2}^{\mathrm{true}}(h)}\;\|q_{h}-p_{h}\|_{2}\;+\;N_{2}^{\mathrm{true}}(h)\,\|q_{h}-p_{h}\|_{2}^{2},\;} (16) A looser bound follows using the Pearson divergence χ 2 ( q h ∥ p h ) = ∑ a δ a 2 / p h ( a ) ≥ ∥ δ ∥ 2 2 \chi^{2}(q_{h}\,\|\,p_{h})=\sum_{a}\delta_{a}^{2}/p_{h}(a)\geq\|\delta\|_{2}^{2} (finite when q h ≪ p h q_{h}\ll p_{h} ): | C ( q h ) C ( p h ) − 1 | ≤ 2 N 2 true ( h ) χ 2 ( q h ∥ p h ) + N 2 true ( h ) χ 2 ( q h ∥ p h ) . \boxed{\;\left|\,\frac{C(q_{h})}{C(p_{h})}-1\,\right|\;\leq\;2\sqrt{N_{2}^{\mathrm{true}}(h)\,\chi^{2}(q_{h}\,\|\,p_{h})}\;+\;N_{2}^{\mathrm{true}}(h)\,\chi^{2}(q_{h}\,\|\,p_{h}).\;} (17)
Since q h = p h + δ q_{h}=p_{h}+\delta , C ( q h ) − C ( p h ) = 2 ⟨ p h , δ ⟩ + ‖ δ ‖ 2 2 C(q_{h})-C(p_{h})=2\langle p_{h},\delta\rangle+\|\delta\|_{2}^{2} , so C ( q h ) / C ( p h ) − 1 = ( 2 ⟨ p h , δ ⟩ + ‖ δ ‖ 2 2 ) / C ( p h ) C(q_{h})/C(p_{h})-1=(2\langle p_{h},\delta\rangle+\|\delta\|_{2}^{2})/C(p_{h}) . By Cauchy–Schwarz, | ⟨ p h , δ ⟩ | ≤ ‖ p h ‖ 2 ‖ δ ‖ 2 = C ( p h ) ‖ δ ‖ 2 |\langle p_{h},\delta\rangle|\leq\|p_{h}\|_{2}\|\delta\|_{2}=\sqrt{C(p_{h})}\|\delta\|_{2} , so the cross term is at most 2 ‖ δ ‖ 2 / C ( p h ) = 2 N 2 true ( h ) ‖ δ ‖ 2 2\|\delta\|_{2}/\sqrt{C(p_{h})}=2\sqrt{N_{2}^{\mathrm{true}}(h)}\|\delta\|_{2} , while the quadratic term is N 2 true ( h ) ‖ δ ‖ 2 2 N_{2}^{\mathrm{true}}(h)\|\delta\|_{2}^{2} ; the triangle inequality gives ( 16 ). For ( 17 ), absolute continuity confines all sums to the support of p h p_{h} , where 1 / p h ( a ) ≥ 1 1/p_{h}(a)\geq 1 gives ∥ δ ∥ 2 2 ≤ χ 2 ( q h ∥ p h ) \|\delta\|_{2}^{2}\leq\chi^{2}(q_{h}\|p_{h}) . ∎
The ℓ 2 \ell_{2} and χ 2 \chi^{2} forms exploit exact-collision structure. Theorem 1 is more general and uses forward KL, which equals population cross-entropy above the target entropy: KL ( p h ∥ q h ) = 𝔼 a ∼ p h [ − log q h ( a ) ] − 𝔼 a ∼ p h [ − log p h ( a ) ] , \mathrm{KL}(p_{h}\,\|\,q_{h})=\mathbb{E}_{a\sim p_{h}}\!\left[-\log q_{h}(a)\right]-\mathbb{E}_{a\sim p_{h}}\!\left[-\log p_{h}(a)\right], where the first term is the population validation objective and the second is irreducible target entropy. An empirical validation loss estimates the first term only under the same completion objective and context weighting. In particular, the survey fine-tuning loss includes the end-of-sequence (EOS) token, whereas its reported diversity uses an option-conditioned answer-position distribution; aggregate survey validation loss is not a direct estimate of the KL governing that ratio.
Let μ \mu be a specified probability distribution over token contexts, including its token-position weighting, and define KL ¯ μ = 𝔼 h ∼ μ [ KL ( p h ∥ q h ) ] \overline{\mathrm{KL}}_{\mu}=\mathbb{E}_{h\sim\mu}[\mathrm{KL}(p_{h}\|q_{h})] . Cauchy–Schwarz applied to Eq. ( 15 ) gives 𝔼 h ∼ μ | C ( q h ) C ( p h ) − 1 | ≤ 𝔼 h ∼ μ [ N 2 true ( h ) 2 ] KL ¯ μ , \mathbb{E}_{h\sim\mu}\left|\,\frac{C(q_{h})}{C(p_{h})}-1\,\right|\leq\sqrt{\mathbb{E}_{h\sim\mu}\!\big[N_{2}^{\mathrm{true}}(h)^{2}\big]\;\overline{\mathrm{KL}}_{\mu}}\,, (18) provided the displayed moments are finite. Thus average relative collision error is controlled by population excess cross-entropy under the same context measure.
## Appendix B Synthetic alignment before and after fitting
Regime (b): a mode-aligned starting distribution. The second synthetic setting first pre-trains each random GPT by soft cross-entropy to a Zipf-weighted distribution over the 32 32 highest-probability tokens of the target’s global Zipf component. Low entropy alone does not determine alignment: Eq. ( 7 ) is positive only when p h ⊤ q ¯ 0 , h > C ( p h ) p_{h}^{\top}\bar{q}_{0,h}>C(p_{h}) . The construction is chosen to satisfy that initial inequality.
After SFT, the normalized alignment term 2 p h ⊤ b h / C ( p h ) 2p_{h}^{\top}b_{h}/C(p_{h}) is positive only for N = 2.3 N=2.3 k– 21.5 21.5 k and peaks at 0.193 0.193 near N = 4,300 N=4{,}300 (Figure 2 (b)). Outside that interval its sign reverses. At N = 500 N=500 , the normalized variance Tr Cov ( q θ ^ , h ) / C ( p h ) \Tr\Cov(q_{\hat{\theta},h})/C(p_{h}) is approximately 3.9 3.9 , and the ensemble normalized collision gap G h = 𝔼 [ C h ( q θ ^ ) ] / C ( p h ) − 1 G_{h}=\mathbb{E}[C_{h}(q_{\hat{\theta}})]/C(p_{h})-1 is 3.53 3.53 . Thus the starting distribution affects the observed trajectory but does not fix the sign after fine-tuning.
The initial sign is observable. Define b 0 , h = q ¯ 0 , h − p h b_{0,h}=\bar{q}_{0,h}-p_{h} . Table 1 evaluates the decomposition before fitting to target data, where it is independent of sample size. Intervals are 95 % 95\% nonparametric bootstrap intervals from 2,000 2{,}000 resamples of the common evaluation contexts. The decomposition identity holds to machine precision ( | derived − measured | ≤ 9 × 10 − 16 |\text{derived}-\text{measured}|\leq 9\times 10^{-16} ), but these initial terms do not identify their counterparts after fitting.
## Appendix C Complete survey results
Among base checkpoints, Qwen3.5-2B is closest to median calibration ( R U = 0.99 R_{\mathrm{U}}=0.99 – 1.10 1.10 ), while the other three are under-dispersed on every survey ( 1.61 1.61 – 2.92 2.92 ). These medians do not establish pairwise distributional fit or reveal the target–bias alignment term, which would require replicated fine-tuning runs.
## Appendix D Mitigation methods and finite-sample estimation