Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs — part 4/7

Cleaned-text code points [45785, 61555). Bibliography omitted here; source.md includes it.

Mitigation via decoding and fitting objectives. Several methods improve text quality and diversity in maximum-likelihood models by modifying decoding or fitting. Decoding methods include Top- k k and nucleus sampling ( Holtzman et al., 2020 ) and truncation sampling ( Finlayson et al., 2024 ) ; the unlikelihood objective instead changes the fitting criterion ( Welleck et al., 2020 ) . Methods specific to SFT include GEM, using entropy-regularized distribution matching ( Li et al., 2025 ) ; SED-SFT, selectively regularizing entropy during exploration ( Chen et al., 2026 ) ; and TOFU, employing tempered focal loss to address the neglect of low-frequency patterns ( Klypa and Cherednichenko, 2026 ) . Finally, fine-tuning objectives enhance diversity in creative writing ( Chung et al., 2025 ) . Further approaches preserve diversity during preference optimization by separating the entropy and reference-cross-entropy components of the standard KL penalty ( Slocum et al., 2025 ) , or by preserving distributions or mixtures of preferences instead of compressing feedback to a single scalar value ( Siththaranjan et al., 2024 ; Chakraborty et al., 2024 ) . These methods provide benchmarks for our work, which contributes a finite-sample law quantifying the portion of observed diversity gaps attributable to estimation error.

Finite-sample estimation of distributional functionals. Plug-in entropy estimators are susceptible to downward bias in finite samples, especially when the alphabet size is large relative to the sample size (Miller–Madow, Basharin; see Paninski, 2003 ). The inverse-Simpson number, N 2 = exp ⁡ H 2 N_{2}=\exp H_{2} , represents the order-2 Hill number. Prior work characterizes minimax estimation of discrete functionals, including entropy and power sums ( Jiao et al., 2015 ) , estimates entropy for linguistic distributions ( Arora et al., 2022 ) , and investigates optimal prediction of unseen species, a task directly relevant to identifying valid, unobserved continuations ( Orlitsky et al., 2016 ) . Finally, diversity diagnostics have been developed, including collision-based birthday-paradox tests for low effective support in generative adversarial networks (GANs) ( Arora and Zhang, 2017 ; Arora et al., 2018 ) and the Vendi score, which measures similarity-aware diversity beyond exact labels via the Shannon entropy of a similarity matrix’s eigenvalues ( Friedman and Dieng, 2023 ) . These results are adapted to the context of autoregressive LLM fine-tuning with shared neural parameters, variable prompts and context lengths, and positive-only supervision, to interpret findings in terms of scaling laws and prior-relative lower bounds.

## Appendix E Sequence-level bounds and response length

Theorem 1 applies directly to the countable response space V ∗ V^{*} . For a bounded sequence kernel k : V ∗ × V ∗ → [ 0 , 1 ] k:V^{*}\times V^{*}\to[0,1] , define C p , k ( x ) = 𝔼 Y , Y ′ ∼ iid p ( ⋅ ∣ x ) [ k ( Y , Y ′ ) ] C_{p,k}(x)=\mathbb{E}_{Y,Y^{\prime}\overset{\mathrm{iid}}{\sim}p(\cdot\mid x)}[k(Y,Y^{\prime})] and define C q , k ​ ( x ) C_{q,k}(x) analogously. Then | C p , k ​ ( x ) − C q , k ​ ( x ) | ≤ KL ( p ( ⋅ ∣ x ) ∥ q ( ⋅ ∣ x ) ) . \big|\,C_{p,k}(x)-C_{q,k}(x)\,\big|\;\leq\;\sqrt{\mathrm{KL}\big(p(\cdot\mid x)\,\big\|\,q(\cdot\mid x)\big)}. (19)

Length enters through total excess cross-entropy. If p p and q q factor over the same tokenization and termination rule, the KL chain rule gives KL ( p ( ⋅ ∣ x ) ∥ q ( ⋅ ∣ x ) ) = 𝔼 Y ∼ p ( ⋅ ∣ x ) [ ∑ t = 1 T ⁡ ( Y ) + 1 KL ( p h t ∥ q h t ) ] , h t = ( x , Y < t ) , \mathrm{KL}\big(p(\cdot\mid x)\,\big\|\,q(\cdot\mid x)\big)\;=\;\mathbb{E}_{Y\sim p(\cdot\mid x)}\!\left[\;\sum_{t=1}^{T(Y)+1}\mathrm{KL}\big(p_{h_{t}}\,\big\|\,q_{h_{t}}\big)\right],\qquad h_{t}=(x,Y_{<t}), (20) where T ⁡ ( Y ) + 1 T(Y)+1 includes EOS. Let L ⁡ ( x ) = 𝔼 Y ∼ p ​ [ T ⁡ ( Y ) + 1 ] L(x)=\mathbb{E}_{Y\sim p}[T(Y)+1] and let KL ¯ tok ​ ( x ) \overline{\mathrm{KL}}_{\mathrm{tok}}(x) be the right-hand side divided by L ⁡ ( x ) L(x) . Equation ( 19 ) becomes | C p , k ​ ( x ) − C q , k ​ ( x ) | ≤ L ⁡ ( x ) ​ KL ¯ tok ​ ( x ) . \big|\,C_{p,k}(x)-C_{q,k}(x)\,\big|\;\leq\;\sqrt{L(x)\,\overline{\mathrm{KL}}_{\mathrm{tok}}(x)}\,. (21)

At fixed mean per-token KL, this upper bound grows as L ⁡ ( x ) \sqrt{L(x)} . It improves on the trivial maximum gap of 1 1 only when total sequence KL is below one nat. The survey statistic is a separate one-step, option-conditioned categorical comparison, whereas its fine-tuning and validation losses cover the answer encoding and EOS over the full vocabulary; those aggregate losses are not the KL in a bound for q opt q_{\mathrm{opt}} . CodeNet responses can be hundreds of tokens long, and the paper does not estimate their sequence-level KL, so Eq. ( 21 ) is not quantitatively informative for the reported Zhang–Shasha collision ratios.

The relative sequence bound is usually vacuous. Dividing the absolute exact-match bound by target collision incurs the factor 1 / C p ​ ( x ) 1/C_{p}(x) , the target’s effective number of complete responses. For open-ended generation, C p ​ ( x ) C_{p}(x) can be extremely small, so 1 / C p ​ ( x ) 1/C_{p}(x) is large and exact collisions are difficult to estimate at feasible sample sizes. A bounded structural kernel can yield a larger, more estimable C p , k ​ ( x ) C_{p,k}(x) , but the usefulness of its relative bound still depends on the target kernel collision and sequence-level KL.

## Appendix F KL-regularized preference optimization

The accounting framework also distinguishes finite-sample error from a preference objective’s population optimum. For KL-regularized reward optimization, that optimum is the Gibbs tilt π ⋆ ​ ( y ∣ x ) ∝ π ref ​ ( y ∣ x ) ​ exp ⁡ ( r ⁡ ( x , y ) / β ) \pi^{\star}(y\mid x)\propto\pi_{\mathrm{ref}}(y\mid x)\exp\!\big(r(x,y)/\beta\big) , where r r is the reward, π ref \pi_{\mathrm{ref}} is the reference policy, and β > 0 \beta>0 is the KL strength. Under its preference-model assumptions, DPO parameterizes the same implicit optimum ( Rafailov et al., 2023 ) . Whether that optimum is more concentrated than a human-response target depends on the reward, reference, and target; we do not establish such concentration here. C ⁡ ( π ⋆ ) = ∑ y π ref ​ ( y ∣ x ) 2 ​ exp ⁡ ( 2 ​ r ​ ( x , y ) / β ) [ ∑ y π ref ​ ( y ∣ x ) ​ exp ⁡ ( r ⁡ ( x , y ) / β ) ] 2 , C(\pi^{\star})=\frac{\sum_{y}\pi_{\mathrm{ref}}(y\mid x)^{2}\exp\!\big(2r(x,y)/\beta\big)}{\Big[\sum_{y}\pi_{\mathrm{ref}}(y\mid x)\exp\!\big(r(x,y)/\beta\big)\Big]^{2}}, (22) For a random fitted policy π ^ \hat{\pi} , add and subtract the collision of π ⋆ \pi^{\star} : 𝔼 ⁡ [ C ⁡ ( π ^ ) ] − C ⁡ ( p ) = ( 𝔼 ⁡ [ C ⁡ ( π ^ ) ] − C ⁡ ( π ⋆ ) ) ⏟ estimation, approximation, and optimization + ( C ⁡ ( π ⋆ ) − C ⁡ ( p ) ) ⏟ objective-induced . \boxed{\;\mathbb{E}[C(\hat{\pi})]-C(p)=\underbrace{\big(\mathbb{E}[C(\hat{\pi})]-C(\pi^{\star})\big)}_{\text{estimation, approximation, and optimization}}+\underbrace{\big(C(\pi^{\star})-C(p)\big)}_{\text{objective-induced}}.\;} (23) The first term can vanish with growing data only under suitable consistency, capacity, and optimization conditions. The second can remain even then because π ⋆ \pi^{\star} need not equal p p . Equation ( 23 ) is an accounting identity, not an empirical result of this paper; residual error may reflect either term.

## Appendix G Empirical check of the bound

Proposition 1 provides a relative bound for fixed fitted-model conditionals. Across 150,000 150{,}000 held-in contexts from 100 100 GPTs—one per pre-training sample size from 32 32 k to 1 1 M for a fixed 6.14 6.14 M-parameter model—the measured | C ⁡ ( q h ) / C ⁡ ( p h ) − 1 | |C(q_{h})/C(p_{h})-1| never exceeds the ℓ 2 \ell_{2} bound. The median measured-to-bound ratio is 0.140 0.140 and the maximum is 0.980 0.980 : the bound is typically loose and occasionally near-tight. The χ 2 \chi^{2} and forward-KL forms also hold everywhere but are looser. The minimum measured KL is 0.0353 > 0 0.0353>0 .

Complete per-question performance grids for all four models are available in the code repository accompanying this submission. These grids detail performance on every measurable item.

## Appendix H Experimental details

### H.1 Synthetic languages

How the data are generated. In setting (a), the ground-truth language is a time-homogeneous latent-factor k k -gram process with exact closed-form conditionals, vocabulary | V | = 1024 |V|=1024 and Markov order k = 16 k=16 . The token x t x_{t} at position t t in a sequence depends only on the trailing window h t := x t − 16 : t − 1 h_{t}:=x_{t-16:t-1} , where x < t x_{<t} denotes the sequence prefix before step t t . Its next-token probabilities are given by the tempered softmax of a frozen random linear map ℓ \ell of the positionally pooled context embeddings: p ⁡ ( x t ∣ x < t ) = p ⁡ ( x t ∣ h t ) := [ softmax ⁡ ( ℓ ⁡ ( h t ) / ( τ gen ​ s ) ) ] x t , p(x_{t}\mid x_{<t})=p(x_{t}\mid h_{t}):=\bigl[\softmax\!\bigl(\ell(h_{t})/(\tau_{\mathrm{gen}}\,s)\bigr)\bigr]_{x_{t}}, (24) ℓ ⁡ ( h ) := pool w ​ ( E h ) ​ W 1 ⊤ ​ W 2 ⊤ ​ W out ⊤ + b , pool w ​ ( E h ) := ∑ j = 1 16 w j ​ E h j , \textstyle\ell(h):=\mathrm{pool}_{w}(E_{h})\,W_{1}^{\top}W_{2}^{\top}W_{\mathrm{out}}^{\top}+b,\qquad\mathrm{pool}_{w}(E_{h}):=\sum_{j=1}^{16}w_{j}\,E_{h_{j}}, (25) where E ∈ ℝ 1024 × 128 E\in\mathbb{R}^{1024\times 128} is the token embedding table, E h E_{h} collects the context-token embeddings E h j E_{h_{j}} , and w ∈ ℝ 16 w\in\mathbb{R}^{16} contains positional weights w j w_{j} . The matrices W 1 ∈ ℝ 256 × 128 W_{1}\in\mathbb{R}^{256\times 128} and W 2 ∈ ℝ 256 × 256 W_{2}\in\mathbb{R}^{256\times 256} are successive linear maps, W out ∈ ℝ 1024 × 256 W_{\mathrm{out}}\in\mathbb{R}^{1024\times 256} maps to token logits, and b b is the logit-bias vector. The generation temperature is τ gen = 0.6 \tau_{\mathrm{gen}}=0.6 , and s s is the standard deviation of logits on a reference sample of 16-grams, so temperature is invariant to the overall scale of the random weights. Every ingredient is drawn at random once from the master seed 42 42 and fixed, so the language is fully reproducible. There is no intermediate nonlinearity. Because ℓ \ell is linear of rank at most 256 256 , a small GPT can represent the generating distribution p p exactly and any gap between model and truth would not be limited by model capacity. Sequences have length 64 64 : the first k = 16 k=16 tokens are drawn uniformly at random, and each subsequent token is sampled by ancestral multinomial draws from p ( ⋅ ∣ h ) p(\cdot\mid h) on the trailing 16 16 -token window; the pre-training contexts are the 16 16 -token windows at every prediction position 16 , … , 63 16,\dots,63 . The pre-training sample size N N counts sampled sequences, each contributing 48 48 supervised next-token targets (the data seed is derived from 42 42 per N N ). Because the conditionals are closed-form, the exact target diversity N 2 ​ ( p h ) = 1 / ∑ a p ​ ( a ∣ h ) 2 N_{2}(p_{h})=1/\sum_{a}p(a\mid h)^{2} is known at every context; it is high and varied (median 177 177 , range 7 7 – 398 398 ). At k = 16 k=16 exact contexts essentially never repeat, so the count-based/tabular prediction of Appendix I is structurally undefined here. Diffuse prior in setting (a). For each seed, we initialize the GPT using its default random initialization, without preliminary fitting. The starting distribution q 0 , h q_{0,h} is its softmax output at context h h before pre-training on the synthetic language. It is generally nonuniform and varies with context and initialization seed.

What we measure. We report the ensemble normalized gap G h = 𝔼 ⁡ [ R ] − 1 = 𝔼 ⁡ [ C h ​ ( q θ ^ ) ] / C h ​ ( p ) − 1 G_{h}=\mathbb{E}[R]-1=\mathbb{E}[C_{h}(q_{\hat{\theta}})]/C_{h}(p)-1 and its three C ⁡ ( p h ) C(p_{h}) -normalized terms from Eq. ( 6 ). Isolating them requires independently pre-trained models at each N N . We pre-train 100 100 seeds at each of 100 100 log-spaced sequence counts from 32 32 k to 1 1 M ( 10,000 10{,}000 GPTs; 6.135 6.135 M parameters) and evaluate them on 2,000 2{,}000 common contexts. Each seed draws its own data, initialization, and minibatch stream, so the covariance captures their total across-run variation. The closed-form p h p_{h} permits direct estimation of b h b_{h} from the across-seed mean. All validation losses are below the uniform baseline log ⁡ 1024 ≈ 6.93 \log 1024\approx 6.93 nats. Setup. We fine-tune a 27.97 27.97 M-parameter GPT ( d model = 256 d_{\text{model}}=256 , 4 4 heads, 3 3 layers, K = 50,000 K=50{,}000 ) on a heavy-tailed order- 16 16 Zipf-plus-context language with exponent 1.07 1.07 . Each length- 17 17 sequence contributes one prediction target. Before SFT, we select the 32 32 most probable tokens in the target language’s fixed token-frequency distribution and rescale their probabilities to sum to one. We then pre-train each GPT for 400 400 steps on random contexts, using this same desired next-token distribution for every context. The analyzed artifacts contain 100 100 seeds at each of 100 100 sequence counts from 500 500 to 10 6 10^{6} ; every model then undergoes 4,000 4{,}000 SFT steps.

### H.2 Surveys

Datasets. We study three long-running probability surveys of human attitudes. The General Social Survey (GSS) is a nationally representative survey of adults in the United States, conducted since 1972 1972 ; we use the 1972 1972 – 2024 2024 cumulative file ( 74,485 74{,}485 respondents with complete demographics). The World Values Survey (WVS) is a cross-national survey of social, political, religious, and economic values; we use wave 7 (fieldwork 2017 2017 – 2023 2023 ). For computational tractability we restrict the WVS to a random one-third of its countries and a random one-third of its measurable questions, both drawn with a fixed seed ( 42 42 ) and frozen before any distribution is computed; the resulting subset comprises 32,227 32{,}227 respondents with complete demographics across 22 22 countries spanning every inhabited continent (e.g. India, Pakistan, Kenya, Morocco, Serbia, South Korea, Canada, the Netherlands). The American National Election Studies (ANES) is a long-running U.S. political survey; we use the Time-Series Cumulative Data File ( 1948 1948 – 2020 2020 ; 69,784 69{,}784 respondents with complete demographics). From each survey we take every suitable categorical opinion item—substantive attitude/value/policy questions with a small fixed answer set, with value labels transcribed verbatim from each survey’s codebook (no guessed labels)—excluding identifiers, weights, dates, interviewer/geographic metadata, raw numerics, multi-select “mention” batteries, and behavioral/frequency/factual items. For the GSS, items are identified by their attitudinal value-label sets (favor/oppose, agree–disagree, spending too-little/too-much, confidence, importance, likely, should, true/false). We then keep only the measurable items—those with at least one demographic group of ≥ 60 \geq 60 respondents, the same threshold used for respondent-sample target estimation—leaving 376 376 items for the GSS, 159 159 for the WVS (of which the random-third restriction above retains 53 53 ), and 44 44 for the ANES; every retained item is evaluated. The seven curated items used in an earlier version of this experiment (capital punishment cappun , gun permits gunlaw , welfare spending natfare , happiness happy , financial satisfaction satfin , political views polviews , party identification partyid for the GSS, and the analogous WVS seven) are a strict subset of the GSS measurable set, which we exploit for the seven-vs-full comparison (GSS only: the WVS random-third restriction does not retain all seven curated WVS items).