Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs — part 1/7

Cleaned-text code points [0, 15499). Bibliography omitted here; source.md includes it.

# Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs

###### Abstract

Recent work by Doshi and Hauser (2024) , Bisbee et al. (2024) , and Xie et al. (2026) raises concerns that outputs from large language models (LLMs) tend to be under-diverse: they repeat or resemble one another more often than responses from the population they are meant to represent, a phenomenon known as mode collapse. In this work, we show that whether mode-collapse—or its opposite—occurs depends on the specific model and dataset used. Further, with sufficient supervised fine-tuning (SFT) data, LLM output diversity converges toward that of the target distribution from which fine-tuning data are sampled. To quantify this comparison, we measure the probability that two responses sampled independently conditional on the same fixed prompt coincide (collide). In one set of experiments, we use exact token sequences; in the other, we use the generalised version of that—the expected similarity between responses under a kernel. We define miscalibration as a nonzero model-minus-target collision gap. We derive a bias–variance decomposition of the expected gap between the model’s and target’s collision probabilities, showing that SFT is not inherently biased toward mode collapse or its opposite: finite-sample SFT can leave a model either under- or over-dispersed, depending on the model and dataset. Finally, we show that the absolute gap is bounded by the square root of the Kullback–Leibler (KL) divergence from the target distribution to the model. Consequently, a model sufficiently close to optimal under population cross-entropy cannot exhibit arbitrarily miscalibrated diversity. We test the decomposition and the bound in three experiments: (1) we fit 100 small transformers at each of 100 log-spaced sample sizes on each of two synthetic order-16 languages; (2) we fine-tune four LLMs using low-rank adaptation on responses from the General Social Survey, American National Election Studies, and World Values Survey; and (3) we repeat Experiment (2) on CodeNet, a dataset of human solutions to coding tasks, measuring program similarity with normalized Zhang–Shasha edit distance between canonical abstract syntax trees. We find substantial heterogeneity in over- and under-diversity across models and datasets. More target data moves model diversity toward the human (or synthetic target) level in all experiments, consistent with our theoretical predictions. These results show that diversity miscalibration can arise from finite-sample error and shrink as SFT better approximates the target distribution. Accordingly, as the sample size of target-distribution data increases, model diversity moves toward the target level.

## 1 Introduction

Large language models (LLMs) are increasingly used as simulated survey respondents ( Argyle et al., 2023 ) , for creative writing ( Chakrabarty et al., 2024 ) , as brainstorming partners ( Terwiesch et al., 2024 ) , and in other domains where faithfulness of the LLM’s outputs to the diversity of the target population is important. Empirical studies show that LLMs often exhibit a narrow range of outputs, a phenomenon known as mode collapse: the model’s outputs repeat, or resemble one another, more often than samples from the population they are meant to represent. For instance, stories written with artificial intelligence (AI) assistance tend to be similar to one another ( Doshi and Hauser, 2024 ) ; AI-assisted users may generate ideas that are less semantically distinct ( Anderson et al., 2024 ) ; and synthetic survey respondents can reproduce aggregate patterns while suppressing individual and group-level variation ( Bisbee et al., 2024 ; Wang et al., 2025 ; Xie et al., 2026 ) .

These findings are often interpreted as evidence that low output diversity is an inherent limitation of LLMs ( Xie and Xie, 2025 ) . However, when a pre-trained or instruction-tuned model is fine-tuned on a new dataset, its starting distribution is being compared with a target distribution it has not yet learned. It is unclear whether this under-diversity persists as a model learns the target distribution more accurately. If mode collapse is inherent, increasing the amount of target-distribution fine-tuning data need not eliminate it; if, instead, mode collapse and other distribution miscalibrations simply reflect finite-sample error, model diversity should converge toward the target level with sufficient fine-tuning. To settle this distinction, we begin with formalizing it theoretically.

We quantify LLM output diversity by collision probability—the probability that two responses sampled independently conditional on the same fixed prompt coincide—or, more generally, by their expected similarity under a kernel (Section 3 ). For next-token comparisons, we instead fix the full context: the prompt and token prefix. Model and target are compared under the same conditioning and similarity measure. Their collision gap is the model’s collision probability minus the target’s; where zero indicates calibrated diversity, a positive gap indicates mode collapse, and a negative gap indicates over-dispersion.

This formalization allows us to make three claims (Figure 1 ). (1) SFT does not inherently produce under- or over-dispersion; the direction depends on the model and dataset. The difference in collision probabilities, averaged over independent fits, decomposes into nonnegative variance and squared-bias terms, which promote concentration, and a sign-indefinite target–bias alignment term that can offset them (Section 3.1 ). (2) Magnitude: the absolute collision gap is bounded by the square root of the Kullback–Leibler (KL) divergence from the target distribution to the model (Theorem 1 ). This bound implies (3) Calibration in the limit: as this KL divergence approaches zero, the collision gap necessarily approaches zero. In practice, this leads to the intuitive conclusion that if more target data and loss minimization bring population cross-entropy toward the value achieved by the target distribution itself, model diversity must approach the target level.

To support our claims empirically, we set up three experiments (Section 4 ). Experiment 1 tests claim (1) and checks the bound of claim (2): in two synthetic order- 16 16 language settings with exactly computable targets, models with the generative pre-trained transformer (GPT) architecture fitted across 100 100 log-spaced sample sizes produce collision distortions of both signs (Section 4.1 , Figure 2 ), and the bound holds on every evaluated context (Appendix G , Figure 5 ). Experiments 2 and 3 test claim (3) from opposite starting points. We use low-rank adaptation (LoRA) to fine-tune four instruction-tuned LLMs ( gemma-2-2b-it ( Gemma Team, 2024 ) , gemma-3-4b-it ( Gemma Team, 2025 ) , Qwen3.5-2B and Qwen3.5-4B ( Qwen Team, 2026 ) ) on General Social Survey (GSS) ( Davern et al., 2025 ) , American National Election Studies (ANES) ( American National Election Studies, 2022 ) , and World Values Survey (WVS) ( Haerpfer et al., 2022 ) . We find that three of the four base models are 1.6 1.6 to 2.9 2.9 times under-dispersed relative to the human populations, and SFT on human answers restores population-level heterogeneity for every model on every survey (Section 4.2 , Figure 3 ). Next, we apply the same method to CodeNet ( Puri et al., 2021 ) , a dataset of human solutions to coding tasks, where we measure distance between solutions using the normalized Zhang–Shasha similarity kernel ( Zhang and Shasha, 1989 ) . On this same dataset, some base models are under-diverse and others over-diverse ( R = 0.74 R=0.74 – 1.64 1.64 ): mode collapse depends on the model. Supervised fine-tuning narrows the gap from both directions, as predicted by theory.

## 2 Related work

Evidence of output mode collapse. Generative AI can enhance individual story ratings and increase similarity between AI-assisted stories ( Doshi and Hauser, 2024 ) , while co-writing with InstructGPT reduces content diversity in essays ( Padmakumar and He, 2024 ) . ChatGPT users generate more numerous and detailed ideas than users of another creativity-support tool, but these ideas exhibit less semantic distinctiveness across users ( Anderson et al., 2024 ) . Sourati et al. (2026) argue that LLMs reinforce dominant styles of language and reasoning, and that widespread reliance on the same few models amplifies this convergence. Our results show that SFT is not inherently biased toward mode collapse or its opposite: the direction of miscalibration depends on the model and dataset, and fine-tuning can bring diversity toward the target from either direction. LLMs are also employed as proxies for human participants in “silicon sampling” studies ( Argyle et al., 2023 ) , but recent work shows that synthetic surveys can match aggregate statistics while failing to capture individual variation and regression structures ( Bisbee et al., 2024 ) , and can misrepresent and flatten identity groups ( Wang et al., 2025 ) . A benchmark of 15 LLMs across seven surveys documents the same compression of heterogeneity ( Xie et al., 2026 ) . Recursive fitting on model-generated data also removes data points from the tails of the original distribution ( Shumailov et al., 2024 ; Seddik et al., 2024 ) , illustrating the ease with which generative resampling can erase rare modes.

Preference optimization and diversity. Preference optimization has been shown to shrink output diversity. Reinforcement learning from human feedback (RLHF) with proximal policy optimization (PPO) reduces both within- and across-prompt diversity for summarization, as assessed by multiple lexical and semantic metrics, though the same study finds no meaningful diversity differences for instruction following ( Kirk et al., 2024 ) . The stage at which this contraction occurs varies across model lineages, with some models showing the largest reduction during SFT and others during direct preference optimization (DPO) ( Karouzos et al., 2026 ) . These effects are linked to preference data and annotator heterogeneity. Annotators tend to favor responses to which base models assign higher likelihood even when responses are matched on correctness; consequently, a perfectly estimated preference reward can preserve typicality bias toward conventional responses ( Zhang et al., 2025 ) . Finally, aggregating annotators with heterogeneous criteria implicitly implements the Borda-count rule rather than recovering an observer-independent utility ( Siththaranjan et al., 2024 ) , and a single scalar reward may not represent multimodal population preferences ( Chakraborty et al., 2024 ) . For a reference policy π ref \pi_{\mathrm{ref}} , population reward r ⋆ r^{\star} , and KL penalty strength β \beta , the optimal policy has the form π ⋆ ​ ( y ∣ x ) ∝ π ref ​ ( y ∣ x ) ​ exp ⁡ ( r ⋆ ​ ( x , y ) / β ) \pi^{\star}(y\mid x)\propto\pi_{\mathrm{ref}}(y\mid x)\exp\!\big(r^{\star}(x,y)/\beta\big) , and DPO directly fits a policy within this implicit-reward class ( Rafailov et al., 2023 ) . Finally, common reward structures and regularization regimes can produce unimodal or concentrated optima under both forward and reverse KL divergence, indicating that the KL penalty alone does not preserve the reference distribution’s diversity ( GX-Chen et al., 2026 ) . These population-level mechanisms persist as preference data size increases. We focus instead on the finite-sample distortion around the population objective, adding a finite-sample component to the objective-induced gap C ⁡ ( π ⋆ ) − C ⁡ ( p ) C(\pi^{\star})-C(p) from existing literature (Appendix F ). Because form-sensitive and content-sensitive diversity metrics capture different aspects of generator behavior ( Tevet and Berant, 2021 ) , our conclusions are metric-specific.

Relation to prior work. Xie et al. (2026) find compressed heterogeneity in survey microdata generated by 15 LLMs; fine-tuning one model on 1,000 records improves aggregate realism on held-out surveys. Our analysis shows that the collision gap can have either sign and must vanish as excess population cross-entropy approaches zero (§ 3.1 , Theorem 1 ). Whereas the entropy-regularized game-theoretic method GEM ( Li et al., 2025 ) , Selectively Encouraging Diversity in Supervised Fine-Tuning (SED-SFT) ( Chen et al., 2026 ) , and tempered focal (TOFU) loss ( Klypa and Cherednichenko, 2026 ) modify the SFT objective to protect diversity, our experiments show that plain SFT moves diversity toward the target across four models, three surveys, and CodeNet, from both directions. Banayeeanzade et al. (2026) attribute sequence-level diversity loss to miscalibrated token probabilities; our decomposition and bound hold for any fitting procedure.

## 3 Theory

Let V V be the token vocabulary, and V ∗ V^{*} the set of finite token sequences. Let prompts x x be drawn from a distribution ν \nu , and let p ⁡ ( y ∣ x ) p(y\mid x) be the target distribution over responses y ∈ V ∗ y\in V^{*} . The model outputs token a t a_{t} given context h t := ( x , y < t ) h_{t}:=(x,y_{<t}) , where y < t y_{<t} is the response prefix before step t t . The true conditional distribution is p h ​ ( a ) := p ​ ( a ∣ h ) p_{h}(a):=p(a\mid h) . For an LLM with parameters θ \theta , f θ ​ ( h ) f_{\theta}(h) is the vector of pre-softmax scores (logits) predicted by the model given context h h . Applying softmax gives q θ ​ ( a ∣ h ) := softmax ⁡ ( f θ ​ ( h ) ) a q_{\theta}(a\mid h):=\softmax(f_{\theta}(h))_{a} , the model’s probability of token a a given context h h . SFT fits θ \theta by minimizing empirical cross-entropy, the average of − log ⁡ q θ ​ ( a ∣ h ) -\log q_{\theta}(a\mid h) over observed context–token pairs ( h , a ) (h,a) in the fine-tuning data. A model fit produces parameters θ ^ \hat{\theta} and model q θ ^ q_{\hat{\theta}} .

We now define collision for an arbitrary distribution π \pi over tokens. Collision is the probability that two tokens drawn independently from π \pi are equal: c ⁡ ( π ) := ∑ a ∈ V π ​ ( a ) 2 . \textstyle c(\pi):=\sum_{a\in V}\pi(a)^{2}. (1)

The inverse Simpson index is the number of equally likely choices that would give the same collision probability. For K K equally likely tokens, it equals K K . For target and model distributions p p and q q , define N 2 ​ ( p ) := 1 C ⁡ ( p ) , N 2 ​ ( q ) := 1 C ⁡ ( q ) , \textstyle N_{2}(p):=\frac{1}{C(p)},\qquad N_{2}(q):=\frac{1}{C(q)}, (2) where C C denotes collision. At a fixed context h h , C ⁡ ( p ) = c ⁡ ( p h ) C(p)=c(p_{h}) and C ( q ) = c ( q θ ( ⋅ ∣ h ) ) C(q)=c(q_{\theta}(\cdot\mid h)) .

We compare model and target diversity through the collision ratio R := C ⁡ ( q ) C ⁡ ( p ) = N 2 ​ ( p ) N 2 ​ ( q ) , \textstyle R:=\frac{C(q)}{C(p)}=\frac{N_{2}(p)}{N_{2}(q)}, (3) where C ⁡ ( p ) > 0 C(p)>0 : R = 1 R=1 indicates calibrated diversity, R > 1 R>1 mode collapse, and R < 1 R<1 over-dispersion.

For a fixed prompt x x , draw complete responses Y Y and Y ′ Y^{\prime} independently from p ( ⋅ ∣ x ) p(\cdot\mid x) . Their joint law is the product measure p ( . | x ) ⊗ p ( . | x ) p(.|x)\otimes p(.|x) ; independence is conditional on x x , and averaging over a shared random prompt need not preserve it. Under a similarity kernel k ⁡ ( y , y ′ ) ∈ [ 0 , 1 ] k(y,y^{\prime})\in[0,1] , kernel collision is C p , k ( x ) = 𝔼 Y , Y ′ ∼ p ( ⋅ ∣ x ) [ k ( Y , Y ′ ) ] . C_{p,k}(x)=\mathbb{E}_{Y,Y^{\prime}\sim p(\cdot\mid x)}[k(Y,Y^{\prime})]. (4)