Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs — part 5/7
Cleaned-text code points [61555, 77534). Bibliography omitted here; source.md includes it.
Grouping, conditioning, and respondent-sample target. Respondents are partitioned into demographic groups X X defined by a tuple of variables, brought to GSS parity across surveys: • GSS : X = ( decade , age bucket , sex , race , region , degree ) X=(\text{decade},\ \text{age bucket},\ \text{sex},\ \text{race},\ \text{region},\ \text{degree}) ;
• WVS : X = ( country , survey year , urban/rural , age bucket , sex , education ) X=(\text{country},\ \text{survey year},\ \text{urban/rural},\ \text{age bucket},\ \text{sex},\ \text{education}) ;
• ANES : X = ( decade , age bucket , sex , race , region , education ) X=(\text{decade},\ \text{age bucket},\ \text{sex},\ \text{race},\ \text{region},\ \text{education}) ,
Age was bucketed into groups { 18 – 29 , 30 – 44 , 45 – 59 , 60 + } \{18\text{--}29,30\text{--}44,45\text{--}59,60+\} in the GSS and { 16 – 29 , 30 – 44 , 45 – 59 , 60 + } \{16\text{--}29,30\text{--}44,45\text{--}59,60+\} in the WVS and ANES. ANES and WVS aggregate calendar years (to decade and fieldwork year, respectively) to avoid sparse groups. For each fixed question and group X X , with Y Y denoting the answer, observed counts m a m_{a} among m m respondents define the unweighted plug-in conditional p ^ a = m a / m \hat{p}_{a}=m_{a}/m and plug-in collision C ^ plug = ∑ a p ^ a 2 \hat{C}_{\mathrm{plug}}=\sum_{a}\hat{p}_{a}^{2} . We report instead the distinct-pair U-statistic C ^ U = ∑ a m a ( m a − 1 ) / [ m ( m − 1 ) ] = ( m C ^ plug − 1 ) / ( m − 1 ) \hat{C}_{\mathrm{U}}=\sum_{a}m_{a}(m_{a}-1)/[m(m-1)]=(m\hat{C}_{\mathrm{plug}}-1)/(m-1) . The target uses all eligible respondents, including fine-tuning and validation records. It represents this unweighted sample, not a held-out or survey-weighted estimate of the wider population. We compute each pair’s collision ratio R U = C ( q opt ) / C ^ U R_{\mathrm{U}}=C(q_{\mathrm{opt}})/\hat{C}_{\mathrm{U}} before taking medians across pairs. Evaluation requires at least 60 60 respondents for a question–group pair. Table 3 gives the sample sizes.
Prompt and measurement. The prompt presents the group as a first-person persona followed by the question and lettered options, then requests one option letter; Tables 4 , 5 , and 6 show the templates. At the generation position, we take the first token identifier of each letter’s tokenizer encoding, assert that the candidate identifiers are distinct, and apply a softmax only across their logits. The resulting q opt ( Y ∣ X ) q_{\mathrm{opt}}(Y\mid X) is exact conditional on the candidate set and has no sampling noise, but it is not the full next-token distribution. We fine-tune four models of at most 4 4 B parameters— gemma-2-2b-it , Qwen3.5-2B , Qwen3.5-4B , and gemma-3-4b-it —with shared hyperparameters and model-specific LoRA target scopes. A respondent-level split prevents leakage between the fine-tuning and validation splits, while the empirical target intentionally uses both splits. The pipeline saves an early-stopped adapter and the final adapter; callback state is not persisted across requeues, so the former is not guaranteed to be the global minimum-validation-loss checkpoint. The sample-size grid halves each survey’s fine-tuning pool N N down to the smallest n ≥ 8 n\geq 8 (GSS: 20 20 points, 4,706,940 4{,}706{,}940 to 8 8 ; WVS: 18 18 , 1,462,645 1{,}462{,}645 to 11 11 ; ANES: 18 18 , 1,662,788 1{,}662{,}788 to 12 12 ). For multimodal gemma-3-4b-it , LoRA and measurement use only the text decoder.
It is the {decade}. You are a {age} year-old {race} {sex} living in the {region} region of the United States. Your highest education credential is: {degree}. {question} Options: A) {option A} B) {option B} … Respond with only the single letter of the option that best matches your view.
It is {year}. You live in {urban/rural area} of {country}. You are a {age} year-old {sex}. Your highest level of education is {education}. {question} Options: A) {option A} B) {option B} … Respond with only the single letter of the option that best matches your view.
It is the {decade}s. You are a {age} year-old {sex} living in {region} of the United States. You identify as {race} and your highest level of education is {education}. {question} Options: A) {option A} B) {option B} … Respond with only the single letter of the option that best matches your view.
Fine-tuning hyperparameters. All four models use LoRA adapters with shared hyperparameters and model-specific target scope. Fine-tuning and validation use the same shifted mean cross-entropy over the encoded answer-letter token(s) followed by EOS; prompt and padding positions are masked. Runs use bf16 , gradient checkpointing, and base use_cache disabled. Tables 7 and 8 give shared settings and per-model scope. The text models apply LoRA to seven attention and multilayer perceptron (MLP) projections in every decoder layer; gemma-3-4b-it applies the same projections only within its language_model , leaving the vision tower fixed. The adapters modify 0.47 % 0.47\% – 0.79 % 0.79\% of parameters ( 10.9 10.9 – 29.8 29.8 million).
### H.3 CodeNet
Dataset and estimand. Project CodeNet contains programming-contest problems, submitted programs, and metadata including language, user, timestamp, and judge status ( Puri et al., 2021 ) . We restrict the target to parseable, accepted Python submissions and compare implementation structure rather than lexical form or execution output. We measure expected similarity under the normalized Zhang–Shasha kernel in Equation ( 10 ), conditional on both programs parsing successfully. For each problem and distribution, we average the kernel over a deterministic seed- 42 42 sample of up to 32 32 distinct unordered pairs, using every pair when fewer are available.
The metadata contain 13,916,868 13{,}916{,}868 submissions, including 3,286,314 3{,}286{,}314 labeled Python and 1,796,563 1{,}796{,}563 both Python and accepted, across 3,113 3{,}113 problems. We impose no restriction on users, source length, or revisions per user; multiple accepted revisions remain separate observations. A submission enters the usable target only if its description and source are available and its Python AST parses successfully.
We split solutions deterministically 90 90 – 10 10 at the submission level. A problem may appear in both partitions, but a submission identifier cannot. The 90 % 90\% partition supplies SFT data and the 10 % 10\% partition validation loss; the structural target draws up to 32 32 parseable accepted solutions from both. Problems with one usable solution remain available for SFT, while target collision requires at least two. Fine-tuning configuration. We use google/gemma-2-2b-it , Qwen/Qwen3.5-2B , Qwen/Qwen3.5-4B , and google/gemma-3-4b-it . For fine-tuning-pool size N N , the grid successively halves nested prefixes of a deterministic submission order down to n ≥ 8 n\geq 8 . Each model runs for three epochs with the survey experiment’s LoRA rank, scaling, dropout, learning rate, cosine schedule, 4,000 4{,}000 -row validation subset, 12 12 scheduled validation evaluations, and effective batch size 16 16 . Fine-tuning and validation use shifted mean cross-entropy over code plus the tokenizer’s assistant terminator, with the prompt masked. We report the final adapter after three epochs; validation loss monitors fine-tuning but does not select the reported checkpoint, and loss magnitudes are not compared across tokenizers.
Decoding and parse conditioning. For each problem and checkpoint, we draw 32 32 samples with temperature 1.0 1.0 , top- p = 1.0 p=1.0 , and at most 1,024 1{,}024 new tokens. The generation call does not set top- k k or enable_thinking , so loaded model and chat-template defaults supply them; model revisions and the effective generation configuration were not logged. The loaded Qwen templates are asymmetric: Qwen3.5-2B defaults to non-thinking output, whereas Qwen3.5-4B defaults to thinking output. Decoding is therefore not matched across models. Seeds derive deterministically from seed 42 42 , problem identifier, fine-tuning size, and checkpoint. Outputs that fail AST parsing are recorded but excluded from collision; a ratio is finite only when at least two outputs parse. Thus cross-model diversity and parseability comparisons are confounded by decoding defaults, and all Zhang–Shasha diversity results are conditional on parsing.
Comparison on the same problems. For each model, we retain problems with at least two parseable outputs both before fine-tuning and after fine-tuning on the full dataset. In Figure 4 panel order, these sets contain 2,646 2{,}646 , 2,569 2{,}569 , 986 986 , and 2,646 2{,}646 problems. Their median collision ratios change from 1.035 1.035 to 0.971 0.971 , 0.740 0.740 to 0.915 0.915 , 0.844 0.844 to 1.003 1.003 , and 1.637 1.637 to 0.974 0.974 , respectively. Every model’s median therefore moves closer to 1 1 when evaluated on the same problems at both stages. Both estimates remain conditional on parsing.
Code-generation prompt. The problem description is inserted after the following instruction in one user turn; each model’s chat template supplies its own control tokens. You are an expert competitive programmer. Return only a complete Python 3 program that solves the problem. Do not use Markdown fences or explanations. {problem description}
Two accepted-solution examples. The following pairs illustrate the primary metric using accepted Python submissions. Similarities are recomputed with exactly the normalized Zhang–Shasha kernel in Equation ( 10 ). Each table has four cells in the requested order: source code A, canonical AST A, source code B, and canonical AST B. The AST cells are complete automatic exports from the metric representation. They contain every retained node and every ordered, field-typed child edge; nothing is manually pruned. Source locations and Load / Store context nodes are absent because the metric itself excludes them.
High structural similarity: divisor rule ( p03125 ). The problem gives positive integers A A and B B . If A A divides B B , the program prints A + B A+B ; otherwise, it prints B − A B-A . Both accepted submissions use the same conditional implementation. Their normalized Zhang–Shasha similarity is 0.970588 0.970588 (edit distance 2 2 across two 34 34 -node trees).
## Appendix I Tabular intuition for finite-sample collision
The classical per-context histogram provides a deliberately restricted comparison to the neural decomposition in Section 3.1 . Fix a context h h with n h n_{h} independent and identically distributed (i.i.d.) samples from p h p_{h} and empirical maximum-likelihood estimate (MLE) p ^ h ( a ) = n h , a / n h \hat{p}_{h}(a)=n_{h,a}/n_{h} . Then 𝔼 [ p ^ h ( a ) ] = p h ( a ) \mathbb{E}[\hat{p}_{h}(a)]=p_{h}(a) : finite samples favor no token in expectation, although any realized dataset overweights some tokens.
Since n h , a ∼ Binomial ( n h , p h ( a ) ) n_{h,a}\sim\mathrm{Binomial}(n_{h},p_{h}(a)) , 𝔼 [ p ^ h ( a ) ] = 𝔼 [ n h , a ] / n h = p h ( a ) \mathbb{E}[\hat{p}_{h}(a)]=\mathbb{E}[n_{h,a}]/n_{h}=p_{h}(a) . ∎
Conditioning on one realized histogram makes its sampling fluctuations shared across all subsequent generations.
Simple example. For p = ( 1 / 3 , 1 / 3 , 1 / 3 ) p=(1/3,1/3,1/3) and n = 6 n=6 , realized counts might be ( 4 , 1 , 1 ) (4,1,1) or ( 1 , 4 , 1 ) (1,4,1) . The favored token changes across datasets, but both histograms have higher collision than p p .
### I.1 Expected collision inflation
For arbitrary p h p_{h} , 𝔼 [ p ^ h ( a ) 2 ] = p h ( a ) 2 + p h ( a ) ( 1 − p h ( a ) ) n h \mathbb{E}[\hat{p}_{h}(a)^{2}]=p_{h}(a)^{2}+\frac{p_{h}(a)(1-p_{h}(a))}{n_{h}} . Summing over a a , 𝔼 [ C ( p ^ h ) ] = C ( p h ) + 1 − C ( p h ) n h , \mathbb{E}[C(\hat{p}_{h})]=C(p_{h})+\frac{1-C(p_{h})}{n_{h}}, (26) so the local expected collision ratio is 𝔼 [ C ( p ^ h ) ] C ( p h ) = 1 + 1 − C ( p h ) n h C ( p h ) = 1 + N 2 ( p h ) − 1 n h . \frac{\mathbb{E}[C(\hat{p}_{h})]}{C(p_{h})}=1+\frac{1-C(p_{h})}{n_{h}\,C(p_{h})}=1+\frac{N_{2}(p_{h})-1}{n_{h}}. (27) This exact identity is the local scaling law for the tabular MLE: local expected collision ratio = 1 + target effective diversity N 2 ( p h ) − 1 sample count n h . \text{local expected collision ratio}=1+\frac{\text{target effective diversity }N_{2}(p_{h})-1}{\text{sample count }n_{h}}. (28) The expected collision ratio is also exactly 𝔼 [ N 2 ( p h ) / N 2 ( p ^ h ) ] \mathbb{E}[N_{2}(p_{h})/N_{2}(\hat{p}_{h})] , because N 2 ( p h ) / N 2 ( p ^ h ) = C ( p ^ h ) / C ( p h ) N_{2}(p_{h})/N_{2}(\hat{p}_{h})=C(\hat{p}_{h})/C(p_{h}) for every dataset. It generally differs from N 2 ( p h ) / 𝔼 [ N 2 ( p ^ h ) ] N_{2}(p_{h})/\mathbb{E}[N_{2}(\hat{p}_{h})] .
Fix the context h h and abbreviate p ( a ) = p h ( a ) p(a)=p_{h}(a) , n = n h n=n_{h} . The observed counts ( n h , a ) a (n_{h,a})_{a} are Multinomial ( n , p h ) \mathrm{Multinomial}(n,p_{h}) ; in particular each n h , a ∼ Binomial ( n , p ( a ) ) n_{h,a}\sim\mathrm{Binomial}(n,p(a)) , so the empirical MLE p ^ h ( a ) = n h , a / n \hat{p}_{h}(a)=n_{h,a}/n has 𝔼 [ p ^ h ( a ) ] = p ( a ) , Var ( p ^ h ( a ) ) = p ( a ) ( 1 − p ( a ) ) n . \mathbb{E}[\hat{p}_{h}(a)]=p(a),\qquad\Var(\hat{p}_{h}(a))=\frac{p(a)\big(1-p(a)\big)}{n}. (29) By the variance–mean-square identity 𝔼 [ X 2 ] = Var ( X ) + ( 𝔼 X ) 2 \mathbb{E}[X^{2}]=\Var(X)+(\mathbb{E}X)^{2} , 𝔼 [ p ^ h ( a ) 2 ] = p ( a ) ( 1 − p ( a ) ) n + p ( a ) 2 . \mathbb{E}[\hat{p}_{h}(a)^{2}]=\frac{p(a)\big(1-p(a)\big)}{n}+p(a)^{2}. (30) Summing over a a and using linearity of expectation (which needs no independence across tokens, so the within-multinomial correlations are irrelevant), 𝔼 [ C ( p ^ h ) ] = ∑ a 𝔼 [ p ^ h ( a ) 2 ] = ∑ a p ( a ) 2 + 1 n ∑ a p ( a ) ( 1 − p ( a ) ) = C ( p h ) + 1 − C ( p h ) n , \mathbb{E}[C(\hat{p}_{h})]=\sum_{a}\mathbb{E}[\hat{p}_{h}(a)^{2}]=\sum_{a}p(a)^{2}+\frac{1}{n}\sum_{a}p(a)\big(1-p(a)\big)=C(p_{h})+\frac{1-C(p_{h})}{n}, (31) where the last equality uses ∑ a p ( a ) = 1 \sum_{a}p(a)=1 , hence ∑ a p ( a ) ( 1 − p ( a ) ) = 1 − ∑ a p ( a ) 2 = 1 − C ( p h ) \sum_{a}p(a)(1-p(a))=1-\sum_{a}p(a)^{2}=1-C(p_{h}) . Dividing by C ( p h ) C(p_{h}) and substituting N 2 ( p h ) = 1 / C ( p h ) N_{2}(p_{h})=1/C(p_{h}) , 𝔼 [ C ( p ^ h ) ] C ( p h ) = 1 + 1 − C ( p h ) n C ( p h ) = 1 + N 2 ( p h ) − 1 n , \frac{\mathbb{E}[C(\hat{p}_{h})]}{C(p_{h})}=1+\frac{1-C(p_{h})}{n\,C(p_{h})}=1+\frac{N_{2}(p_{h})-1}{n}, (32) because ( 1 − C ( p h ) ) / C ( p h ) = N 2 ( p h ) − 1 (1-C(p_{h}))/C(p_{h})=N_{2}(p_{h})-1 . Restoring n = n h n=n_{h} gives the stated law. ∎
The critical factor is not the overall size of the SFT dataset, but rather the local effective sample size, n h n_{h} , representing the number of fine-tuning examples that contribute meaningfully to context h h .
###### Example 1 (Why any finite sample inflates collision (Jensen)) .
Collision C ( r ) = ∑ a r ( a ) 2 C(r)=\sum_{a}r(a)^{2} is convex. Since 𝔼 [ p ^ h ] = p h \mathbb{E}[\hat{p}_{h}]=p_{h} , Jensen’s inequality gives 𝔼 [ C ( p ^ h ) ] ≥ C ( p h ) \mathbb{E}[C(\hat{p}_{h})]\geq C(p_{h}) . The excess ( 1 − C ( p h ) ) / n h (1-C(p_{h}))/n_{h} is the summed multinomial variance and vanishes as n h → ∞ n_{h}\to\infty .
Homogeneous-tree illustration. If every context has a uniform K = 100 K=100 continuation distribution, each context has n h = 1000 n_{h}=1000 independent observations, and continuation collision is identical across sibling branches, the local factor is 1 + 99 / 1000 = 1.099 1+99/1000=1.099 . Under these restrictive assumptions, 50 50 decisions yield the product 1.099 50 ≈ 112 1.099^{50}\approx 112 . This is an illustration, not a neural or generic sequence law.
###### Remark 1 (This is tabular, not neural) .