Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs — part 6/7
Cleaned-text code points [77534, 93310). Bibliography omitted here; source.md includes it.
The identity above is exact for a per-context empirical MLE. A transformer shares parameters across contexts, so its conditional need not have multinomial variance and no effective-count substitution restores this formula in general. The neural experiments therefore use Eq. ( 6 ) and Theorem 1 instead.
### I.2 Amplification over sequences
For variable-length autoregressive models, sequence collision is not a simple product of probabilities over fixed positions, due to the dependence of contexts on sampled prefixes. Analysis of these models therefore requires a branching recursive framework. C p ( h ) = ∑ a p ( a ∣ h ) 2 C p ( h a ) , C_{p}(h)=\sum_{a}p(a\mid h)^{2}\,C_{p}(ha), (33) Each summand is the probability that two draws both choose branch a a and then collide from h a ha . Thus the collision-biased next-token weight is proportional to p ( a ∣ h ) 2 C p ( h a ) p(a\mid h)^{2}C_{p}(ha) , not to p ( a ∣ h ) 2 p(a\mid h)^{2} alone; the recursion is generally a sum of branch products rather than a product of aggregate one-step and continuation collisions.
Let D D denote the random tabular dataset and let R p ( y ∣ x ) = p ( y ∣ x ) 2 / C p ( x ) R_{p}(y\mid x)=p(y\mid x)^{2}/C_{p}(x) be the exact-match collision-biased path distribution. For small local terms and approximately independent per-context estimates, a first-order expansion motivates the homogeneous-tree heuristic log 𝔼 D [ C p ^ ( x ) ] C p ( x ) ≈ 𝔼 Y ∼ R p ( ⋅ ∣ x ) [ ∑ t = 1 T ( Y ) + 1 N 2 ( p h t ) − 1 n h t ] , \log\frac{\mathbb{E}_{D}[C_{\hat{p}}(x)]}{C_{p}(x)}\approx\mathbb{E}_{Y\sim R_{p}(\cdot\mid x)}\!\left[\sum_{t=1}^{T(Y)+1}\frac{N_{2}(p_{h_{t}})-1}{n_{h_{t}}}\right], (34) This approximation additionally treats R p R_{p} as fixed and requires C p ( h a ) C_{p}(ha) to be approximately constant across sibling branches. Without those conditions, the R p R_{p} -average of local factors is not the expected sequence collision ratio.
### I.3 Entropy analogue
For a fixed finite support of size S S , the classical plug-in bias is 𝔼 [ H ( p ^ h ) ] = H ( p h ) − ( S − 1 ) / ( 2 n h ) + o ( n h − 1 ) \mathbb{E}[H(\hat{p}_{h})]=H(p_{h})-(S-1)/(2n_{h})+o(n_{h}^{-1}) under its regularity conditions ( Paninski, 2003 ; Jiao et al., 2015 ) . Replacing S S by exp H ( p h ) \exp H(p_{h}) or summing this local approximation along generated paths is heuristic; neither substitution is used in the empirical claims.
## Appendix J Limits of temperature scaling
Temperature can change a model’s collision value without generally calibrating its distribution. To make the global sequence transformation well-defined for every T > 0 T>0 , fix a distribution q ( ⋅ ∣ x ) q(\cdot\mid x) with finite support and write β = 1 / T \beta=1/T , q T ( y ∣ x ) = q ( y ∣ x ) β / Z β ( x ) q_{T}(y\mid x)=q(y\mid x)^{\beta}/Z_{\beta}(x) , and Z β ( x ) = ∑ y q ( y ∣ x ) β Z_{\beta}(x)=\sum_{y}q(y\mid x)^{\beta} .
###### Proposition 2 (Global temperature monotonically increases order-2 diversity) .
C T ( x ) = ∑ y q T ( y ∣ x ) 2 = Z 2 β ( x ) Z β ( x ) 2 = Z 2 / T ( x ) Z 1 / T ( x ) 2 . C_{T}(x)=\sum_{y}q_{T}(y\mid x)^{2}=\frac{Z_{2\beta}(x)}{Z_{\beta}(x)^{2}}=\frac{Z_{2/T}(x)}{Z_{1/T}(x)^{2}}. (35) Moreover, C T ( x ) C_{T}(x) is nonincreasing and N 2 T ( x ) = 1 / C T ( x ) N_{2}^{T}(x)=1/C_{T}(x) is nondecreasing in T T .
With q T ( y ) = q ( y ) β / Z β q_{T}(y)=q(y)^{\beta}/Z_{\beta} and Z β = ∑ y q ( y ) β Z_{\beta}=\sum_{y}q(y)^{\beta} , C T = ∑ y q T ( y ) 2 = ∑ y q ( y ) 2 β / Z β 2 = Z 2 β / Z β 2 C_{T}=\sum_{y}q_{T}(y)^{2}=\sum_{y}q(y)^{2\beta}/Z_{\beta}^{2}=Z_{2\beta}/Z_{\beta}^{2} . For monotonicity in T T , let ψ ( β ) = log Z β \psi(\beta)=\log Z_{\beta} . Then ψ ′ ( β ) = 𝔼 q β [ log q ] \psi^{\prime}(\beta)=\mathbb{E}_{q_{\beta}}[\log q] where q β ∝ q β q_{\beta}\propto q^{\beta} , and ψ ′′ ( β ) = Var q β ( log q ) ≥ 0 \psi^{\prime\prime}(\beta)=\Var_{q_{\beta}}(\log q)\geq 0 , so ψ ′ \psi^{\prime} is nondecreasing. Since log C T = ψ ( 2 β ) − 2 ψ ( β ) \log C_{T}=\psi(2\beta)-2\psi(\beta) , d d β log C T = 2 [ ψ ′ ( 2 β ) − ψ ′ ( β ) ] ≥ 0 \frac{d}{d\beta}\log C_{T}=2\big[\psi^{\prime}(2\beta)-\psi^{\prime}(\beta)\big]\geq 0 for β > 0 \beta>0 . Thus C T C_{T} is nondecreasing in β = 1 / T \beta=1/T , i.e. nonincreasing in T T , with equality only in the degenerate cases where log q \log q is q β q_{\beta} -a.s. constant (uniform or point mass); equivalently the order-2 diversity N 2 T ( x ) = 1 / C T ( x ) N_{2}^{T}(x)=1/C_{T}(x) is nondecreasing in T T . ∎The reachable values depend only on the fixed probability profile of q q .
### J.1 The temperature-reachable diversity interval
Raising global sequence temperature increases diversity, but only within a fixed interval determined by the model distribution. For a given prompt x x , let S ( x ) S(x) denote the number of sequences with non-zero probability under q q given x x ( S ( x ) := | supp q ( ⋅ ∣ x ) | S(x):=|\mathrm{supp}\,q(\cdot\mid x)| ), and let g ⋆ ( x ) g_{\star}(x) denote the number of modes of q q given x x ( g ⋆ ( x ) := | arg max y q ( y ∣ x ) | g_{\star}(x):=|\arg\max_{y}q(y\mid x)| ).
###### Lemma 1 (Temperature-reachable diversity interval) .
For a fixed model distribution q ( ⋅ ∣ x ) q(\cdot\mid x) with finite support, N 2 T ( x ) = Z β ( x ) 2 / Z 2 β ( x ) N_{2}^{T}(x)=Z_{\beta}(x)^{2}/Z_{2\beta}(x) is continuous and nondecreasing in T T , with limits lim T → 0 + N 2 T ( x ) = g ⋆ ( x ) , lim T → ∞ N 2 T ( x ) = S ( x ) . \lim_{T\to 0^{+}}N_{2}^{T}(x)=g_{\star}(x),\qquad\lim_{T\to\infty}N_{2}^{T}(x)=S(x). (36) Hence as T T ranges over ( 0 , ∞ ) (0,\infty) , N 2 T ( x ) N_{2}^{T}(x) takes every value in the open interval ( g ⋆ ( x ) , S ( x ) ) (\,g_{\star}(x),\,S(x)\,) ; the endpoints g ⋆ ( x ) g_{\star}(x) and S ( x ) S(x) are the limits as T → 0 + T\to 0^{+} and T → ∞ T\to\infty and are not attained at any finite T T except in degenerate cases ( q q already a point mass, resp. already uniform on its support). Its closure is [ g ⋆ ( x ) , S ( x ) ] [\,g_{\star}(x),\,S(x)\,] (generically [ 1 , S ( x ) ] [1,S(x)] ). One degenerate case also affects the lower endpoint: if q ( ⋅ ∣ x ) q(\cdot\mid x) is uniform on its entire support, then every token is a maximizer, so g ⋆ ( x ) = S ( x ) g_{\star}(x)=S(x) and N 2 T ( x ) = S ( x ) N_{2}^{T}(x)=S(x) at every finite T T ; the reachable interval then collapses to the single point { S ( x ) } \{S(x)\} . The “point mass / uniform limit” description above is thus the generic non-degenerate picture, but is incomplete for this collapsed (uniform- q q ) case.
Monotonicity is the previous result ( C T C_{T} nonincreasing in T T , so N 2 T = 1 / C T N_{2}^{T}=1/C_{T} nondecreasing). For T → 0 + T\to 0^{+} ( β → ∞ \beta\to\infty ): writing q max = max y q ( y ) q_{\max}=\max_{y}q(y) and g ⋆ g_{\star} for the number of maximizers, Z β = g ⋆ q max β ( 1 + o ( 1 ) ) Z_{\beta}=g_{\star}q_{\max}^{\beta}(1+o(1)) and Z 2 β = g ⋆ q max 2 β ( 1 + o ( 1 ) ) Z_{2\beta}=g_{\star}q_{\max}^{2\beta}(1+o(1)) , so N 2 T = Z β 2 / Z 2 β → g ⋆ 2 q max 2 β / ( g ⋆ q max 2 β ) = g ⋆ N_{2}^{T}=Z_{\beta}^{2}/Z_{2\beta}\to g_{\star}^{2}q_{\max}^{2\beta}/(g_{\star}q_{\max}^{2\beta})=g_{\star} . For T → ∞ T\to\infty ( β → 0 \beta\to 0 ): q ( y ) β → 𝟏 { q ( y ) > 0 } q(y)^{\beta}\to\mathbf{1}\{q(y)>0\} , so Z β → S Z_{\beta}\to S and Z 2 β → S Z_{2\beta}\to S , giving N 2 T → S 2 / S = S N_{2}^{T}\to S^{2}/S=S . Continuity in T T and the intermediate value theorem give every value in the open interval; the endpoints are limits, attained only in the stated degenerate cases. ∎
When S ( x ) > g ⋆ ( x ) S(x)>g_{\star}(x) , the fraction of this interval reached at T T is ρ T ( x ) := N 2 T ( x ) − g ⋆ ( x ) S ( x ) − g ⋆ ( x ) ∈ [ 0 , 1 ] , \rho_{T}(x):=\frac{N_{2}^{T}(x)-g_{\star}(x)}{S(x)-g_{\star}(x)}\in[0,1], (37) which is a normalized diversity location, not a temperature parameter. It is undefined when the interval collapses.
###### Theorem 2 (Temperature only redistributes existing support) .
Assume a finite candidate space (e.g. a finite vocabulary together with a maximum sequence length), so that S ( x ) = | supp q ( ⋅ ∣ x ) | < ∞ S(x)=|\mathrm{supp}\,q(\cdot\mid x)|<\infty . For every temperature T > 0 T>0 the temperature-scaled model q T q_{T} has the same support as q q , and its diversity is bounded by that support size: N 2 T ( x ) ≤ S ( x ) , sup T > 0 N 2 T ( x ) = S ( x ) , N_{2}^{T}(x)\leq S(x),\qquad\sup_{T>0}N_{2}^{T}(x)=S(x), (38) the supremum corresponding to the uniform distribution on supp q ( ⋅ ∣ x ) \mathrm{supp}\,q(\cdot\mid x) and approached only as T → ∞ T\to\infty (not attained at any finite T T unless q q is already uniform on its support). Consequently, relative to the true conditional p ( ⋅ ∣ x ) p(\cdot\mid x) : (i) if N 2 ( p ( ⋅ ∣ x ) ) > S ( x ) N_{2}\big(p(\cdot\mid x)\big)>S(x) , no temperature reaches the true diversity—the model’s support is simply too small;
(ii) if N 2 ( p ( ⋅ ∣ x ) ) N_{2}\big(p(\cdot\mid x)\big) lies in the open interval ( g ⋆ ( x ) , S ( x ) ) (g_{\star}(x),S(x)) , some finite temperature attains the true diversity value by the intermediate value theorem; the boundary values g ⋆ ( x ) g_{\star}(x) and S ( x ) S(x) are only approached as T → 0 + T\to 0^{+} and T → ∞ T\to\infty and are not attained at any finite T T except in the degenerate cases of the interval Lemma; but
(iii) attaining that value does not reproduce p p : q T = p q_{T}=p holds only if p ( ⋅ ∣ x ) ∝ q ( ⋅ ∣ x ) β p(\cdot\mid x)\propto q(\cdot\mid x)^{\beta} for some β > 0 \beta>0 .
Temperature scaling preserves support: q T ( y ) > 0 ⇔ q ( y ) > 0 q_{T}(y)>0\iff q(y)>0 for every finite T > 0 T>0 , so q T q_{T} is supported on a set of size S ( x ) S(x) . For any distribution r r over S S outcomes, Cauchy–Schwarz gives 1 = ( ∑ a r ( a ) ) 2 ≤ S ∑ a r ( a ) 2 1=\big(\sum_{a}r(a)\big)^{2}\leq S\sum_{a}r(a)^{2} , i.e. N 2 ( r ) = 1 / ∑ a r ( a ) 2 ≤ S N_{2}(r)=1/\sum_{a}r(a)^{2}\leq S , with equality iff r r is uniform on its support. Applying this to r = q T r=q_{T} yields N 2 T ( x ) ≤ S ( x ) N_{2}^{T}(x)\leq S(x) ; the bound is approached as q T → Unif ( supp q ) q_{T}\to\mathrm{Unif}(\mathrm{supp}\,q) when T → ∞ T\to\infty and is not attained at finite T T unless q q is already uniform on its support. This gives the displayed bounds and case (i): if N 2 ( p ) > S ( x ) N_{2}(p)>S(x) , no q T q_{T} can reach it. For case (ii), N 2 T ( x ) N_{2}^{T}(x) is continuous in T T and takes every value in the open interval ( g ⋆ ( x ) , S ( x ) ) (g_{\star}(x),S(x)) (Lemma), so whenever N 2 ( p ) N_{2}(p) lies in this open interval the intermediate value theorem supplies a finite T T with N 2 T ( x ) = N 2 ( p ) N_{2}^{T}(x)=N_{2}(p) ; the endpoints are limits, not attained at finite T T outside the degenerate cases. For case (iii), { q T : T > 0 } \{q_{T}:T>0\} is the one-parameter family log q T ( y ) = β log q ( y ) − log Z β \log q_{T}(y)=\beta\log q(y)-\log Z_{\beta} ; matching p p exactly requires log p ( y ) = β log q ( y ) − log Z β \log p(y)=\beta\log q(y)-\log Z_{\beta} for all y y , i.e. p ∝ q β p\propto q^{\beta} . ∎
Interpretation. Temperature preserves support and moves the global sequence distribution from the uniform distribution over its maximizers as T → 0 + T\to 0^{+} toward the uniform distribution over its support as T → ∞ T\to\infty . It can therefore match a target collision value while missing the target distribution, and the high-temperature limit recovers the target only when the target itself is uniform on the model’s support.
###### Lemma 2 (Temperature scaling is an order-preserving one-parameter family) .
Fix a prompt x x and write q T ( ⋅ ∣ x ) ∝ q ( ⋅ ∣ x ) 1 / T q_{T}(\cdot\mid x)\propto q(\cdot\mid x)^{1/T} . (i) Log-odds identity. For every T > 0 T>0 and any outcomes a , b a,b with q ( b ∣ x ) > 0 q(b\mid x)>0 , log q T ( a ∣ x ) q T ( b ∣ x ) = 1 T log q ( a ∣ x ) q ( b ∣ x ) . \log\frac{q_{T}(a\mid x)}{q_{T}(b\mid x)}=\frac{1}{T}\,\log\frac{q(a\mid x)}{q(b\mid x)}. (39)
(ii) Order preservation. Hence temperature scaling is an order-preserving reweighting: the rank ordering of { q T ( y ∣ x ) } y \{q_{T}(y\mid x)\}_{y} is the same for all T > 0 T>0 . The reachable set is a one-parameter curve from the uniform distribution over arg max y q ( y ∣ x ) \argmax_{y}q(y\mid x) as T → 0 + T\to 0^{+} to the uniform distribution over supp q ( ⋅ ∣ x ) \operatorname{supp}q(\cdot\mid x) as T → ∞ T\to\infty .
(iii) Unreachability. Consequently q T = p ( ⋅ ∣ x ) q_{T}=p(\cdot\mid x) for some T > 0 T>0 only if p ( ⋅ ∣ x ) ∝ q ( ⋅ ∣ x ) β p(\cdot\mid x)\propto q(\cdot\mid x)^{\beta} , i.e. log p \log p is an affine function of log q \log q . In particular, if the rank ordering of p ( ⋅ ∣ x ) p(\cdot\mid x) differs from that of q ( ⋅ ∣ x ) q(\cdot\mid x) , no temperature matches p p .
For (i), q T ( y ) = q ( y ) 1 / T / Z 1 / T q_{T}(y)=q(y)^{1/T}/Z_{1/T} , so the normalizer cancels in the ratio and q T ( a ) / q T ( b ) = ( q ( a ) / q ( b ) ) 1 / T q_{T}(a)/q_{T}(b)=(q(a)/q(b))^{1/T} ; take logarithms. For (ii), u ↦ u 1 / T u\mapsto u^{1/T} is strictly increasing on ( 0 , ∞ ) (0,\infty) , so q T ( a ) ≥ q T ( b ) q_{T}(a)\geq q_{T}(b) exactly when q ( a ) ≥ q ( b ) q(a)\geq q(b) ; this common ordering holds for all T T , with the two endpoints given by the interval Lemma above. For (iii), a fixed ordering rules out any q T q_{T} with a different one, while q T = p q_{T}=p forces p ∝ q 1 / T p\propto q^{1/T} . ∎
Temperature rescales every log-odds ratio by the same factor. It can flatten or sharpen relative-probability errors, but it cannot reorder outcomes or independently correct them. Figure 8 gives a six-outcome example.
### J.2 Effective support
The finite-support Theorem 2 requires a finite candidate space to ensure S ( x ) < ∞ S(x)<\infty . While standard softmax assigns strictly positive probability to all tokens and, subject to a length constraint, to most sequences, the Theorem’s condition of finite support may be violated without a maximum length, rendering the claim about temperature’s inability to recover missing support inapplicable. A more broadly applicable statement is probabilistic: rare, valid continuations are not entirely absent from q T q_{T} , but are effectively absent when sampling from a feasible number of samples.
###### Proposition 3 (Effective absence of valid missing mass under sampling) .
Fix a prompt x x and consider a sequence of fitted models q n q_{n} indexed by dataset size n n , with temperature-scaled tail mass ε n , T ( x ) = q n , T ( U ( x ) ∣ x ) \varepsilon_{n,T}(x)=q_{n,T}(U(x)\mid x) on a set U ( x ) U(x) of valid but unobserved or severely underweighted continuations. Drawing M n M_{n} independent generations from q n , T ( ⋅ ∣ x ) q_{n,T}(\cdot\mid x) , Pr ( at least one of the M n samples lands in U ( x ) ) = 1 − ( 1 − ε n , T ( x ) ) M n ≤ M n ε n , T ( x ) . \Pr\big(\text{at least one of the }M_{n}\text{ samples lands in }U(x)\big)=1-\big(1-\varepsilon_{n,T}(x)\big)^{M_{n}}\leq M_{n}\,\varepsilon_{n,T}(x). (40) Hence if ε n , T ( x ) → 0 \varepsilon_{n,T}(x)\to 0 and M n ε n , T ( x ) → 0 M_{n}\varepsilon_{n,T}(x)\to 0 along a sequence of fitted models, these continuations are effectively absent from the generated samples. This is an assumed asymptotic regime over models, not a consequence of changing T T at fixed q n q_{n} .