Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation — part 7/8

Cleaned-text code points [93625, 109461). Bibliography omitted here; source.md includes it.

We ignore additive constants throughout, as they change neither gradients nor minimizers. We also suppress any conditioning (such as a class label or timestep), as every statement below holds per condition. The loss between a generation y ^ \hat{y} and a datapoint x x is then − log ⁡ k σ ​ ( y ^ , x ) -\log k_{\sigma}(\hat{y},x) , where for squared error the kernel is a Gaussian centered on the generation, k σ ​ ( y ^ , x ) = 𝒩 ⁡ ( x , y ^ , σ 2 ​ I ) k_{\sigma}(\hat{y},x)=\mathcal{N}(x;\,\hat{y},\,\sigma^{2}I) . Blurring each side by this kernel gives the two densities of the overview, p θ = g θ ∗ k σ , p σ ∗ = p ∗ ∗ k σ , p_{\theta}=g_{\theta}*k_{\sigma},\qquad p^{*}_{\sigma}=p^{*}*k_{\sigma}, so that p θ p_{\theta} represents the model as the loss sees it, and p σ ∗ p^{*}_{\sigma} represents the data in the same way. Finally, we analyze the smooth form of exploration, which scores the K K explored candidates by their average kernel, − log 1 K ∑ i k σ ( y ^ i , x ) -\log\tfrac{1}{K}\sum_{i}k_{\sigma}(\hat{y}_{i},x) , rather than by the best one alone. This soft min differs from the hard min by at most log ⁡ K \log K , and we describe where the two diverge later in this subsection.

#### Proposition 1 (Forward XM is maximum likelihood).

At every K K , the smooth Forward XM objective (the soft counterpart of Equation 1 ) L F ( θ ) = 𝔼 x ∼ p ∗ 𝔼 y ^ 1 : K ∼ g θ [ − log 1 K ∑ i k σ ( y ^ i , x ) ] L_{F}(\theta)=\mathbb{E}_{x\sim p^{*}}\,\mathbb{E}_{\hat{y}_{1:K}\sim g_{\theta}}\!\left[-\log\tfrac{1}{K}\textstyle\sum_{i}k_{\sigma}(\hat{y}_{i},x)\right] is exactly the expected negative log-likelihood of the data under the mixture of the model’s own K K explored generations, 1 K ​ ∑ i k σ ​ ( y ^ i , ⋅ ) \tfrac{1}{K}\sum_{i}k_{\sigma}(\hat{y}_{i},\cdot) , so Forward XM performs maximum likelihood at every K K . As K → ∞ K\to\infty this mixture converges to the blurred model p θ p_{\theta} , so minimizing L F L_{F} becomes minimizing KL ( p ∗ ∥ p θ ) \mathrm{KL}(p^{*}\,\|\,p_{\theta}) , recovering p ∗ p^{*} as σ → 0 \sigma\to 0 .

###### Proof sketch.

For a fixed draw of candidates, 1 K ​ ∑ i k σ ​ ( y ^ i , ⋅ ) \tfrac{1}{K}\sum_{i}k_{\sigma}(\hat{y}_{i},\cdot) is a normalized density, so the inner term is exactly the negative log-likelihood of x x under this mixture, and subtracting the constant data entropy H ⁡ ( p ∗ ) H(p^{*}) leaves the forward KL to the mixture. The same average is an unbiased estimate of p θ ​ ( x ) p_{\theta}(x) , so by Jensen L F L_{F} upper-bounds 𝔼 p ∗ ​ [ − log ⁡ p θ ] \mathbb{E}_{p^{*}}[-\log p_{\theta}] for every K K , tightening as K K grows by the argument of IWAE [ 104 ] and converging by the law of large numbers (with domination), where 𝔼 p ∗ [ − log p θ ] = KL ( p ∗ ∥ p θ ) + H ( p ∗ ) \mathbb{E}_{p^{*}}[-\log p_{\theta}]=\mathrm{KL}(p^{*}\,\|\,p_{\theta})+H(p^{*}) . ∎

#### Remark (the optimum at fixed σ \sigma ).

Over all generator densities, the optimum is the KL projection of p ∗ p^{*} onto the blurred family { g ∗ k σ } \{g*k_{\sigma}\} ; it equals p ∗ p^{*} exactly when the σ \sigma -deconvolution of p ∗ p^{*} exists as a density, and in general the match becomes exact only as σ → 0 \sigma\to 0 .

#### The role of K K .

Since Forward XM performs maximum likelihood at every K K , what K K controls is the density each draw fits. At K = 1 K{=}1 this density is a single Gaussian of width σ \sigma , whose best fit (for squared error) is a point mass at the data mean, the blurred mean. At larger K K it is a K K -component mixture that can place mass on many modes, becoming the full blurred model p θ p_{\theta} as K → ∞ K\to\infty . At inference, however, the model draws a single sample from g θ g_{\theta} , which the loss scores only through its blur p θ p_{\theta} ; from that fixed density’s view, L F L_{F} is an upper bound on its negative log-likelihood that tightens monotonically as K K grows [ 104 ] . The gap between the two views is a Jensen gap: because the mixture’s components are sampled from g θ g_{\theta} rather than placed freely, finite K K penalizes draws that miss a mode. When per-draw scores concentrate this gap is O ⁡ ( 1 / K ) O(1/K) and merely favors slightly lower-variance generators; with well-separated modes and few candidates, misses are costly enough that minimizers hedge mass between modes (Proposition 3), an effect that fades as K K outgrows the mode count. Exploration therefore turns a unimodal regressor into a multimodal likelihood model.

#### The hard min.

The hard min actually implemented differs from the soft one by at most log ⁡ K \log K , and the difference matters most in the limit. As K → ∞ K\to\infty , the hard objective only asks that every datapoint have some generation arbitrarily close to it, so any model whose generations cover the data’s support becomes optimal, regardless of how it spreads mass over that support. The hard min is therefore a coverage objective, which shares the mode-covering character of the forward KL without pinning down the density, and the clean likelihood statements are those of the smooth form above.

#### Proposition 2 (Reverse XM targets the reverse KL, but collapses without an entropy term).

At every K K , the smooth Reverse XM objective (the soft counterpart of Equation 2 ) L R ( θ ) = 𝔼 y ^ ∼ g θ 𝔼 x 1 : K ∼ p ∗ [ − log 1 K ∑ i k σ ( y ^ , x i ) ] L_{R}(\theta)=\mathbb{E}_{\hat{y}\sim g_{\theta}}\,\mathbb{E}_{x_{1:K}\sim p^{*}}\!\left[-\log\tfrac{1}{K}\textstyle\sum_{i}k_{\sigma}(\hat{y},x_{i})\right] equals, in expectation over the sampled targets, KL ( g θ ∥ p ^ σ ) + H ( g θ ) \mathrm{KL}(g_{\theta}\,\|\,\hat{p}_{\sigma})+H(g_{\theta}) , where p ^ σ = 1 K ​ ∑ i k σ ​ ( ⋅ , x i ) \hat{p}_{\sigma}=\tfrac{1}{K}\sum_{i}k_{\sigma}(\cdot,x_{i}) is the kernel density estimate of the K K data targets, converging to KL ( g θ ∥ p σ ∗ ) + H ( g θ ) \mathrm{KL}(g_{\theta}\,\|\,p^{*}_{\sigma})+H(g_{\theta}) as K → ∞ K\to\infty . Because each generation is scored on its own, independently of how often the model produces it, L R L_{R} is linear in g θ g_{\theta} , so its infimum is approached by collapse onto a point mass (at K = 1 K{=}1 , the data mean) and bare Reverse XM does not recover p ∗ p^{*} . Adding an entropy bonus, i.e. minimizing L R − H ⁡ ( g θ ) L_{R}-H(g_{\theta}) , cancels the entropy term and leaves the pure reverse KL, minimized as K → ∞ K\to\infty at g θ = p σ ∗ g_{\theta}=p^{*}_{\sigma} , reaching p ∗ p^{*} as σ → 0 \sigma\to 0 .

###### Proof sketch.

For a fixed draw of targets, p ^ σ \hat{p}_{\sigma} is a normalized density, so the inner expectation is the cross-entropy between g θ g_{\theta} and p ^ σ \hat{p}_{\sigma} , which decomposes as KL ( g θ ∥ p ^ σ ) + H ( g θ ) \mathrm{KL}(g_{\theta}\,\|\,\hat{p}_{\sigma})+H(g_{\theta}) ; by the law of large numbers p ^ σ ​ ( y ^ ) → p σ ∗ ​ ( y ^ ) \hat{p}_{\sigma}(\hat{y})\to p^{*}_{\sigma}(\hat{y}) , giving the limit. The objective weights each generation by a score that does not depend on how often the model produces it, so it is linear in g θ g_{\theta} , and its infimum over densities is approached by densities concentrating toward a point mass at the minimizer of the per-generation cost (the data mean at K = 1 K{=}1 , a global mode of p σ ∗ p^{*}_{\sigma} as K K grows). Subtracting H ⁡ ( g θ ) H(g_{\theta}) in the limit leaves the reverse KL, minimized at g θ = p σ ∗ g_{\theta}=p^{*}_{\sigma} . ∎

#### Where the blur sits.

One further asymmetry is where the blur sits. Forward XM compares the raw data against the blurred model, whereas Reverse XM compares the raw model against the blurred data, and as σ → 0 \sigma\to 0 both become comparisons between the raw model and the raw data, in opposite directions.

#### On the ELBO.

Only the Forward side carries a genuine evidence bound, as the inner log ⁡ 1 K ​ ∑ i k σ ​ ( y ^ i , x ) \log\tfrac{1}{K}\sum_{i}k_{\sigma}(\hat{y}_{i},x) is, in expectation, an importance-weighted lower bound on the model log-likelihood log ⁡ p θ ​ ( x ) \log p_{\theta}(x) , as in IWAE [ 104 ] . The entropy-corrected Reverse objective L R − H ⁡ ( g θ ) L_{R}-H(g_{\theta}) is instead a variational free energy for the target p σ ∗ p^{*}_{\sigma} as K → ∞ K\to\infty , while bare L R L_{R} keeps only its energy term, which is why it collapses and bounds no model likelihood. For this reason, we describe Reverse XM as reverse-KL or variational rather than as an ELBO.

#### Assumptions.

These statements require (i) the loss to be, up to an additive constant, the negative log of a kernel whose normalizer is independent of y ^ \hat{y} and θ \theta (squared error ↔ \leftrightarrow Gaussian; for non-normalizable losses such as bounded or perceptual ones the reading fails and these statements do not apply), (ii) the smooth surrogate rather than the hard min, (iii) K → ∞ K\to\infty only to replace the K K -sample mixtures with the blurred densities p θ p_{\theta} and p σ ∗ p^{*}_{\sigma} , and σ → 0 \sigma\to 0 to reach p ∗ p^{*} exactly, and (iv) mild regularity: a bounded, symmetric, translation-invariant kernel (so p σ ∗ p^{*}_{\sigma} is a density), finite expected loss and differential entropies, domination to justify the limit interchanges, a continuous loss vanishing only at y ^ = x \hat{y}=x (for the hard-min limit), and minimizers ranging over all densities. Finally, these statements analyze end-to-end XMs, but because conditioning is suppressed throughout, they extend to any single prediction of a factored model whose K K explored candidates share their conditioning and target, with p ∗ p^{*} read as that prediction’s target conditional. For discrete predictions the kernel is the candidate’s own softmax, so Proposition 1’s mixture-likelihood reading carries over with no blur ( σ \sigma plays no role), which covers our XMDLMs; the separation-based Proposition 3 and the kernel-density reading of Proposition 2 are squared-error statements and do not transfer. The noise-searching form of our continuous hybrids instead pairs each candidate with its own corruption of the datapoint, a coupling search rather than a best-of- K K against a fixed target, and we leave its analysis, along with how per-prediction gains compose across a sampler’s steps, to future work.

#### Proposition 3 (Expressivity of smooth Forward XM under separation).

Suppose p ∗ p^{*} is a mixture of M ∗ ≤ K M^{*}\leq K unimodal components with near-equal masses, each concentrated at scale σ \sigma or below, whose modes are pairwise separated by distances Δ ≫ σ \Delta\gg\sigma , with Δ 2 / σ 2 ≫ ( log ⁡ K ) / min m ⁡ w m \Delta^{2}/\sigma^{2}\gg(\log K)/\min_{m}w_{m} for component masses w m w_{m} , and let the assumptions of this section hold. Then every minimizer g θ g_{\theta} of the smooth Forward XM objective L F L_{F} places positive mass in a neighborhood of each mode of p ∗ p^{*} —so M ⁡ ( g θ ) ≥ M ∗ M(g_{\theta})\geq M^{*} , and taking M ∗ = K M^{*}=K gives E ≥ K E\geq K —though at finite K K minimizers may also hedge mass between modes, as insurance against candidate draws that miss a mode. Once candidates sufficiently outnumber modes, K ​ ( 1 − min m ⁡ w m ) K ​ Δ 2 / σ 2 ≲ 1 K(1-\min_{m}w_{m})^{K}\,\Delta^{2}/\sigma^{2}\lesssim 1 , misses are rare enough that hedging no longer pays: all but a negligible fraction of mass sits at the modes, and the blurred density p θ = g θ ∗ k σ p_{\theta}=g_{\theta}*k_{\sigma} has exactly M ∗ M^{*} strict local maxima, one per component—equivalently, what the model deploys, g θ g_{\theta} , carries exactly M ∗ M^{*} modes read at the loss scale.

###### Proof sketch.

By Proposition 1, minimizing L F L_{F} is maximum likelihood of p ∗ p^{*} under the mixture of K K candidates drawn i.i.d. from g θ g_{\theta} . If g θ g_{\theta} assigns a mode zero mass, no draw ever covers it, and the objective then pays of order w m ​ Δ 2 / σ 2 w_{m}\Delta^{2}/\sigma^{2} for a mode of mass w m w_{m} , while concentrating that mass elsewhere gains at most log ⁡ K \log K in likelihood, plus what it saves by making misses rarer at the remaining modes—with near-equal masses a constant fraction of what the dropped mode costs—so neither term covers the loss, and every minimizer covers every mode with positive mass. The same log ⁡ K \log K budget also forces concentration: spreading mode m m ’s share to width ρ \rho costs its covered datapoints of order w m ​ ρ 2 / σ 2 w_{m}\rho^{2}/\sigma^{2} against a total possible gain of log ⁡ K \log K , so under the separation condition each mode’s mass sits within o ⁡ ( Δ ) o(\Delta) of its center. Each region’s peak density therefore dominates the between-mode bridges, whose height is capped by the same budget, and hedge mass—lying at distance ∼ Δ \sim\Delta from the peaks at lower density—leaves each peak a strict local maximum, hence M ⁡ ( g θ ) ≥ M ∗ M(g_{\theta})\geq M^{*} ; concentration to O ⁡ ( σ ) O(\sigma) and the exact count M ∗ M^{*} require the rare-miss condition below. Because candidates are sampled rather than placed, a draw misses a mode of weight w w with probability ( 1 − w ) K (1-w)^{K} , at a cost of order Δ 2 / σ 2 \Delta^{2}/\sigma^{2} , so minimizers also keep insurance mass between modes—the bridges visible at small K K in Figure 2 —which likewise perturbs the optimal mode weights. Enumerating K K learned discrete latents instead puts a candidate at every mode on every step, so misses never occur and the bridges vanish at any K K , though K K latents alone then cap how many distinct outputs a condition can produce (it’s possible a mixture of discrete and continuous is optimal). Under the additional condition, misses are rare enough that insurance costs more in diluted likelihood than it saves, so all but a negligible fraction of mass concentrates in the mode neighborhoods (the concentration hypothesis leaves exponentially little of p ∗ p^{*} elsewhere). Convolving with k σ k_{\sigma} then merges each neighborhood’s sub- σ \sigma structure into a single bump, with exponentially small cross-terms and no spurious maxima between components, so p θ p_{\theta} has exactly M ∗ M^{*} strict local maxima. ∎

## Appendix G Frequently Asked Questions

Here we answer some common questions about Explorative Modeling.

#### What’s the main takeaway? How is this paper ‘novel’ if it’s just best-of- K K ?

Introducing best-of- K K is not the central contribution or claim of this paper, as sampling candidates and keeping the best has appeared many times before [ 33 , 34 , 35 ] (Section E ). What is new is the realization of what this simple loop does: it increases generative expressivity without factoring generation, which is a completely different factorization axis from popular generative models such as Autoregression and Diffusion (Figure 1 ). The paper’s central messages follow from this realization: generative expressivity is worth scaling through exploration alongside model parameters and data as a new pretraining axis, as it improves both the performance and generalization of existing generative models (a way of trading training compute for generalization). Because generative expressivity increasingly becomes the bottleneck as parameters and data grow, gains from exploration grow with scale rather than saturate, making exploration more important as scale increases. Exploration can also stand in for factoring generation, turning how end-to-end a model is into something we can scale rather than a fixed design choice, all the way to fully end-to-end XMs that match diffusion baselines at a fraction of the inference compute. Finally, best-of- K K is just one implementation of exploration—we have already seen other implementations succeed (Reverse XM, Section A ), and expect more versions to eventually become viable (e.g., gradient-based search, Section 6 ).

#### Does exploration make inference more expensive?