Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation — part 2/8

Cleaned-text code points [14794, 30792). Bibliography omitted here; source.md includes it.

However, factoring generation does not remove this generative expressivity limitation entirely, as even highly scalable generative modeling approaches, such as diffusion and autoregression, can leave modes uncaptured when single predictions inside their factored procedure face many valid targets at once—that is, when the generative expressivity of those single predictions is too low. Notably, this phenomenon worsens with scale: as we add parameters and data, model expressivity and what models can learn cease to be the limiting factors, and generative expressivity increasingly becomes the bottleneck (we show this in Section 4 ). Evidence for this already exists in how heavily today’s models lean on guidance. Classifier-free guidance [ 25 ] sharpens samples by pushing them away from the unconditional model, which, with less conditioning to pin down each prediction, blurs modes more than the conditional model. This extrapolation helps primarily because the original model itself blurs modes. Autoguidance [ 26 ] reinforces this perspective even further, improving samples by pushing away from a deliberately worse, more mode-collapsed version of the model, demonstrating that the core functionality of guidance is to push away from the conditional mean due to challenges in capturing the true distribution. Together, this evidence points to the idea that even today’s best generative models may improve from added generative expressivity, which we confirm in Section 4 .

### 2.2 End-to-End Generation

Increasing Exploration K K → \rightarrow Increases Generative Expressivity

We call a generative model end-to-end when it samples the same way during training and inference, so it is never exposed to inputs at inference that it was not trained on. 4 4 4 This concerns the model’s own generation procedure, not test-time distribution shift—an end-to-end model can still face out-of-distribution test data, which is the ordinary generalization problem. This is worth seeking for the same reason it has reshaped the rest of deep learning: ever since AlexNet [ 1 ] , learning a task in an end-to-end manner has continued to beat splitting tasks into hand-designed stages. In generative modeling, this means learning the entire mapping from noise to data, including both its representations and its trajectory, rather than hand-specifying any part of it. End-to-end generative modeling also inherits the practical benefits of end-to-end training, where it removes exposure bias [ 8 , 9 , 10 , 11 , 12 ] and the train-inference mismatch that drifts a model onto out-of-distribution inputs and compounds errors [ 5 , 6 , 7 ] , while as a byproduct enabling inference to be more efficient.

Yet general end-to-end reconstructive generation remains unsolved. 5 5 5 Contrastive models such as GANs [ 20 ] and CD EBMs [ 21 , 22 ] have been end-to-end for a long time, but as noted in Section 2.1 they have struggled to scale. Existing scalable reconstructive models capture modes by factoring generation into many steps, which makes training and inference inherently different—a diffusion model trained on a single denoising step is unrolled over hundreds of them at inference. Although recent methods push inference down to a single step [ 14 , 15 ] , during training they still anchor each prediction to the multi-step trajectory—conditioning on noised states or tying to the flow field—so that targets stay near-unimodal and avoid blurring. Their training therefore rarely simulates the one-step sampling they use at inference, so the train-inference mismatch remains and they are still not end-to-end.

### 2.3 Existing Generative Models

To study Explorative Modeling as a new scaling axis for existing generative models, our experiments build on two main families of generative models. First, we build on top of Diffusion and Flow Matching models [ 13 , 23 ] which generate by repeatedly denoising via small steps from noise to data. Because Diffusion and Flow are equivalent formulations [ 28 ] , we use the terms interchangeably throughout, and every Diffusion model over continuous data in this work is trained with the Flow Matching objective, as Flow has generally performed best [ 29 ] . Second, we experiment with Jumpy generative models [ 30 ] , which generalize the idea of Diffusion/Flow by varying the number of steps, or jumps , interpolating between direct end-to-end regression (a single jump, the most end-to-end) and continuous-time flow (infinitely many jumps). This interpolation via the number of jumps enables a tradeoff between how end-to-end models are (fewer jumps) and how generatively expressive models are (more jumps). We return to this tradeoff when measuring how different models scale with exploration (Section 4 ).

## 3 Explorative Modeling Approach

### 3.1 Explorative Modeling Intuition

The goal of all reconstructive generative models is to design a training objective such that the loss minimizer captures modes instead of averaging them. The reason for this goal is demonstrated in Figure 2 for XM-1, where performing end-to-end direct regression with a naive training objective generates samples that do not belong to the data distribution. Existing generative models achieve this goal by factoring the generation procedure into a sequence of smaller steps, keeping each step’s target nearly unimodal so that no single prediction is forced to average across modes (Section 2.1 ). While these approaches have resulted in highly performant large-scale generative models [ 31 , 32 ] , they prevent models from being end-to-end (described in Section 2.2 ), which directly hurts performance and generalization due to exposure bias [ 8 , 9 , 10 , 11 , 12 ] . Therefore, instead of factorizing the generation procedure, the goal of Explorative Modeling is to enable a new factorization axis —the training loop itself (Figure 1 ).

In their simplest form, XMs are just best-of- K K , an idea that has appeared many times in prior work [ 33 , 34 , 35 ] (Section E ). At each training step, the model generates K K candidate samples and trains on only the one closest to the data, implemented as a simple for loop (Algorithm 1 ) where only the best generation receives gradients. Formally, for a data sample x x , generations y ^ 1 , … , y ^ K ∼ G θ \hat{y}_{1},\ldots,\hat{y}_{K}\sim G_{\theta} , and a reconstruction loss J J such as squared error, the objective is ℒ ⁡ ( θ ) = min i ∈ { 1 , … , K } ⁡ J ⁡ ( y ^ i , x ) . \mathcal{L}(\theta)=\min_{i\in\{1,\ldots,K\}}J(\hat{y}_{i},x). (1) Intuitively, the purpose of this for loop is to change the loss minimizer from the mean of the data samples toward the true data samples themselves . This matters because for most data, the mean of samples is not on the data manifold, as demonstrated in Figure 2 . When the loss minimizer is the true data samples, models generate data that looks like real samples instead of a blurred, off-manifold average. Throughout this section we describe XMs as standalone models for intuition, though exploration can be added on top of existing generative models (Section 4.1 ).

There are several intuitions for what XMs are doing to enable handling multimodal distributions:

#### Scalable Training of Latent Variable Models

Explorative Modeling can be seen as conditioning the generator on a latent variable (e.g., noise for diffusion models, or a learned embedding for language models) that resolves which of the many valid targets an input maps to: conditioned on the right latent, the one-to-many coupling (Section 2.1 ) becomes one-to-one, so the target becomes unimodal, and the blur disappears. The challenge with latent variable models is that we do not know which latent goes with which datapoint in advance. Variational Autoencoders (VAEs) [ 36 ] learn an encoder to infer this latent, which adds a KL term and risks posterior collapse, and consequently VAEs have struggled to scale well as standalone generative models [ 37 , 38 ] . Explorative Modeling instead recovers the pairing through exploration , exploring possible matches between what it generates and the data and training on the closest. This trades extra training compute for end-to-end search of the latent variables.

#### A Scalable Way to Resolve Coupling

Reconstructive models must pair each input with a target, referred to as a coupling , which we do not know in advance. Pairing at random is what ties one input to many targets, which blurs them (Section 2.1 ). Computing a better coupling directly does not scale: exact optimal transport is cubic in the number of samples, and its minibatch approximations [ 39 ] only match within a batch, a biased proxy for the true global pairing [ 40 ] . Instead of computing a global or minibatch coupling, Explorative Modeling searches for a coupling aligned with the model’s own samples. Because each pairing keeps only the best match rather than forcing an assignment over a whole batch, and searches the model’s own samples rather than fixed noise, the coupling avoids minibatch OT’s bias and co-adapts with the model throughout training.

#### Generative Modeling via Search

Explorative Modeling can be seen as recasting generative modeling as a search problem, looking at training time for the latent, or coupling, that best explains the data. This framing is promising, as search and learning are the two methods the bitter lesson identifies as scaling well with computation [ 41 ] .

#### Spreading Predictions Across Modes

Geometrically, exploration changes what the best prediction strategy is. Consider guessing where darts land on a dartboard: with a single guess, the loss minimizer is the mean of all the throws, which is often a spot where few darts actually land. Forward XM (Equation 1 ) instead makes K K guesses and scores only the closest, so the mean of the dartboard is no longer the loss minimizer—the best strategy becomes spreading the guesses so each covers a different cluster of throws. For the model, this means different latents specialize to different modes, so larger K K captures more modes instead of blurring them together (Figure 2 ).

#### Minimizing an Implicit Energy

Exploration can also be seen as implicit training-time energy minimization, where the loss acts as an implicit energy over pairings of a generation with the data, so searching for the lowest-loss match is a search for the minimum-energy, best-coupled samples. In this work, we search this landscape at random, but this search could instead be gradient-based (we describe this further in Section 6 ).

### 3.2 Forward and Reverse Explorative Modeling

Exploration can search in either of two directions, which differ in what is held fixed and what is searched over (Figure 3 ).

#### Forward XM.

Forward XM fixes a data target and explores its own generations: it draws K K candidates and trains on the one closest to the target, exactly the best-of- K K objective of Equation 1 (Algorithm 1 ), which we denote ℒ Forward \mathcal{L}_{\text{Forward}} . Because every datapoint pulls in its nearest generation, no part of the data is ignored, so Forward XM is mass-covering —it errs toward recall , covering the full distribution. The challenge with Forward XM is compute—each of the K K candidates is a separate generation, so covering more modes takes more forward passes.

#### Reverse XM.

Reverse XM fixes a generated model sample and searches the data: it draws a single sample y ^ ∼ G θ \hat{y}\sim G_{\theta} and trains it toward the closest of K K data targets x 1 , … , x K ∼ 𝒟 x_{1},\dots,x_{K}\sim\mathcal{D} (Algorithm 2 ), flipping the objective to ℒ Reverse ​ ( θ ) = min i ∈ { 1 , … , K } ⁡ J ⁡ ( y ^ , x i ) . \mathcal{L}_{\text{Reverse}}(\theta)=\min_{i\in\{1,\dots,K\}}J(\hat{y},x_{i}). (2) Each generation is pulled onto the data manifold, so Reverse XM errs toward precision . Reverse XM is also cheap because it searches over data rather than generations, so each loss calculation only costs a single generation no matter how many targets it is compared against, which is useful for the large K K values needed to handle highly multimodal data. Reverse XM’s weakness is that searching from the generation side applies no pressure to cover every mode , so on its own it is mode-seeking and can collapse onto a subset of the data. The two are therefore complementary—Forward XM focuses on recall/coverage whereas Reverse XM focuses on precision. In practice, the two can be combined to control for precision and recall. Moreover, exploration can also be added onto existing generative models and applied to partial, masked, or noised samples rather than only full generations, as in the hybrid XMs of Section 4.1 . We discuss implementation details for both variants in Appendix C .

#### What Forward and Reverse XM Optimize.

We can make the recall and precision behaviors of Forward and Reverse XM precise by asking what distribution each one drives the model toward. The starting point is that the squared error between a generation y ^ \hat{y} and a datapoint x x is, up to a constant, the negative log of a Gaussian k σ ​ ( y ^ , x ) k_{\sigma}(\hat{y},x) of width σ \sigma centered on the generation ( σ \sigma is an analysis device set by the loss scale, not a hyperparameter). Each generation can therefore be seen as placing a small bump of density around itself, and averaging these bumps over everything the model generates gives the model a density of its own, p θ ​ ( x ) = 𝔼 y ^ ∼ G θ ​ [ k σ ​ ( y ^ , x ) ] = ( g θ ∗ k σ ) ​ ( x ) , p_{\theta}(x)=\mathbb{E}_{\hat{y}\sim G_{\theta}}\!\left[k_{\sigma}(\hat{y},x)\right]=(g_{\theta}*k_{\sigma})(x), where g θ g_{\theta} is the distribution of the model’s generations and ∗ * is convolution, so p θ p_{\theta} is just g θ g_{\theta} blurred by the kernel. Blurring the data distribution p ∗ p^{*} the same way gives p σ ∗ = p ∗ ∗ k σ p^{*}_{\sigma}=p^{*}*k_{\sigma} . In their smooth, large- K K forms, 6 6 6 The smooth form scores the K K candidates by − log 1 K ∑ i k σ ( y ^ i , x ) -\log\tfrac{1}{K}\sum_{i}k_{\sigma}(\hat{y}_{i},x) rather than by the best alone, and differs from the hard min by at most log ⁡ K \log K . Forward and Reverse XM then minimize KL ( p ∗ ∥ p θ ) + H ( p ∗ ) ⏟ Forward XM and KL ( g θ ∥ p ∗ σ ) + H ( g θ ) ⏟ Reverse XM , \underbrace{\mathrm{KL}(p^{*}\,\|\,p_{\theta})+H(p^{*})}_{\text{Forward XM}}\qquad\text{and}\qquad\underbrace{\mathrm{KL}(g_{\theta}\,\|\,p^{*}_{\sigma})+H(g_{\theta})}_{\text{Reverse XM}}, where KL \mathrm{KL} measures the mismatch between two distributions and H H is entropy. The two objectives are mirror images except for the entropy each carries. Forward’s entropy is the data ’s, a constant the model cannot change, so Forward XM is maximum likelihood of the mixture its explored candidates form, and its large- K K optimum recovers the data distribution up to that blur. Notably, this maximum likelihood reading holds at every K K ; what changes with K K is the density the likelihood is fit over. At K = 1 K{=}1 that density is a single Gaussian—the familiar fact that squared-error regression is Gaussian maximum likelihood—so the best the model can do is fit the blurred mean (Figure 2 ), while at larger K K it is a mixture of the K K candidates that can hold K K modes, one per candidate. In other words, K K scales how many modes the density can capture (its generative expressivity), which is why maximum likelihood alone can be misleading (Section 2.1 ). Reverse XM targets the reverse KL , the mode-seeking direction. However, the entropy in its objective is the model ’s own, so the model can lower its loss simply by shrinking its spread, potentially resulting in collapse. One solution is to add an entropy bonus that cancels the model’s entropy term and leaves the pure reverse KL; another is to combine Reverse XM with Forward XM, which is mass-covering. We give precise statements, assumptions, and proof sketches for these claims in Appendix F .

## 4 Experimentation and Results