Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation — part 1/8

Cleaned-text code points [0, 14794). Bibliography omitted here; source.md includes it.

# Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

###### Abstract

The deep learning revolution, kicked off by AlexNet, taught us that end-to-end training beats decomposing a problem into hand-designed stages. Generative modeling, however, has remained the exception---despite generative models being remarkably capable, they are still not trained end-to-end. 1 1 1 The term end-to-end generative modeling is often used loosely. We provide a stricter definition in Section 2 —simply put, sampling during training should be the same as sampling during inference. This is because, at its core, generative modeling is about handling multimodal distributions , 2 2 2 By multimodal we mean a probability distribution with many modes (distinct peaks), not data of different modalities such as text and images. and existing scalable approaches handle this multimodality the same way, by factoring the generation procedure , which prevents end-to-end generation. In this work, we introduce Explorative Modeling , a new paradigm that instead factors the training loop, exploring K K candidate matches between model generations and data, and training on the best, so predictions commit to modes rather than blurring them. We find Explorative Models ( XMs ) useful in two settings. First, increasing exploration adds a third pretraining axis beyond parameters and data for existing generative models—where scaling exploration monotonically improves performance across both continuous and discrete domains (images, video, and language). Notably, gains from exploration increase with scale , climbing from 7 % 7\% to 36 % 36\% as data scales and from 13 % 13\% to 23 % 23\% as models grow, with efficiency gains more than doubling at 3 × 3\times the compute. Concretely, exploration improves FLOP efficiency by 4.1 × 4.1\times , sample efficiency by 6.2 × 6.2\times , parameter efficiency by 47 % 47\% , lifts the strongest of image-generation recipes to a near-state-of-the-art 1.43 FID on ImageNet without guidance, enables scaling how end-to-end existing models are , and unlocks scaling generalization . Second, XMs enable end-to-end reconstructive generative modeling, matching diffusion on control tasks with 16 16 - 256 × 256\times fewer inference steps. Together, these results establish XMs as both a new pretraining axis for existing generative models and a standalone end-to-end generative modeling paradigm .

We scale the size of generative models and how much data we train them on… so why haven’t we scaled what they can generate?

## 1 Introduction

AlexNet kicked off the deep learning revolution when end-to-end training beat hand-designed layer-wise training, demonstrating that learning everything performs better than hand-engineering [ 1 ] . Since then, end-to-end neural networks have broadly replaced many hand-built pipelines across the field including image classification [ 2 , 1 ] , object detection [ 3 ] , and image segmentation [ 4 ] , learning each task directly from data. Much of this success comes down to a single property: end-to-end models perform inference exactly as they were trained, so they are never exposed to inputs unlike those seen in training, which avoids distribution shifts and exposure bias that degrade performance and generalization [ 5 , 6 , 7 , 8 , 9 ] .

Despite this trend, generative modeling has remained the holdout: its most common and scalable recipes today— reconstructive generative models (described in Section 2 )—are not end-to-end, sampling completely differently at inference than during training. For example, autoregressive and diffusion models are trained to predict a single step, but used at inference as recurrent neural networks over hundreds to thousands of predicted tokens or denoising steps, so per-step errors feed into the next step, drifting inputs off the training distribution and compounding errors [ 8 , 9 , 10 , 11 , 12 ] . This shortcoming raises a simple question: “Why can’t we train reconstructive generative models end-to-end?”

We argue this is because generative modeling is fundamentally about handling multimodal probability distributions . To achieve this at scale, existing reconstructive models—whether autoregressive, diffusion [ 13 ] , or single-step [ 14 , 15 ] —factor the generation procedure into smaller steps during training, making each step’s target nearly unimodal so that a reconstruction loss no longer blurs distinct modes into their average. This factorization, however, is exactly what prevents current generative models from being end-to-end, so what else can we factor instead?

A generative model has only two processes to decompose—how it generates and how it trains (Figure 1 ). Because we’ve ruled out factoring generation, we factor the training loop itself—a new paradigm we call Explorative Modeling : at each training step, the model explores K K possible matches between what it generates and the data, and trains on the closest. Because this happens entirely during training, Explorative Models ( XMs ) capture multimodal distributions while enabling end-to-end generation.

Exploration works by searching for which latent should be matched to which datapoint, something standard generation factorization completely sidesteps. In generative modeling, there is nothing that determines which datapoint each latent, such as input noise, should produce, so a latent is typically paired with targets at random. When models are trained to reconstruct many different valid targets at random, the best a single prediction can do is predict their average, a blur that matches no real datapoint (Figure 2 XM-1). XMs instead search for the latent whose generation is already closest to each datapoint, so each explored candidate can commit to a different mode, meaning the number of modes a model can capture, its generative expressivity (Section 2 ), grows directly with the amount of exploration.

As a consequence, we find XMs are valuable in two settings. First, added to existing generative models, exploration is a new pretraining axis beyond parameters and data. Existing generative models fix generative expressivity at training time through how they factor generation, so when they cannot capture every mode in the data, performance is capped no matter how far parameters and data scale. Because exploration scales generative expressivity directly, it relieves a bottleneck the other axes cannot, monotonically improving performance across both continuous and discrete domains---including images, video, and language. Crucially, like scaling parameters or data, 3 3 3 Under compute-optimal scaling, parameters and data have to grow together: increasing one while holding the other fixed becomes increasingly suboptimal [ 16 ] . Our findings show that exploration largely acts the same way: as scale increases, models without exploration fall increasingly short of compute-optimal performance. these gains grow with scale rather than saturate, rising from 7 % 7\% to 36 % 36\% as data grows, 13 % 13\% to 23 % 23\% as models grow, and with efficiency gains more than doubling at 3 × 3\times the compute, so these numbers likely understate the gains at larger scale. Concretely, exploration improves FLOP efficiency by 4.1 × 4.1\times , sample efficiency by 6.2 × 6.2\times , parameter efficiency by 47 % 47\% , lifts the strongest of image-generation recipes to a near-state-of-the-art 1.43 1.43 FID on ImageNet without guidance, enables a compute-generalization tradeoff where more exploration improves generalization, and increasing exploration enables existing generative models to be trained more end-to-end.

Second, as a standalone approach, exploration enables end-to-end reconstructive generative modeling . We find end-to-end XMs match Diffusion Policy [ 17 ] on behavior cloning and Diffuser [ 18 ] on goal-conditioned world modeling, while taking as little as a single forward pass in place of hundreds ( 16 16 - 256 × 256\times fewer).

In summary, we make the following contributions: • We introduce Explorative Modeling , a new paradigm for handling multimodal distributions that works by factoring the training loop instead of the generation procedure.

• We show exploration is a new scaling axis for existing generative models, with gains in FLOP, parameter, and sample efficiency that increase with scale .

• We find exploration enables a compute-generalization tradeoff , where spending more training compute on exploration directly improves generalization.

• We show factorizing training and generation are substitutable : as exploration increases, the optimal generative model becomes more end-to-end, indirectly improving generalization.

• We present a scalable end-to-end reconstructive generative modeling approach , matching diffusion on control tasks at 16 − 256 × 16-256\times less inference compute.

Ultimately, exploration lets us scale how end-to-end existing generative models are, and taken to its limit, makes generative modeling fully end-to-end—extending to generative modeling the end-to-end training that has driven the rest of deep learning.

## 2 Background

In standard supervised learning, such as classification or regression, each input generally has a single correct output, so a deterministic mapping is sufficient. Generative modeling has no such mapping: a request like “generate a dog” has no single right answer, as there are billions, or even infinitely many, valid dog images. These valid outputs are the modes of the data distribution, and this large number of modes is what makes generation hard, so capturing them is the central focus of generative modeling.

### 2.1 Mode Forcing

Generative models broadly fall into two families: reconstructive and contrastive [ 19 ] . Contrastive generative models, such as GANs [ 20 ] and contrastive-divergence energy-based models (CD EBMs) [ 21 , 22 ] , are trained by contrasting generated data against true data—leveraging relative supervision from comparing samples with no explicit target—but have struggled with scalability. We therefore focus on reconstructive generative models—the most common family that has scaled best thus far—which are trained by mapping a self-produced input, such as noise or a corrupted sample, back to an explicit data target that supervises each prediction, and include autoregressive, diffusion [ 13 ] , and flow [ 23 ] models. This pairing of an input with the target it should map to is called the coupling , and the challenge with reconstructive models is that we do not know this coupling beforehand, so a single input is typically coupled to many valid targets across the dataset. This one-to-many coupling is what causes mode blurring when doing generative modeling naively, as the reconstruction loss minimizer of many targets is the mean, which lands between modes and matches no real datapoints (demonstrated in Figure 2 for XM-1).

Recent work on Mode Forcing [ 19 ] points out that every scalable reconstructive model is built to dodge exactly this blur, with the central thesis that modern generative modeling is the art of designing a reconstructive objective whose loss minimizer captures modes instead of averaging them . Existing approaches for achieving this at scale function by factoring generation into a sequence of smaller, nearly unimodal steps, so no single prediction is forced to average across modes. For instance, autoregressive models reconstruct a target one element at a time, predicting each element from the ones already revealed, which means there is rich conditioning to make the prediction over the next element nearly unimodal. Diffusion and flow models [ 13 , 23 ] instead reveal the target gradually through denoising: each step conditions on a slightly noisier version and predicts a slightly cleaner one, keeping every step nearly unimodal. This factoring of the generation procedure is why scalable generative modeling approaches succeed at generating high quality samples, whereas direct, single-step regression does not.

In general, scaling generative models has meant scaling just two axes: parameters or model size/FLOPs, which govern what a model can represent , and the amount of data and training length, which govern what a model can learn . Mode Forcing suggests these two axes miss a third capacity:

Unlike parameters and data, generative expressivity is set by the training objective itself, so it stays fixed no matter how far the other two axes scale. When an objective allows for capturing fewer modes than the data has, the surplus modes are not dropped but averaged, so a single prediction lands between them and matches no real datapoint (Figure 2 XM-1). Formally, letting M ⁡ ( q ) M(q) denote the mode count of a distribution q q and P θ ( ⋅ ∣ c ) P_{\theta}(\cdot\mid c) the model’s sampling distribution given conditioning c c , generative expressivity is the largest conditional mode count an approach’s loss minimizers can retain over any data distribution, E ≜ sup p ∗ , c sup θ ⋆ ∈ arg ⁡ min θ ⁡ ℒ ⁡ ( θ ) M ( P θ ⋆ ( ⋅ ∣ c ) ) E\triangleq\sup_{p^{*}\!,\,c}\,\sup_{\theta^{\star}\in\arg\min_{\theta}\mathcal{L}(\theta)}M\big(P_{\theta^{\star}}(\cdot\mid c)\big) (further details in Section F.1 ). Direct regressors demonstrate why this axis matters, as they have E = 1 E=1 : even with unlimited parameters and data , their best possible output (loss minimizer) is still a single blurred mean of all the modes (Figure 2 XM-1). This is counterintuitive because direct squared-error regression is itself maximum likelihood under a fixed-variance Gaussian—the likelihood is maximized faithfully, just over a density with a generative expressivity of one, whose best fit to multimodal data is the mean. Therefore, this suggests that the traditional notion of performing some form of maximum likelihood with sufficient data and parameters is not enough , but rather that generative expressivity is an overlooked scaling axis. This may also explain why likelihood has long been observed to correlate poorly with sample quality [ 24 ] , as likelihood measures how well a density is fit while generative expressivity determines how many modes that density can hold. As a consequence, the field’s primary goal of optimizing likelihood alone may be the wrong one to chase, and generative expressivity should be optimized alongside it. At its core, Explorative Modeling is a new way to increase generative expressivity (Figure 1 )—exploring K K candidates with a direct regressor raises generative expressivity to at least K K , sharpening blurred means into distinct modes (Figure 2 ). Because factoring generation exists to supply this same quantity, factorizing generation and training are substitutable , which we confirm empirically in Section 4 .