Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation — part 4/8
Cleaned-text code points [46260, 62048). Bibliography omitted here; source.md includes it.
Much of deep learning progresses by running experiments first and explaining them afterward. Because this work builds on Mode Forcing [ 19 ] , most of its results came about in the opposite order, where the theory predicted them before the experiments were run. Here we outline Mode Forcing’s predictions and their confirmations: • Even the strongest generative models are short on generative expressivity. Mode Forcing argues that factoring generation often still leaves modes uncaptured, with heavy reliance on guidance as evidence for this (Section 2.1 ). Added generative expressivity should therefore improve even the most heavily tuned recipes, and it does: exploration lifts image, video, and language generation performance for all recipes tested. • Generative expressivity increasingly becomes the bottleneck at scale. As parameters and data stop limiting what models can represent and learn, Mode Forcing predicts that generative expressivity set by the training objective should increasingly become the bottleneck. We find exactly this, where gains from exploration climb from 7 % 7\% to 36 % 36\% as data scales and 13 % 13\% to 23 % 23\% as models grow (Figure 10 ), and the FLOP-optimal amount of exploration rises over the course of training (Figures 4(b) and 5(b) ). • Exploration can substitute for generation factorization. As discussed in Section 2.1 , factoring generation exists to supply generative expressivity, so supplying it through exploration instead should reduce how much generation factorization a model needs. We confirm this, where as exploration grows, the optimal amount of generation factorization decreases, and more end-to-end models perform better (Figure 11 ). • XMs enable end-to-end generation. Mode Forcing argues that multimodal distributions are the core reason generation has to be factored, so if exploration handles multimodal distributions during training, end-to-end reconstruction should work. Our Explorative Policy and World Model confirm this, matching diffusion baselines with as little as a single forward pass instead of hundreds (Tables 2 and 3 ). #### The Benefits of Surplus Generative Expressivity. Interestingly, exploration helps even when generative expressivity is not a large bottleneck. In our video generation experiments, XMs improve performance significantly even though the modeled distribution is not very multimodal—Jumpy models need only 10 steps in this setting (Figure 7 ), far less generation factorization than modern Diffusion models. Why would exploration help when there are few modes to capture? Our hypothesis is that even a weakly multimodal target still pulls each prediction toward multiple competing values over the course of training, and the same conflicting pulls that blur modes also make optimization harder. Exploration relieves this pressure, letting each prediction train toward its nearest match, so targets compromise less and optimization becomes easier. This mirrors overparametrization, where models with far more parameters than needed to fit their data consistently perform and generalize better [ 54 , 53 ] —commonly attributed to smoother loss landscapes and a bias toward simpler solutions [ 66 , 52 ] . In both cases, surplus capacity makes good solutions easier to find, suggesting that exploration, like parameters, is worth scaling past the point where it seems strictly necessary. ## 6 Future Works and Broader Impact XMs open several research directions, below we highlight some of these directions. #### Exploration as a Scaling Axis for More Generative Models. We demonstrated exploration acts as a scaling axis for Diffusion/Flow, Jumpy, and masked diffusion language models; we believe other generative models likely benefit from exploration in the same manner. Autoregressive LLMs have proven the hardest case, for the reasons discussed in our limitations (Section 7 ). Evaluation is also part of the challenge, as language modeling lacks robust distributional metrics like FID and FVD that would reveal mode coverage. We see two promising paths to build on these early gains. Multi-token prediction [ 67 ] targets are more multimodal, so multi-token prediction suffers more from limited generative expressivity and gives exploration more to offer. The Free Transformer [ 68 ] conditions a decoder on a latent variable inferred by a VAE, which is exactly the kind of latent exploration searches over, and training it with Explorative Modeling instead would remove the VAE entirely, along with the exposure bias of training on inferred latents (Section 3.1 ). Beyond language models, few-step models such as MeanFlow [ 15 ] are a natural fit for exploration as well, since exploration can supply the generative expressivity their shortened trajectories amortize. We also believe in combining XMs with Energy-Based Transformers (EBTs) [ 69 ] , where the biggest documented challenge with EBTs has been end-to-end generation and handling highly multimodal distributions, which is exactly what XMs enable. Paired together, XMs and EBTs could enable more dynamic reasoning, search, and generalization over entire sequences. #### End-to-end XM Applications. End-to-end XMs are especially well suited to new applications such as inpainting and super-resolution due to low amounts of multimodality in generated distributions. They could also pair with feature-based world models like JEPA [ 70 ] to build end-to-end world models. In the short term this pairing is especially practical for Forward XM, as feature spaces often contain far fewer modes than raw observations [ 71 ] , so a small K K suffices (in the long run, we believe XMs can scale to arbitrarily multimodal settings via Reverse XM, Section A ). Exploration would also resolve a core JEPA challenge: next-state prediction and trajectory-level planning are multimodal, especially in non-deterministic environments, which feature regression blurs but exploration captures. Another appealing direction is combining exploration with moment matching [ 72 ] , which would let models reconstruct features at the right granularity. Finally, end-to-end XMs further enable setting the number of modes a model captures, which existing generative models struggle with, and because Forward XM favors recall while Reverse XM favors precision, choosing between or combining them gives direct control over generation diversity. #### Improving and Understanding XMs. There is plenty of room to improve the core mechanism of exploration itself; in this paper we primarily used the simplest approach of sampling many different random noise candidates for Diffusion/Flow/Jumpy models. In principle, architectures could condition on discrete latent embeddings for each explorative factor, which could give better controllability, enable more uniform sampling of modes, and improve mode coverage (we did this for MDLMs, but no other models). Additionally, there are likely better exploration approaches that exist. In this work, exploration was done by drawing K K independent candidates. In principle, however, searching for the optimal latent could be done in better ways, such as by treating the reconstruction loss as an energy, and finding the best latent by gradient descent [ 73 ] . The risk in using this approach is that it could cause a mismatch reminiscent of VAE prior holes [ 74 ] , where the latents found by search differ from those sampled at inference, though applying such search only late in training or using other tricks could avoid this. Forward XM’s cost could also be cut with a cheaper scorer, such as a smaller proxy that ranks candidates so only the winner is generated in full. Training on the soft min rather than the hard min is another variant, letting every candidate contribute gradients and carrying the cleaner maximum likelihood interpretation (Section F.2 ). Another interesting idea would be to unify exploration with an end-to-end learned encoder, so search happens over learned latents (this could be combined with recent work on learning generative models and encoders jointly, such as Unified Latents [ 75 ] ). Finally, exploration deserves the same scaling-law treatment as parameters and data: the compute-optimal amount of exploration already grows with scale (Figures 4(b) and 5(b) ), so understanding how to optimally allocate compute between exploration, parameters, and data, similar to Chinchilla [ 16 ] , would be insightful. #### Scaling Reverse XMs. Despite Reverse XMs’ potential to collapse (Section 3.2 ), we see them as more promising in the long run over Forward XMs, as they add almost no extra FLOPs and scale more gracefully with the number of modes. With discrete conditioning, Reverse XMs come essentially for free, since loading a larger batch with K K data points per condition lets each generation pick its best match (we could have done this for our image generation experiments, but chose not to in order to keep implementations simple and modality/domain agnostic). Doing Reverse XM with continuous conditioning is harder, as data points rarely share the exact same condition, so each generation has no ready-made set of valid targets to search, and the central design question becomes how data is loaded. More ambitiously, a vector database over the whole dataset would let each generation search all training data in logarithmic time, so the number of modes explored can in principle reach the dataset size, directly matching the generated distribution to the training distribution. We have already made Reverse XMs work this way on language modeling tasks (more on this in Section A ). One remaining challenge is that a generation’s nearest datapoint can flip-flop across training steps, blurring the effective target; sticky couplings that persist matches across steps could prevent this. #### Exploration beyond Pretraining. The mode collapse XMs address during pretraining also often shows up in post-training, where RL fine-tuning is known to sharpen models onto a narrow set of behaviors [ 76 ] . Recent fixes such as pass@ k k rewards [ 77 ] and best-of- N N -aware fine-tuning [ 78 ] can be seen through our lens as Forward XM, with a verifier standing in for ground truth data. These fixes act only during post-training, though; pretraining with exploration may yield base models that capture more modes in the first place, leaving RL more to select among. ## 7 Limitations and Conclusion In this work, we introduced Explorative Modeling, a new paradigm for handling multimodal distributions that factors the training loop instead of the generation procedure. Exploration increases generative expressivity, adding a new pretraining axis for existing generative models, and enabling end-to-end generative modeling. #### Limitations. As a scaling axis for existing models, exploration is easier to integrate into some model families than others. This is because exploration requires a latent variable to search over when selecting the best of K K candidates. Continuous generative models benefitted most easily, as they already condition on a noise z z ; MDLMs benefited once given a learned latent variable embedding for exploration. We found autoregressive language models harder to improve with exploration, likely because injecting a latent into them is less natural, and because they are less bottlenecked by generative expressivity than many other models. Despite this, we have achieved initial modest results showing improved data efficiency, suggesting exploration can benefit autoregressive LLMs further with more effort. Exploration also changes the training objective, so losses are no longer directly comparable across exploration levels. This makes distributional metrics such as FID and FVD, as well as downstream metrics such as accuracy, more important for evaluation. Similarly, existing guidance techniques were designed without exploration in mind, and some transferred to XMs better than others: autoguidance worked decently, while classifier-free guidance helped less than it does for base models, despite XMs’ stronger unguided FID indicating they capture the underlying density better. Guidance is known to not transfer uniformly across models—for example, vanilla CFG also fails to improve models trained on representation autoencoder latents by default [ 45 ] —suggesting even our autoguidance results likely undershoot what XMs could achieve with guidance designed for exploration. Exploration also supplies a signal base models lack, namely K K candidates and a notion of which was best, so we believe using it to design guidance tailored specifically to XMs is one of the most important open problems. Fully end-to-end XMs face a couple of challenges. The most significant challenge with end-to-end XMs is in handling highly multimodal data: fully end-to-end Forward XMs need K K to grow with the number of modes, which is currently too expensive for distributions with very many modes (e.g., image generation). Therefore, while the world remains somewhat compute constrained, Reverse XMs are a natural solution for handling high distribution multimodality, where a single model generation can search arbitrarily many training data points. Reverse XMs do bring their own considerations: being mode-seeking, they require an entropy term or coverage constraint to avoid collapse, and searching data efficiently requires good representations along with a dataloader or vector database supporting the search. Another challenge is that end-to-end XMs give up the implicit regularization of factored generation: when each step trains on corrupted or partial inputs, memorization is harder, so fully end-to-end models are more prone to memorizing when data is scarce. This concern fades with scale, however, as having abundant data itself acts as regularization [ 79 , 80 , 81 ] . Taken to the limit, with enough data and compute, it’s plausible that generative modeling simply becomes exploration for good latents using a very high K K . #### Conclusion. Across both continuous and discrete domains, we found exploration acts as a third pretraining axis alongside parameters and data, with gains that grow with scale rather than saturate—rising from 7 % 7\% to 36 % 36\% as data scales, 13 % 13\% to 23 % 23\% as models grow, and with efficiency gains more than doubling at 3 × 3\times the compute. Because gains from exploration keep climbing with scale, the numbers we report are likely a floor for benefits at increased model scale. Concretely, exploration improves FLOP efficiency by 4.1 × 4.1\times , sample efficiency by 6.2 × 6.2\times , and parameter efficiency by 47 % 47\% , while lifting the strongest of image-generation recipes to a near-state-of-the-art 1.43 1.43 FID on ImageNet without guidance. Beyond efficiency, exploration enables scaling generalization : spending more training compute on exploration improves generalization directly, and improves it indirectly by enabling existing models to become more end-to-end. Taken to its limit, exploration enables fully end-to-end reconstructive generation, matching diffusion on control tasks with as little as a single forward pass in place of hundreds. For over a decade we have scaled how large generative models are and how much data they train on ; XMs let us scale what models can generate . ## Author Contributions #### Alexi Gladstone led the project from ideation to execution, conceiving Explorative Modeling, developing the theory and method, designing and running all experiments, and writing the paper. #### Heng Ji and Yilun Du advised the project throughout, providing invaluable mentorship, feedback on the ideas and writing, and compute support. Yilun had crucial initial ideas on XMs for training EBMs. ## Acknowledgement