ELT: Elastic Looped Transformers for Visual Generation — part 3/3

Cleaned-text code points [31388, 40151). Bibliography omitted here; source.md includes it.

We analyze the impact of ILSD for image generation across masked generative models (refer to Figure 8(a) ) and diffusion models (refer to Figure 8(b) ). Models trained without ILSD exhibit significant divergence from their fixed training depth ( L m ​ a ​ x L_{max} ). In contrast, ELT maintains stable, high-quality generation across the entire inference loop spectrum (see 1 ). We further analyse class-conditional video generation on UCF-101 in Figure 7 . Interestingly, ILSD enables the model to maintain reasonable quality even at unseen depths ( L > L max L>L_{\text{max}} ). On UCF-101, the model achieves a peak FVD of 69.20 at L = 6 L=6 despite being trained with L max = 4 L_{\text{max}}=4 , suggesting that ILSD regularizes the iterative process sufficiently for modest extrapolation beyond training depth. We note that this extrapolation behavior warrants further investigation across datasets and scales.

## 6 Conclusion

We proposed a novel parameter-efficient approach to visual generation using recurrent transformers called Elastic Looped Transformers (ELT). Our approach achieves a strong empirical performance, similar to baselines, with 4 × 4\times fewer parameters in iso-inference-compute setting in both image and video generation tasks. Beyond significantly improved performance per parameter, we identified fundamental scaling properties of looped transformers: while increasing model width remains a primary driver of quality, recursive looping provides a unique “test-time” compute lever. Through our proposed Intra-Loop Self Distillation (ILSD) strategy, we train a single model that is performant across a variable number of iterations. This strategy effectively yields a continuous family of models from a single training run, enabling Any-Time inference where practitioners can traverse the pareto front to balance image quality and GFLOPs dynamically.

Looking forward, we note that ELTs can potentially unlock more efficient inference for diffusion models. While existing approaches rely on the same network (and hence allocate same compute) across denoising steps, ELT can dynamically allocate compute across denoising steps, spending more compute where it matters the most. Additionally, in the context of recent one-step generative modeling paradigms such as consistency models [ 67 ] and drifting models [ 11 ] , ELT can enable true elasticity: since there is only one sampling step, one can dynamically control the quality of the model at inference by varying the number of loops, without having to pre-determine the number of sampling steps as is the case with traditional multi-step diffusion models. We believe this paradigm of flexible, weight-efficient scaling offers a promising direction for deploying high-fidelity generative models on resource-constrained hardware.

## Appendix A Implementation Details

We explain our training and inference algorithms in Algorithm 1 and Algorithm 2 respectively. We detail the ELT’s implementation details below: (a) Masked Generative Models : Refer Table 5 for details of configurations and hyperparameters of ELT-based masked generative models. We experiment with several model scales (refer Table 6 for architecture details). Note that for model scale ≥ \geq H, we increase the weight decay for AdamW [ 46 ] optimizer by a factor of 5. We sweep over multiple classifier-free guidance (cfg) [ 28 ] scales for ELT as well as baseline. We do not use cfg for UCF-101 class-conditional video generation. Sampling Temperature (STemp) mentioned in Table 5 controls the randomness of the sampling from the categorical distribution of logits. Tokens are sampled from logits/STemp. STemp is calculated as: STemp = b ​ i ​ a ​ s + s ​ c ​ a ​ l ​ e × ( 1 − ( k + 1 ) / K ) \text{STemp}=bias+scale\times(1-(k+1)/K) (2) where b ​ i ​ a ​ s bias and s ​ c ​ a ​ l ​ e scale are hyperparameters, k k is current sampling step and K K is total number of sampling steps. Throughout our experiments, we use b ​ i ​ a ​ s = 0.5 bias=0.5 and s ​ c ​ a ​ l ​ e = 0.8 scale=0.8 . (b) Diffusion Models : Refer Table 5 for details of configurations and hyperparameters of ELT-based diffusion models. For diffusion models, we experiment with two model architectures (refer Table 7 for details).

Note: Algorithm is for a single sampling step. Compute is dynamically scalable by adjusting the loop budget L L , allowing for flexible inference-time compute without model retraining.

## Appendix B Additional Experiments

Cost and Efficiency : ELT drastically reduces the unique param count, which in turn helps in following: (i) Hardware Costs : The primary bottleneck in training and deploying modern generative models is often memory (HBM/VRAM). ELT reduces the memory budget needed to train and serve larger effective depths. (ii) Training Speedup : As ELT trains intermediate loops using a single forward pass with negligible cost of the prediction head, training step time remains same. Table 8 (corresponding to Section 5.2 results) profiles DiTs training on TPU v6e with 8N × 4L looping config, using a TPU v6e with 8 ​ N × 4 ​ L 8N\times 4L looping config. 𝒟 \mathcal{D} (effective depth) is 32 for all methods. This infrastructure efficiency compounds with the algorithmic convergence speed-up demonstrated in Figure 6 , these complementary benefits yield a overall reduction in training wall-clock time.

While matching the baseline’s theoretical FLOPs, ELT inherently reduces peak HBM memory by 34%, which acts as a budget to speedup training . For instance, ELT opt uses unrolling (“for loop") instead of “jax.lax.scan" across layers axis. This generates a new computational graph for every layer, removing some sequential computation, utilizing the ELT’s memory savings and reducing step time. Other orthogonal optimizations include increasing batch-size and gradient checkpointing. (iii) Inference Speedup : Please refer to Section 5.2 for quantitative comparison. ELT eliminates redundant memory transfers (HBM → \rightarrow VRAM), achieving 3.5x peak throughput speedup in iso-inference compute settings.

Losses Ablation : Figure 7 (video gen) and Figure 8 (image gen) show the ablation of vanilla looping vs ELT across masked generative models and DiTs. We further ablate the importance of ILSD loss in Figure 13 .

Iterative Refinement : We analyzed the per-iteration update magnitudes (L2 norm) across the loops and sampling steps ( Figure 14 ). It shows that initial denoising steps have more updates for higher loops than later denoising steps. These observations can potentially be used to make the number of loops adaptive across denoising steps.

## Appendix C Limitations

While ELT demonstrates strong parameter efficiency and elastic inference capabilities, we identify some limitations of the current work.

Failure Cases : We observe that ELT performance degrades in two notable scenarios: (i) when the number of unique layers N N is too small (e.g., 1 ​ N × 32 ​ L 1N\times 32L configuration in Section 5.2 ), a single layer lacks representational capacity regardless of loop count, and (ii) when loop count L L significantly exceeds L max L_{\text{max}} at inference, quality can deteriorate as the shared block over-iterates beyond its trained convergence regime.

Deployment Considerations : While ELT offers significant parameter savings and throughput gains, practical deployment requires careful selection of the operating point ( N N , L L ) based on the target hardware. The optimal loop count L L for a given quality target is model-scale dependent and should be calibrated per deployment tier.

## Appendix D Acknowledgments

We thank Nikunj Saunshi, Thomas Mensink and Ashwini Pokle for their constructive comments and valuable suggestions, which enhanced the overall quality of this manuscript.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .