LLaDA2.0: Scaling Up Diffusion Language Models to 100B — part 1/4

Cleaned-text code points [0, 15776). Bibliography omitted here; source.md includes it.

# LLaDA2.0: Scaling Up Diffusion Language Models to 100B

###### Abstract

This paper presents LLaDA2.0 — a tuple of discrete diffusion large language models (dLLM) scaling up to 100B total parameters through systematic conversion from auto-regressive (AR) models — establishing a new paradigm for frontier-scale deployment. Instead of costly training from scratch, LLaDA2.0 upholds knowledge inheritance, progressive adaption and efficiency-aware design principle, and seamless converts a pre-trained AR model into dLLM with a novel 3-phase block-level WSD based training scheme: progressive increasing block-size in block diffusion (warm-up), large-scale full-sequence diffusion (stable) and reverting back to compact-size block diffusion (decay). Along with post-training alignment with SFT and DPO, we obtain LLaDA2.0-mini (16B) and LLaDA2.0-flash (100B), two instruction-tuned Mixture-of-Experts (MoE) variants optimized for practical deployment. By preserving the advantages of parallel decoding, these models deliver superior performance and efficiency at the frontier scale. Both models were open-sourced.

Huggingface: https://hf.co/collections/inclusionAI/llada-20

## 1 Introduction

Large Language Models have achieved remarkable success through the AR paradigm, modeling sequences via next-token prediction with strict left-to-right causal dependencies ( Hurst et al., 2024 , Grattafiori et al., 2024 , Yang et al., 2025 ) . This approach naturally aligns with the sequential structure of language and enables efficient training through next-token likelihood maximization. However, the very success of this paradigm creates fundamental limitations: the sequential generation process imposes severe inference bottlenecks, precluding parallelization, and increasing latency at scale, while the rigid causal structure can be suboptimal for tasks requiring bidirectional reasoning and holistic understanding.

Discrete Masked Diffusion Language Models (MDLM) have emerged as a compelling alternative to the prevailing AR paradigm. By reconstructing sequences from random masked inputs, these models inherently support parallel generation and leverage a full bidirectional context, offering a different architectural approach ( Gong et al., 2025 , Yu et al., 2025 ) . Although these conceptual advantages are clear, the field is still in an early developmental stage. Current research is actively focused on key challenges, including the refinement of specialized training regimes, the design of efficient sampling strategies, the efficient inference of opensource models, and reinforcement learning for MDLM. As a result of this ongoing exploration, most existing diffusion models, including recent advancements like Block Diffusion Language Models (BDLMs) ( Arriola et al., 2025 ) , operate at a smaller scale ( e.g. , ≤ \leq 8B parameters). Bridging this scale difference to the hundreds of billions of parameters seen in the leading mainstream AR models is a primary frontier for enabling diffusion models to fully capture complex linguistic patterns for practical deployment.

In this work, we introduce LLaDA2.0 series with 100B/16B total parameters diffusion language models that resolves these fundamental challenges through a novel two-stage continual pre-training (CPT) paradigm. Rather than attempting to train diffusion models from scratch, we leverage existing AR checkpoints as the foundation for a systematic conversion process that preserves linguistic knowledge while introducing diffusion capabilities.

The first stage, CPT aims to transform the foundational AR model into a capable diffusion language model. However, direct conversion is challenging due to the inherent data distribution gap between left-to-right generation and bidirectional denoising. Although the BDLM formulation partially reduces this gap through blockwise masked reconstruction, it suffers from low data utilization, limiting the effective exploitation of large-scale corpora. To this end, we introduce the Warmup–Stable–Decay (WSD) strategy, smoothly bridging the AR-to-dLLM gap while substantially improving CPT efficiency. WSD gradually expands the model’s receptive field to introduce diffusion-style context ( Warmup ), strengthens global denoising under full-sequence training ( Stable ), and then refines the model into an efficient blockwise structure ( Decay ). This progressive adjustment enables a stable and data-efficient transition to diffusion-based learning. Additionally, under full attention in packed training sequences, diffusion models risk forming spurious dependencies across document boundaries, leading to semantic confusion and instability in bidirectional training. To prevent such cross-document interference, we introduce a document-level attention mask that restricts self-attention within individual documents, ensuring coherent context modeling.

The second stage, Post-training for Practical Deployment, transitions the model from a raw predictive engine into a capable and efficient assistant. The random masking nature of the diffusion fine-tuning objective means any single sample provides only a partial learning signal. We address this by employing a complementary masking strategy, which ensures near-100% data utilization and accelerates convergence by guaranteeing every token contributes to the model’s learning. With an efficient foundation for instruction tuning, we then align the model with human preferences by adapting modern techniques like Direct Preference Optimization (DPO)—originally designed for AR models—by reformulating the objective over the model’s reconstruction loss. Beyond alignment, practical deployment hinges on inference speed. To realize the full promise of parallel decoding, which is often limited by a model’s lack of predictive confidence, we incorporate an auxiliary confidence prediction loss. This trains the model to be “sharper” and more certain, unlocking aggressive and efficient parallel generation without degrading quality.

We release instruction-tuned variants for practical deployment: LLaDA2.0-mini (16B parameters) for resource-constrained applications and LLaDA2.0-flash (100B parameters) for high-performance scenarios. Both variants retain the parallel decoding advantages of our diffusion training while being optimized for instruction following and safety through comprehensive post-training alignment.

Our contributions provide a practical recipe for the community to leverage AR stability while achieving diffusion parallelism, opening new possibilities for efficient large-scale language modeling.

## 2 Related Work

### 2.1 Train dLLMs from scratch

Auto-regressive language models ( Ling et al., 2025 , Moonshot, 2025 , Liu et al., 2024 , Meta-AI, 2025 ) are typically trained by maximizing the likelihood of predicting the next token. Under this paradigm, model performance has been shown to scale effectively with increasing model size, dataset volume, and computational resources, following well-established scaling laws. Recently, MDLMs ( Song et al., 2025 , Ye et al., 2025 , Nie et al., 2025 ) have emerged as an alternative generative framework, reformulating text generation as an iterative denoising process. In each forward step, a subset of tokens is randomly masked, and the model is trained to recover the original tokens conditioned on the remaining unmasked context.

Encouraged by this paradigm shift, several studies have explored training MDLMs from scratch to assess their full potential. For instance, LLaDA ( Nie et al., 2025 ) demonstrated that a 8B dense MDLM, trained entirely from scratch, achieves performance competitive with similarly sized AR counterparts. Building upon this, LLaDA-MoE ( Zhu et al., 2025 ) introduced the Mixture-of-Experts (MoE) architecture into the MDLM for the first time, showing that a scratch-trained MoE-based MDLM can surpass dense models in both efficiency and capability, thereby validating the compatibility and scalability of MDLMs with advanced MoE designs. Moreover, due to the fundamentally different training dynamics compared to AR models, established training practices and hyperparameter recipes from the AR domain are often suboptimal for MDLMs. To address this gap, recent efforts such as Quakka ( Ni et al., 2025 ) and OpenMoE2 ( Ni and team, 2025 ) have begun investigating the scaling properties and optimal training strategies specifically tailored for MDLMs, laying the groundwork for principled scaling in this emerging paradigm.

However, from-scratch trained MDLMs still lag behind state-of-the-art AR models in overall performance. This gap can be largely attributed to the disparity in training data volume and the maturity of infrastructure support—factors that have been extensively optimized over years of development for AR models. Moreover, due to the high computational cost and long training cycles required for pretraining from scratch, MDLMs mentioned above are typically limited in model scale ( ≤ \leq 8B), whereas leading AR models now routinely scale into tens or even hundreds of billions.

### 2.2 Scaling dLLMs with AR initialization

Given the strong knowledge capacity and performance of AR models, several recent studies have explored initializing dLLMs from pre-trained AR models to reduce training costs and narrow the performance gap between AR models and dLLMs. For instance, DiffusionLLaMA ( Gong et al., 2025 ) and Dream-7B ( Ye et al., 2025 ) adopt a mask annealing strategy to gradually transition from causal attention to bidirectional attention during training, while employing a CART-based loss reweighting scheme to balance token-level learning dynamics. In contrast, RND1 ( Keshigeyan et al., 2025 ) takes a more direct approach by immediately converting the causal attention mechanism of the AR model into a bidirectional one upon initialization. Notably, RND1 observes that when initializing DLM training from an AR model, preserving knowledge-intensive capabilities requires constraining updates to the model’s dense layers to prevent catastrophic forgetting.

Block Diffusion Language Models (BDLMs) ( Arriola et al., 2025 ) provide a hybrid paradigm that balances efficiency and performance by combining diffusion and AR modeling. Tokens are generated block-wise: within each block, a diffusion process reconstructs masked tokens, while blocks are produced auto-regressively. This design enables variable-length generation and supports KV-cache reuse during decoding, enhancing inference efficiency. Consequently, BDLMs can be effectively initialized from AR models, narrowing the performance gap. For example, SDAR ( Cheng et al., 2025 ) leverages the Qwen-3 series ( Yang et al., 2025 ) to train more efficient BDLMs. By exploring various block sizes and optimization strategies, it achieves performance comparable to its AR base model.

However, one key limitation across all existing methods is their restricted model scale—ranging only from 7B to 30B parameters—leaving the feasibility and scalability of AR-initialized diffusion models largely unexplored at larger scales. Besides, the low training efficiency of block diffusion hinders its widely application to large-scale corpus for large-size models. Whether such initialization strategies can effectively generalize to models beyond the 30B scale remains an open question.

### 2.3 dLLMs post-training

Beyond pre-training, post-training is crucial for unlocking the full potential of dLLMs by aligning them with specific tasks and human preferences. This process typically involves supervised fine-tuning (SFT) to instill instruction-following capabilities, reinforcement learning (RL) to enhance complex reasoning, and inference optimization to address efficiency bottlenecks.

Recent work has explored SFT to adapt dLLMs for specialized domains. For instance, Dream-Coder ( Xie et al., 2025 ) fine-tunes a 7B dLLM for code generation, demonstrating unique abilities like adaptive ”sketch-then-fill” strategies for complex algorithms. Similarly, the general-purpose model Dream-7B ( Ye et al., 2025 ) leverages SFT to achieve performance on par with top-tier AR models, while uniquely excelling at tasks requiring complex planning and constraint satisfaction. Other studies have investigated specialized fine-tuning strategies to balance quality and efficiency. Seed-Diffusion ( Song et al., 2025 ) , for example, employs a two-stage curriculum learning strategy to train a high-speed code generation model, while LiDAR ( Liu et al., 2025 ) introduces a hybrid ”think in diffusion, generate in AR” architecture through fine-tuning, significantly boosting inference throughput while maintaining quality.

To further enhance dLLMs’ reasoning abilities, researchers have begun adapting reinforcement learning techniques. However, applying standard policy gradient methods is challenging due to the intractable log-likelihood of dLLMs. To address this, SPG ( Wang et al., 2025a ) proposes a novel Sandwich Policy Gradient algorithm that obtains a more robust and less biased gradient by maximizing an evidence lower bound for high-reward samples and minimizing an evidence upper bound for low-reward ones. Another line of work, TraceRL ( Wang et al., 2025d ) , focuses on aligning the training objective with the model’s multi-step generation trajectory. This framework led to the TraDo series of models, which have not only surpassed strong AR models on reasoning benchmarks but also produced the first dLLM capable of long-chain-of-thought reasoning.

A significant challenge for dLLMs is their slow inference speed, stemming from the iterative nature of the denoising process. To mitigate this, several acceleration methods have been proposed. DPad ( Chen et al., 2025a ) offers a training-free solution by treating future tokens as a dynamic ”scratchpad” and using a sliding window and distance-based pruning to reduce redundant computations, achieving a dramatic speedup, especially for long sequence generation. In contrast, D2F ( Wang et al., 2025c ) introduces a hybrid autoregressive-diffusion paradigm that enables parallel denoising of future text blocks even before preceding ones are fully generated. This approach allows dLLMs to leverage KV-caching and, for the first time, surpass the inference speed of equivalently sized AR models.

Despite these advances, the field of dLLM post-training is still nascent. Systematic exploration of how these techniques—SFT, RL, and acceleration—interact with one another, and how they scale to models with hundreds of billions of parameters, remains an open and critical area for future research.

## 3 LLaDA2.0 Training Paradigm

Figure ( 2 ) illustrates the holistic training pipeline of LLaDA2.0, a staged and scalable framework designed to transform AR language models into highly efficient diffusion language models. Our paradigm follows a three-stage progression: (1) Continual Pre-training from AR to MDLM, (2) Block Diffusion Pre-training to transition from token-level to block-level diffusion modeling, and (3) Post-training for alignment and task specialization.

The process begins with a strong AR base model. We first perform continual pre-training to adapt this model into an MDLM, where it learns to reconstruct randomly masked tokens in a bidirectional, denoising fashion. This phase bridges the gap between AR and diffusion-based generation while preserving the representational geometry of the original model.

Building upon the trained MDLM, we then introduce block diffusion pre-training , during which the model is further trained to denoise contiguous spans of text—referred to as “blocks”—rather than individual tokens. This shift enables higher computational efficiency and better long-range coherence during generation.