LLaDA2.0: Scaling Up Diffusion Language Models to 100B — part 2/4

Cleaned-text code points [15776, 31300). Bibliography omitted here; source.md includes it.

Finally, after mastering non-autoregressive generation at both token and block levels, the model undergoes post-training–including SFT and DPO to align its outputs with human intent, instruction-following capability, and downstream application requirements. This stage ensures that the powerful generative backbone developed during diffusion pre-training translates into practical performance gains across diverse tasks.

Overall, LLaDA2.0’s training paradigm emphasizes knowledge inheritance , progressive adaptation , and efficiency-aware design , enabling seamless evolution from AR models to fluent, flexible, and fast diffusion large language models.

## 4 Continual Pre-training via Warmup-Stable-Decay (WSD)

Converting a pre-trained AR language model into a high-performance diffusion language model is fundamentally challenging due to the misalignment in architectural inductive biases and training objectives. While AR models generate tokens sequentially from left to right, diffusion-based models rely on bidirectional context and learn to reconstruct corrupted sequences in arbitrary unmasking orders. A direct objective switch often leads to unstable optimization and severe degradation of pretrained knowledge.

To address this gap, we propose a Warmup–Stable–Decay (WSD) continual pre-training strategy that enables a smooth, stable, and effective transition from AR to dLLM. WSD decomposes the conversion into three coordinated phases: • Warmup : Progressively increase the block size in block diffusion language models (BDLM) to gradually transform the AR model into a full-sequence masked diffusion language model (MDLM).

• Stable : Stabilize and enrich the model’s understanding of diffusion dynamics through large-scale training under the MDLM paradigm.

• Decay : Revert back to a compact BDLM with smaller block sizes to achieve better speed-efficiency trade-offs during inference.

This progressive schedule preserves the AR model’s priors while steadily adapting it to the structural requirements of diffusion modeling.

Moreover, the document-level attention mask is applied throughout training to all input sequences. This mechanism is crucial for handling packed heterogeneous documents, preventing the model from forming spurious connections across unrelated texts, thereby ensuring semantic coherence and improving learning stability within each document. In addition, we adopt a top-k checkpoint merging strategy ( Tian et al., 2025 ) , to enhance generalization by averaging the parameters of the best-performing checkpoints, smoothing the parameter landscape, and yielding a more robust final model with boosted performance.

### 4.1 Warmup-Stable-Decay Conversion Strategy

We begin with the AR base models Ling-mini-2.0 and Ling-flash-2.0 ( Ling et al., 2025 ) , which can be viewed as a special case of BDLM with block size 1. This perspective allows us to treat AR models as the initial BDLM configuration with minimal granularity.

#### Phase-1: Progressive Block Size Warmup

The core idea of the warmup phase is to gradually increase the block size , thereby expanding the receptive field within which the model performs joint denoising. Starting from block size L B = 1 L_{B}=1 , we incrementally scale it up to 4, 32, then 64, and ultimately reach L B = 4096 L_{B}=4096 — at which point the entire sequence is treated as one single block. To avoid fragmented blocks, we require the sequence length to be divisible by the current block size. At the final enlargement, the BDLM becomes equivalent to a standard MDLM that operates over fully masked sequences with global attention. Crucially, each block-size transition is trained on moderate-scale data to ensure smooth adaptation. This progressive enlargement allows the model to smoothly adapt its internal representations to handle larger contextual spans and more complex masking patterns.

#### Phase-2: Large Scale Stable Training

Once the block size reaches 4096 and the model transitions to the MDLM pattern, the “clean” part of the attention computation (see Figure 2 ) no longer needs to be maintained. This significantly reduces the computational cost of attention, allowing data to be processed far more efficiently under the MDLM paradigm. With the model now fully adapted to this regime, the stable training phase focuses on deepening its understanding of diffusion dynamics through extensive training on large-scale corpora. At this stage, the block size is fixed at 4096, effectively making every input a single-block sequence, equivalent to the classical MDLM setting.

#### Phase-3: Block Size Decay

Finally, after large-scale MDLM training, we gradually reduce the block size from 4096 to a small block size (e.g., 32) to convert the model back into an efficient BDLM. This decay process distills the global contextual knowledge learned during MDLM into a compact blockwise structure. By decreasing the block size step-by-step (e.g., starting from 4096 to 2048) rather than abruptly, the model smoothly adapts from global to local conditioning, preserving its semantic understanding while regaining BDLM’s practical benefits such as KV-cache reuse and fast variable-length generation.

#### Overall Training Objective

The optimization objective of BDLM ( Arriola et al., 2025 ) is designed to enable the model to accurately reconstruct the original, uncorrupted tokens within these designated masked blocks using a standard cross-entropy loss. Specifically, we define the training loss during warmup and decay phases (phase-1&3) under the BDLM paradigm as: ℒ BDLM ( θ ) = − 𝔼 t , 𝒙 0 , 𝒙 𝒕 [ α t ′ 1 − α t ∑ k = 1 K ∑ i = 1 L B 𝟙 [ x t , k i = [MASK] ] log p θ ( 𝒙 0 , k i | 𝒙 0 , < k , 𝒙 t , k ) ] , \mathcal{L}_{\text{BDLM}}(\theta)=-\mathbb{E}_{t,\bm{x}_{0},\bm{x_{t}}}\left[\frac{\alpha_{t}^{\prime}}{1-\alpha_{t}}\sum_{k=1}^{K}\sum_{i=1}^{L_{B}}\mathbb{1}[x_{t,k}^{i}=\text{[MASK]}]\log p_{\theta}(\bm{x}_{0,k}^{i}|\bm{x}_{0,<k},\bm{x}_{t,k})\right], (1) where the expectation is over timestep t t , the clean sequence 𝒙 0 \bm{x}_{0} , and its corrupted version 𝒙 t \bm{x}_{t} (tokens masked with probability 1 − α t 1-\alpha_{t} ). Indicator 𝟙 ​ [ ⋅ ] \mathbb{1}[\cdot] ensures predictions are made only for masked tokens, and − α t ′ / ( 1 − α t ) -\alpha_{t}^{\prime}/(1-\alpha_{t}) is the diffusion‑derived time weight. Here K = L total / L B K=L_{\text{total}}/L_{B} is the number of blocks, L B L_{B} the block size, x t , k i x_{t,k}^{i} the i i -th token in block k k , 𝒙 0 , < k \bm{x}_{0,<k} the preceding clean blocks, and 𝒙 t , k \bm{x}_{t,k} the noisy version of the current block.

During the stable training (phase-2) of MDLM (i.e., K=1), the objective simplifies to: ℒ MDLM ( θ ) = − 𝔼 t , 𝒙 0 , 𝒙 t [ α t ′ 1 − α t ∑ i = 1 L 𝟙 [ x t i = [MASK] ] log p θ ( x 0 i | 𝒙 t ) ] . \mathcal{L}_{\text{MDLM}}(\theta)=-\mathbb{E}_{t,\bm{x}_{0},\bm{x}_{t}}[\frac{\alpha^{\prime}_{t}}{1-\alpha_{t}}\sum_{i=1}^{L}\mathbb{1}[x_{t}^{i}=\text{[MASK]}]\log p_{\theta}(x^{i}_{0}|\bm{x}_{t})]. (2)

### 4.2 Document-level Attention Mask

Our training sequences are formed by packing heterogeneous documents into fixed-length segments to maximize throughput. However, this introduces artificial long-range dependencies across semantically unrelated texts. Without careful handling, standard attention would incorrectly attend across document boundaries, leading to contextual confusion and significantly hindering the model’s ability to perform robust bidirectional modeling crucial for denoising.

To mitigate this fundamental challenge and preserve semantic coherence, we redefine the attention mechanism with a specialized block-wise document-level attention mask . This mask ensures that attention operates strictly within document boundaries, preventing cross-document contamination and allowing the model to fully leverage bidirectional context for accurate reconstruction of corrupted blocks. The native Block Diffusion vectorizes the training process to achieve parallel training of blocks, and this mask is applied accordingly. Specifically, for a concatenated sequence x f ​ u ​ l ​ l x_{full} of length 2 ​ L 2L (comprising x t x_{t} followed by x 0 x_{0} ), and assuming tokens i i and j j are already confined to the same document segment (as enforced by the initial document-level mask), the attention mask 𝑴 ∈ { 0 , 1 } 2 ​ L × 2 ​ L \bm{M}\in\{0,1\}^{2L\times 2L} is constructed by dividing each sequence ( x t x_{t} and x 0 x_{0} ) into contiguous blocks. Let b ⁡ ( k ) = ⌊ k / L B ⌋ b(k)=\lfloor k/L_{B}\rfloor denote the block index for token k k given a block size L B L_{B} . The mask is defined as: 𝑴 i ​ j = { 𝟙 b ⁡ ( i ) = b ⁡ ( j ) if ​ i ∈ x t ​ and ​ j ∈ x t 𝟙 b ⁡ ( i ) > b ⁡ ( j − L ) if ​ i ∈ x t ​ and ​ j ∈ x 0 𝟙 b ⁡ ( i − L ) ≥ b ⁡ ( j − L ) if ​ i ∈ x 0 ​ and ​ j ∈ x 0 0 otherwise \bm{M}_{ij}=\begin{cases}\mathbb{1}_{b(i)=b(j)}&\text{if }i\in x_{t}\text{ and }j\in x_{t}\\ \mathbb{1}_{b(i)>b(j-L)}&\text{if }i\in x_{t}\text{ and }j\in x_{0}\\ \mathbb{1}_{b(i-L)\geq b(j-L)}&\text{if }i\in x_{0}\text{ and }j\in x_{0}\\ 0&\text{otherwise}\end{cases} (3) Where i , j ∈ { 0 , 1 , … , 2 ​ L − 1 } i,j\in\{0,1,\dots,2L-1\} are the indices in the full sequence. The first condition ( 𝟙 b ⁡ ( i ) = b ⁡ ( j ) \mathbb{1}_{b(i)=b(j)} ) implements block-diagonal attention within the noisy sequence x t x_{t} . The second ( 𝟙 b ⁡ ( i ) > b ⁡ ( j − L ) \mathbb{1}_{b(i)>b(j-L)} ) enables cross-attention from x t x_{t} to x 0 x_{0} , but only from blocks in x t x_{t} to earlier blocks in x 0 x_{0} . The third ( 𝟙 b ⁡ ( i − L ) ≥ b ⁡ ( j − L ) \mathbb{1}_{b(i-L)\geq b(j-L)} ) imposes a causal block attention pattern within the clean sequence x x , allowing a block to attend to itself and all preceding blocks. The ”otherwise” condition corresponds to a zero matrix, explicitly preventing attention from queries in x 0 x_{0} to keys in x t x_{t} . This allows each block to leverage context from relevant blocks (according to the mask) for reconstruction, capturing inter-block dependencies while maintaining the causal and block-diagonal principles essential for stable diffusion training. During our exploration, we also experimented with other tricks like random-length ( Xie et al., 2025 ) and CART ( Ye et al., 2025 ) . However, the results demonstrate that the document-level attention mask is more fundamental in CPT training compared to these techniques, and it consistently achieves superior performance. As illustrated in Figure 2 , this forms a structured attention layout that balances locality and global document coherence.

For MDLM, the document-level attention mask simplifies to 𝑴 ∈ { 0 , 1 } L × L \bm{M}\in\{0,1\}^{L\times L} , where: 𝑴 i ​ j = { 1 , if i , j belong to the same document , 0 , otherwise . \bm{M}_{ij}=\begin{cases}1,&\text{if $i,j$ belong to the same document},\\ 0,&\text{otherwise}.\end{cases} (4)

### 4.3 Top-k Checkpoint Merge

To further enhance the generalization and robustness of our Block Diffusion Language Model, we employ a top-k checkpoint merging strategy. Upon completion of BDLM pre-training, we identify the top k k best-performing model checkpoints, typically selected based on validation metrics like perplexity. The parameters (weights and biases) of these k k checkpoints are then arithmetically averaged to form a single, unified BDLM. Based on WSM scheduler ( Tian et al., 2025 ) , this merge strategy can effectively ensemble diverse ”knowledge” captured by the model at various optimal or near-optimal training states. This smooths the parameter landscape, mitigates overfitting, and yields a more stable and generalizable model. A key advantage of the WSM approach is its optimizer-agnostic nature, allowing seamless integration without altering the underlying training pipeline. Crucially, this post-training Top-k Merge fundamentally differs from the Exponential Moving Average (EMA). While EMA is an in-training technique that continuously smooths parameters, merging is an offline procedure. It explicitly selects and averages distinct, high-performing model states, consolidating their strengths rather than merely smoothing the final training step.

## 5 Post-training

### 5.1 Supervised Fine-Tuning with Block Diffusion

Following the pre-training phase, the model is aligned to follow user instructions through supervised fine-tuning (SFT). This is achieved by adapting the diffusion training objective to be conditional on an input prompt, 𝒄 \bm{c} . The model is thus trained to generate the desired response 𝒙 0 \bm{x}_{0} by minimizing the following loss function: ℒ SFT ( θ ) = − 𝔼 t , ( 𝒄 , 𝒙 0 ) , 𝒙 t [ α t ′ 1 − α t ∑ k = 1 K ∑ i = 1 L B 𝟙 [ x t , k i = [MASK] ] log p θ ( x 0 , k i | 𝒄 , 𝒙 0 , < k , 𝒙 t , k ) ] . \mathcal{L}_{\text{SFT}}(\theta)=-\mathbb{E}_{t,(\bm{c},\bm{x}_{0}),\bm{x}_{t}}\left[\frac{\alpha_{t}^{\prime}}{1-\alpha_{t}}\sum^{K}_{k=1}\sum_{i=1}^{L_{B}}\mathbb{1}[x_{t,k}^{i}=\text{[MASK]}]\log p_{\theta}(x^{i}_{0,k}|\bm{c},\bm{x}_{0,<k},\bm{x}_{t,k})\right]. (5) Here, the model p θ p_{\theta} learns to predict the original tokens x 0 , k i x^{i}_{0,k} of a clean response from a noisy version 𝒙 t \bm{x}_{t} . The loss is computed only on masked tokens within the current noisy block 𝒙 t , k \bm{x}_{t,k} . To do this, the prediction is conditioned on the prompt 𝒄 \bm{c} , auto-regressive context from prior clean blocks 𝒙 0 , < k \bm{x}_{0,<k} , and the current noisy block 𝒙 t , k \bm{x}_{t,k} that it must denoise.

#### Padding strategies & Mask ratio bandwidth

To ensure compatibility with our block-wise attention mask, we quantize each sequence’s length. Specifically, the original length is rounded up to the nearest multiple of the block size, b b . This process defines an ”effective length” for each sequence, guaranteeing its boundaries align perfectly with the block boundaries required by the attention mechanism.

To optimize the training dynamics, we further implement a “mask ratio bandwidth” strategy. Standard discrete diffusion processes typically sample mask probabilities across the full unit interval, α t ∼ U ⁡ [ 0 , 1 ] \alpha_{t}\sim U[0,1] . However, as identified by Arriola et al. (2025) , extreme masking rates induce high gradient variance while offering minimal learning signal: near-zero masking renders reconstruction trivial, while near-total masking reduces the objective to simply learning data marginals. To mitigate this, we clip the noise schedule, constraining the sampling of mask rates to a bounded interval [ α min , α max ] [\alpha_{\min},\alpha_{\max}] rather than the full range. This bandwidth restriction focuses the training objective on the noise regimes that provide the most informative gradients, thereby stabilizing convergence and improving the model’s generative perplexity.

#### Complementary Masking

Complementary Masking ( Li et al., 2025 ) is a training optimization that enhances the data efficiency of the MDLM objective, ℒ MDLM ​ ( θ ) \mathcal{L}_{\text{MDLM}}(\theta) . The strategy’s core principle is to generate two antithetical training instances from a single source sequence 𝒙 0 \bm{x}_{0} . A primary noised sequence, 𝒙 t \bm{x}_{t} , is formed using a random mask, while a complementary sequence, 𝒙 t ′ \bm{x}^{\prime}_{t} , is simultaneously produced using that mask’s logical inverse 1 1 1 We also try complementary masking in CPT and find it only works fine on corpus less than 100B tokens, while it does not show advantages with more training data, so that we only adopt it in post-training. .