LLaDA2.0: Scaling Up Diffusion Language Models to 100B β€” part 3/4

Cleaned-text code points [31300, 46521). Bibliography omitted here; source.md includes it.

By incorporating both 𝒙 t \bm{x}_{t} and 𝒙 t β€² \bm{x}^{\prime}_{t} into the same training batch, this method provides a deterministic guarantee: every token position across the sequence length L L is presented to the model in its uncorrupted state exactly once within the pair. This not only doubles the effective data utilization from each sample, thereby accelerating convergence, but also entirely eliminates token-level sampling bias. Consequently, the model benefits from a more comprehensive and uniform learning signal at every optimization step, leading to enhanced robustness.

#### Data Recipe Curation

A balanced, high-quality SFT dataset underpins the model’s capabilities, achieved through a strategic composition of tasks spanning three principal pillars: Reasoning, General, and Industrial. The Reasoning pillar hones analytical and logical faculties through mathematics and code generation. The General pillar cultivates linguistic richness and social intelligence via creative and dialogic tasks. The Industrial pillar embeds domain-specific expertise by simulating end-to-end workflows under real-world constraints. This integrated methodology ensures a holistic skill profile, preventing capability skew and enabling fluid shifts between abstract reasoning and applied problem-solving.

### 5.2 Confidence-Aware Parallel Training

To enhance the model’s predictive confidence, which is crucial for efficient parallel decoding, we propose Confidence-Aware Parallel (CAP) Training. We incorporate an auxiliary confidence loss, β„’ conf \mathcal{L}_{\text{conf}} , inspired by dParallel ( Chen et al., 2025b ) . The primary objective, β„’ SFT \mathcal{L}_{\text{SFT}} , ensures correctness but provides diminishing incentive to sharpen the predictive distribution for tokens that are already correctly predicted. The confidence loss addresses this by selectively minimizing the entropy of the model’s output distribution, p ΞΈ ​ ( 𝒙 0 | 𝒙 t , 𝒄 ) p_{\theta}(\bm{x}_{0}|\bm{x}_{t},\bm{c}) , but only for the subset of tokens that are correctly predicted in a given step. This compels the model to increase its certainty on its correct predictions. The final training objective is a weighted combination of the two losses: β„’ ⁑ ( ΞΈ ) = β„’ SFT ​ ( ΞΈ ) + Ξ» ​ β„’ conf ​ ( ΞΈ ) , \mathcal{L}(\theta)=\mathcal{L}_{\text{SFT}}(\theta)+\lambda\mathcal{L}_{\text{conf}}(\theta), (6) where Ξ» \lambda is a hyperparameter that balances the two objectives. As illustrated in Figure 3 , CAP training effectively improves the decoding efficiency of LLaDA2.0-flash while maintaining competitive compression performance, demonstrating a favorable trade-off between generation quality and inference speed.

### 5.3 DPO

Building upon the SFT stage, we further align the policy model Ο€ ΞΈ \pi_{\theta} with human intent using Direct Preference Optimization. To support this, we constructed a comprehensive dataset comprising 1.5 million preference pairs across diverse domains, including general knowledge, mathematics, and instruction following. To ensure a stable transition in optimization, the learning rate for the DPO stage is initialized consistently with the final learning rate of the preceding SFT phase.

Since the policy model Ο€ ΞΈ \pi_{\theta} is trained to reconstruct clean tokens 𝒙 0 \bm{x}_{0} from noisy blocks 𝒙 t \bm{x}_{t} conditioned on context 𝒄 \bm{c} , the standard DPO formulationβ€”which requires exact log-likelihoodsβ€”is intractable. Following established practices for diffusion models, we substitute the conditional log-likelihoods with their ELBO. We first define the conditional Block Diffusion ELBO, B BDLM ​ ( ΞΈ , 𝒙 | 𝒄 ) B_{\text{BDLM}}(\theta,\bm{x}|\bm{c}) , for a response 𝒙 \bm{x} . This term mirrors the inner objective of our SFT loss (equation 5 ) and is estimated via a single Monte Carlo sample over timesteps and noise: B BDLM ( ΞΈ , 𝒙 | 𝒄 ) = 𝔼 t , 𝒙 t [ Ξ± t β€² 1 βˆ’ Ξ± t βˆ‘ k = 1 K βˆ‘ i = 1 L B πŸ™ [ x t , k i = [MASK] ] log p ΞΈ ( x k i | 𝒄 , 𝒙 < k , 𝒙 t , k ) ] . B_{\text{BDLM}}(\theta,\bm{x}|\bm{c})=\mathbb{E}_{t,\bm{x}_{t}}\left[\frac{\alpha_{t}^{\prime}}{1-\alpha_{t}}\sum_{k=1}^{K}\sum_{i=1}^{L_{B}}\mathbb{1}[x_{t,k}^{i}=\text{[MASK]}]\log p_{\theta}(x^{i}_{k}|\bm{c},\bm{x}_{<k},\bm{x}_{t,k})\right]. (7) Given a preference pair ( 𝒙 w , 𝒙 l ) (\bm{x}_{w},\bm{x}_{l}) , where 𝒙 w \bm{x}_{w} is the preferred response and 𝒙 l \bm{x}_{l} is the dispreferred response, the DPO objective maximizes the margin between the ELBO estimates of the policy Ο€ ΞΈ \pi_{\theta} and the frozen reference model Ο€ ΞΈ ref \pi_{\theta_{\text{ref}}} (initialized from the post-SFT model). The final loss function is defined as: β„’ DPO ​ ( ΞΈ ) = βˆ’ 𝔼 ( 𝒄 , 𝒙 w , 𝒙 l ) ∼ π’Ÿ ​ [ log ⁑ Οƒ ⁑ ( Ξ² ⁑ [ Ξ” ​ B ​ ( 𝒙 w | 𝒄 ) βˆ’ Ξ” ​ B ​ ( 𝒙 l | 𝒄 ) ] ) ] , \mathcal{L}_{\text{DPO}}(\theta)=-\mathbb{E}_{(\bm{c},\bm{x}_{w},\bm{x}_{l})\sim\mathcal{D}}\left[\log\sigma\left(\beta\left[\Delta B(\bm{x}_{w}|\bm{c})-\Delta B(\bm{x}_{l}|\bm{c})\right]\right)\right], (8) where Ξ” ​ B ​ ( 𝒙 | 𝒄 ) = B BDLM ​ ( ΞΈ , 𝒙 | 𝒄 ) βˆ’ B BDLM ​ ( ΞΈ ref , 𝒙 | 𝒄 ) \Delta B(\bm{x}|\bm{c})=B_{\text{BDLM}}(\theta,\bm{x}|\bm{c})-B_{\text{BDLM}}(\theta_{\text{ref}},\bm{x}|\bm{c}) represents the ELBO advantage of the policy over the reference model, and Ξ² \beta is a hyperparameter (set to 0.1) that controls the deviation from the reference policy.

### 5.4 Inference

We sample one block at a diffusion step, conditioned on previously sampled blocks p ΞΈ ​ ( 𝒙 s b | 𝒄 , 𝒙 t < b ) p_{\theta}(\bm{x}^{b}_{s}|\bm{c},\bm{x}^{<b}_{t}) . The generation of each block is itself a multi-step iterative refinement process. At each step, candidate tokens are sampled for all remaining unfilled positions within the block. A hybrid acceptance strategy is then employed: we first accept all tokens whose sampling probability exceeds a predefined confidence β€˜thresholdβ€˜. If an insufficient number of tokens meet this criterion, a low-confidence fallback is triggered, where we instead accept a fixed number of the most probable tokens regardless of their absolute confidence. This dual mechanism ensures steady generation progress.

## 6 Evaluation

### 6.1 Setup

To comprehensively evaluate the quality of instruction-tuned models, we employ a diverse suite of benchmarks categorized into five dimensions:

β€’ Knowledge : MMLU ( Hendrycks et al., 2020 ) , MMLU-Pro ( Wang et al., 2024 ) , GPQA-Diamond ( Rein et al., 2024 ) , ARC ( Clark et al., 2018 ) , CMMLU ( Li et al., 2023a ) C-Eval ( Huang et al., 2023 ) , GAOKAO-Bench ( Zhang et al., 2023 ) , SciBench ( Wang et al., 2023 ) , PHYBench ( Qiu et al., 2025 ) , TriviaQA ( Joshi et al., 2017 )

β€’ Reasoning : SQuAD 2.0 ( Rajpurkar et al., 2018 ) , DROP ( Dua et al., 2019 ) , KOR-Bench ( Ma et al., 2024 ) , HellaSwag ( Zellers et al., 2019 ) , BIG-Bench Hard ( Suzgun et al., 2023 ) , BIG-Bench Extra Hard ( Kazemi et al., 2025 ) , MuSR ( Sprague et al., 2023 ) , ZebraLogic ( Lin et al., 2025 ) , PrOntoQA ( Saparov and He, 2022 ) , PIQA ( Bisk et al., 2020 ) , OCNLI ( Hu et al., 2020 ) , BIG-Bench Hard-CN ( team, 2023c )

β€’ Coding : CRUXEval ( Gu et al., 2024 ) , MBPP ( Austin et al., 2021 ) , MultiPL-E ( Cassano et al., 2023 ) , HumanEval ( Chen et al., 2021 ) , BigCodeBench ( Zhuo et al., 2024 ) , LiveCodeBench ( Jain et al., 2024 ) , Spider ( Yu et al., 2018 ) , BIRD ( Li et al., 2023b ) , HumanEval+ ( Liu et al., 2023 ) , MBPP+ ( Liu et al., 2023 ) , HumanEvalFix ( Muennighoff et al., 2023 ) , Aider ( team, 2023a ) , HumanEval-CN ( team, 2023c )

β€’ Math : GSM8K ( Cobbe et al., 2021 ) , MATH ( Hendrycks et al., 2021 ) , OlympiadBench ( He et al., 2024 ) , AIME 2025 ( AIME, 2025 ) , Omni-MATH ( Gao et al., 2024 ) , HARDMath2 ( Roggeveen et al., 2025 ) , GSM-Plus ( Li et al., 2024 ) , CMATH ( Wei et al., 2023 )

β€’ Agent & Alignment : BFCL ( Patil et al., 2025 ) , IFEval ( Zhou et al., 2023 ) , CodeIF-Bench ( Wang et al., 2025b ) , Nexus Function Calling Benchmark ( team, 2023b )

This extensive evaluation suite, comprising a total of 47 benchmarks, provides a holistic foundation for assessing model capabilities. In our experiments, we compare the LLaDA2.0 series against strong open-source auto-regressive (AR) models.

For all LLaDA2.0 models, we utilize a temperature of 0.0, a block size of 32, and a decoding threshold of 0.95.

### 6.2 Results

The overall results, presented in the following tables, indicate that the LLaDA2.0 architecture is not only highly competitive, but also shows a promising trend of closing the performance gap with, and even surpassing, AR models in specific key areas. Our models consistently demonstrate strong, and often superior, performance in complex, structured tasks. For instance, LLaDA2.0-mini already outperforms a comparable AR model (Qwen3-8B) in the domains of Reasoning, Coding, and Math. This signal is amplified in our larger model, as LLaDA2.0-flash achieves parity with the powerful Qwen3-30B-A3B-Instruct-2507 and establishes a lead in the critical Coding and Agent domains. This suggests that as diffusion models scale, their inherent strengths in structured generation and tool use become increasingly apparent.

As shown in Table 1 , LLaDA2.0-mini achieves a competitive average score of 64.34, closely approaching its AR peer, Ling-mini-2.0 (65.77). This demonstrates the fundamental viability of the diffusion approach. More importantly, it shows promising signals in complex tasks, outperforming its direct competitor on reasoning benchmarks like SQuAD 2.0 (86.50) and demonstrating more robust instruction following on IFEval (80.78). Its strong performance in coding tasks such as HumanEval (86.59) further suggests an early aptitude for structured generation.

This potential becomes even more evident with our larger model, LLaDA2.0-flash . As shown in Table 2 , with an average score of 73.18, it stands firmly on par with strong AR models such as Qwen3-30B-A3B-Instruct-2507 (73.60). Crucially, LLaDA2.0-flash begins to exhibit clear advantages in complex generative tasks, a sign that the diffusion architecture may hold inherent strengths. In the critical domain of coding, it consistently outperforms its AR peers, scoring higher on HumanEval (94.51), MBPP (88.29) and MultiPL-E (74.87). This trend of surpassing AR models also extends to agent capabilities (BFCL v3: 75.43) and advanced mathematics (AIME 2025: 60.00).

In conclusion, the LLaDA2.0 series successfully demonstrates that diffusion-based language models are a powerful and scalable alternative to the dominant auto-regressive paradigm. While rapidly narrowing the gap on general benchmarks, they are already showcasing the potential to surpass traditional architectures in complex, structured domains like code generation and tool use. This positions diffusion models as a highly promising direction for the future of language generation.

### 6.3 Analysis

#### Analysis of Inference Hyper-parameters

In addition to our main evaluation, we conducted a brief analysis to tune key inference hyperparameters. To ensure efficiency, this analysis was performed on our LLaDA2.0-mini model, using a representative subset of our benchmarks to understand the trade-off between generation quality (score) and inference speed (measured as TPF - Tokens Per Forward; higher is faster).

Denoising Threshold . We first investigate the impact of the Denoising Threshold. While keeping the Block Size fixed at 32, we varied the threshold and observed its effect on quality and speed. As shown in Figure 5 , the results reveal a clear trade-off. A threshold of 0.95 achieved the highest quality score (70.15) at the cost of the lowest inference speed (2.55 TPF). Lowering the threshold to 0.85 boosted the speed to its peak (3.31 TPF), but led to an unacceptable degradation in quality, with the score dropping to 67.90.

Block Size . Subsequently, we analyze the effect of Block Size. We set the Denoising Threshold to 0.95, the optimal value identified in the prior experiment. The results in Figure 5 demonstrate a similar trade-off. A block size of 16 yielded the highest score (70.26) but with the slowest inference (2.44 TPF). In contrast, increasing the block size to 32 substantially improved the speed to 2.55 TPF with only a marginal quality drop to 70.15. Further increasing the block size to 64 proved suboptimal, as it degraded both score and speed relative to the size-32 setting. Therefore, a block size of 32 emerges as the most compelling choice, offering a significant speed-up for a negligible performance cost.

In summary , based on this analysis, the configuration for our main evaluation is well-supported. The Denoising Threshold of 0.95 is the clear choice for maximizing quality. For block-size, the setting of 32 represents an optimal balance, providing the highest throughput with virtually no sacrifice in performance compared to the slightly higher-scoring but slower setting of 16.

#### Analysis of Context Length

To rigorously validate our model’s performance across various context lengths, we conducted a series of evaluations using the RULER benchmark.

As shown in Figure 5 , both models demonstrate strong performance and stability within context length of 32k. The LLaDA2.0-flash model is particularly robust, maintaining a score above 93 across all lengths from 4k to 32k. The LLaDA2.0-mini model also achieves high scores, starting at 93.29 for 4k but showing a degradation to 83.94 at 32k.

To test the models’ extrapolation capabilities, we extended the context length to 64k. This was achieved by employing dynamic RoPE scaling during inference, specifically using the YaRN method with a scaling factor of 2.0. However, this extension resulted in a performance degradation for both models, demonstrating a clear trade-off between context length extension and task accuracy.

In summary , this evaluation highlights two key findings: (1) The LLaDA2.0 models are exceptionally robust for long-context tasks within their native 32k window. (2) They can be successfully extended to handle 64k sequences via YaRN scaling, providing flexibility for extreme-length applications, albeit with a predictable performance cost.

## 7 Training & Inference Infrastructure

### 7.1 Pretraining

We adopt Megatron-LM ( Shoeybi et al., 2019 ) as the pretraining backend to enable efficient training of a 100B-parameter model with long sequences, leveraging data parallelism (DP), pipeline parallelism (PP), tensor parallelism (TP), context parallelism (CP), and expert parallelism (EP), as Figure 6 shows. To ensure consistency of masked tokens, we generate masked tokens on a single model-parallel (MP, that is, TP and PP) rank and then broadcast to all other ranks within the MP ranks.

#### Efficient Block Diffusion Training

For flexible support of arbitrary block diffusion attention mask, we utilize cuDNN as the backend for the attention mechanism. This approach achieves more than 1.3x end-to-end speedup and over 90% memory savings in the attention layer compared to the unfused attention implementation in TransformerEngine when training LLaDA2.0-mini. We further apply a zig-zag partitioning strategy to the block diffusion attention mask to achieve effective load balancing across the CP group.

#### Numerical Stability