LLaDA2.0: Scaling Up Diffusion Language Models to 100B — part 4/4

Cleaned-text code points [46521, 51040). Bibliography omitted here; source.md includes it.

During the transition from AR to diffusion models, training can suffer from gradient explosion, especially at high mask ratios within a document. This issue stems from the fact that masked token embeddings are set to zero during AR training, as these tokens are never observed, leading their corresponding weights to gradually decay to zero. A straightforward fix, randomly reinitializing the masked token embeddings upon loading the AR model, may disrupt other well-trained parameters, potentially causing catastrophic forgetting. To mitigate this while preserving pre-trained knowledge, we instead add independent Gaussian noise to the output of the embedding layer for each masked token during the initial iterations of training. This ensures that the L2 norm of the masked token’s embedding remains significant to avoid gradient explosion, thereby stabilizing the training process.

### 7.2 Post-Training

For the post-training phase, we leverage dFactory 2 2 2 https://github.com/inclusionAI/dFactory ( InclusionAI, 2025 ) , a repository providing efficient training recipes for dLLMs. Built upon the VeOmni ( Ma et al., 2025a ) distributed training framework, dFactory allows us to effectively implement complex parallelization schemes. Specifically, our setup for fine-tuning LLaDA2.0 combines Data Parallelism (DP) and Expert Parallelism (EP) to ensure scalable and stable training. To further enhance data throughput and hardware utilization, we adopt a data packing strategy analogous to those used in continued pre-training, which concatenates multiple short sequences into a single longer sequence. This integrated approach provides a robust and high-performance infrastructure for the post-training of our model.

### 7.3 Inference Engine

We adapt dInfer 3 3 3 https://github.com/inclusionAI/dInfer ( Ma et al., 2025b ) --originally built for high-performance diffusion LLM inference--to efficiently support block diffusion inference. This requires the inference engine to leverage optimization techniques traditionally designed for AR models. For instance, the framework can now effectively exploit KV-cache reuse to substantially reduce prefill computation. As block diffusion inference closely resembles auto-regressive generation in the execution pattern, we also incorporated block diffusion inference support into SGLang 4 4 4 https://github.com/sgl-project/sglang/issues/12766 ( Zheng et al., 2024 ) , allowing it to benefit from the same class of system-level optimizations designed for AR models. More mature features in dInfer are undergoing to transport to SGLang.

#### Inference speed

Figure 3 compares the average inference throughput (Tokens Per Second, TPS = #decoding tokens/#total-time) of our optimized LLaDA2.0-flash models against state-of-the-art AR models of similar scale on four reasoning and code-generation benchmarks (HumanEval, MBPP, GSM8K, and CRUXEval). All models are evaluated under a consistent generation setup. For diffusion-based models (LLaDA2.0-flash and LLaDA2.0-flash-CAP), we adopt a threshold decoder with a threshold of 0.95. The AR baselines (Ling-flash-2.0 and Qwen3-30B-A3B-Instruct-2507) are deployed using SGLang, while the diffusion models are served with dInfer, ensuring fair performance comparison in real inference environments. As shown, LLaDA2.0-flash-CAP reaches 535 TPS, outperforming the standard LLaDA2.0-flash (383 TPS) and providing up to 2.1× speed-up over the AR baselines (256 TPS and 237 TPS).

## 8 Conclusion

In this work, we introduced LLaDA2.0, discrete diffusion language models scaling up to 100B total parameters through systematic conversion from auto-regressive models, as well as a set of novel and comprehensive recipes designed to smooth and effectively transform traditional AR language models into highly efficient and performant Masked Diffusion Language Models.

Through extensive evaluations, it validates the feasibility of the training paradigm. The LLaDA2.0-mini and LLaDA2.0-flash models achieve performances that are competitive with their AR counterparts. Slightly surprisingly, LLaDA2.0-flash seems to have demonstrated advantages in complex, structured domains such as code generation, mathematical reasoning, and agentic tool use. These may have opened a new door to future work in the agentic LLM era while solidifying a gaugeable potential of dLLM for test-time scaling.

Future work may point to further scaling of the parameter volume, RL/thinking paradigm and extending the decoding speed to its extreme.