LLaDA2.0: Scaling Up Diffusion Language Models to 100B

arxiv:2512.15745

Original source · Paper JSON

LLMs: architectures, context, training · 2025-12-10

Cites (2) · Cited by (3)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• LLaDA2.0: Scaling Up Diffusion Language Models to 100B
• Abstract
• 1 Introduction
• 2 Related Work
• 2.1 Train dLLMs from scratch
• 2.2 Scaling dLLMs with AR initialization
• 2.3 dLLMs post-training
• 3 LLaDA2.0 Training Paradigm
• 4 Continual Pre-training via Warmup-Stable-Decay (WSD)
• 4.1 Warmup-Stable-Decay Conversion Strategy
• Phase-1: Progressive Block Size Warmup
• Phase-2: Large Scale Stable Training
• Phase-3: Block Size Decay
• Overall Training Objective
• 4.2 Document-level Attention Mask
• 4.3 Top-k Checkpoint Merge
• 5 Post-training
• 5.1 Supervised Fine-Tuning with Block Diffusion
• Padding strategies & Mask ratio bandwidth
• Complementary Masking
• Data Recipe Curation
• 5.2 Confidence-Aware Parallel Training
• 5.3 DPO
• 5.4 Inference
• 6 Evaluation
• 6.1 Setup
• 6.2 Results
• 6.3 Analysis
• Analysis of Inference Hyper-parameters
• Analysis of Context Length
• 7 Training & Inference Infrastructure
• 7.1 Pretraining
• Efficient Block Diffusion Training
• Numerical Stability
• 7.2 Post-Training
• 7.3 Inference Engine
• Inference speed
• 8 Conclusion