Large Language Diffusion Models

arxiv:2502.09992

Original source · Paper JSON

LLMs: architectures, context, training · 2025-02-14

Cites (6) · Cited by (6)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• Large Language Diffusion Models
• Abstract
• 1 Introduction
• 2 Approach
• 2.1 Probabilistic Formulation
• 2.2 Pre-training
• 2.3 Supervised Fine-Tuning
• 2.4 Inference
• 3 Experiments
• 3.1 Scalability of LLaDA on Language Tasks
• 3.2 Benchmark Results
• 3.3 Reversal Reasoning and Analyses
• 3.4 Case Studies
• 4 Related Work
• 5 Conclusion and Discussion
• Acknowledgements
• Appendix A Formulation of Masked Diffusion Models
• A.1 Training
• A.2 Inference
• A.3 Algorithms
• Appendix B Experiments
• B.1 Data Collection and Preprocessing
• B.2 Details about Model Training
• B.3 Ablation on Classifier-free Guidance
• B.4 Details and Ablation on Sampling Strategies
• B.5 Ablation on Generated Length
• B.6 Standard Benchmarks and Evaluation Details
• B.7 Analysis of Sampling Efficiency
• B.8 Evaluation on iGSM Dataset
• B.9 Poem Completion Tasks
• B.10 More Case Studies
• Appendix C Impact Statement
• Instructions for reporting errors