Large Language Diffusion Models — part 3/5
Cleaned-text code points [30597, 46531). Bibliography omitted here; source.md includes it.
Another approach replaces continuous diffusion with discrete processes featuring new forward and reverse dynamics, leading to numerous variants [ 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 ] . The original diffusion model paper [ 38 ] introduced both continuous-state and discrete-state transition kernels under a unified diffusion framework. Austin et al. [16] was among the pioneering works that introduced discrete diffusion models into language modeling, demonstrating the feasibility of this approach. Lou et al. [17] showed that masked diffusion, as a special case of discrete diffusion, achieves perplexity comparable to or surpassing ARMs at GPT-2 scale. Shi et al. [18] , Sahoo et al. [19] , Ou et al. [20] established fundamental theoretical results, which motivated our model design, training, and inference (see Appendix A for details). Nie et al. [27] introduced the scaling laws for MDMs in language modeling and explored how MDMs can be leveraged for language tasks such as question answering at the GPT-2 scale. Gong et al. [72] demonstrated the potential of fine-tuning an ARM within the MDM framework. However, the improvements observed by Gong et al. [72] are limited to specific metrics, and their approach does not address the performance achievable through pure diffusion-based training. Concurrent work [ 73 ] demonstrates the potential of diffusion language models in code generation and highlights their advantages in inference efficiency. Nonetheless, as it is a closed-source product, specific details such as training procedures and sampling methods remain unknown.
In comparison, this study scales MDM to an unprecedented size of 8B parameters from scratch, achieving performance comparable to leading LLMs such as LLaMA 3.
Additionally, a parallel line of work on image generation [ 23 , 74 , 75 ] aligns well with the application of MDMs to text data. Moreover, MDMs have also shown promise in other domains such as protein generation [ 76 , 77 ] , where they have achieved promising results. Notably, a series of studies [ 31 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 ] have explored techniques such as architectural optimization, distillation, and sampling algorithm design to accelerate MDMs sampling.
## 5 Conclusion and Discussion
We introduce LLaDA, a diffusion language model trained from scratch with an unprecedented scale of 8B parameters. LLaDA demonstrates strong capabilities in scalability, in-context learning, and instruction-following, achieving performance comparable to strong LLMs such as LLaMA3. In addition, LLaDA offers unique advantages, such as bidirectional modeling and enhanced robustness, effectively addressing the relevant limitations of existing LLMs. Our findings show the promise of diffusion models for language modeling at scale and challenge the common assumption that these essential capabilities are inherently tied to ARMs. These results represent a new paradigm for language modeling and uncover novel insights, demonstrating a high degree of scientific innovation.
Limitations. While promising, the full potential of diffusion models remains to be fully explored. Several limitations of this work present significant opportunities for future research. The generation length is a user-specified hyperparameter. Although LLaDA is insensitive to this hyperparameter as detailed in Appendix B.5 , we believe that adopting an adaptive generation length would offer a more efficient solution. Due to computational constraints, direct comparisons between LLaDA and ARMs—such as training on identical datasets—were restricted to a computational budget of less than 10 23 10^{23} FLOPs. To allocate resources for training the largest possible LLaDA model and showcasing its potential, we were unable to scale the ARM baseline to the same extent. Moreover, no specialized attention mechanisms or position embeddings were designed for LLaDA, nor were any system-level architectural optimizations such as KV cache applied. On the inference side, more efficient and controllable [ 37 , 88 , 89 ] sampling algorithms remain preliminary. Furthermore, LLaDA has yet to undergo alignment with reinforcement learning [ 90 , 91 ] , which is crucial for improving its performance and alignment with human intent.
Looking ahead, both the model scale and the amount of training data for LLaDA remain smaller than those of leading ARM counterparts [ 6 , 26 , 92 , 93 , 94 , 95 ] , highlighting the need for further scaling to fully evaluate its capabilities. In addition, LLaDA’s ability to process multi-modal data remains unexplored. Its impact on prompt tuning techniques [ 96 ] and integration into agent-based systems [ 97 , 98 ] is still not fully understood. Finally, a systematic investigation into post-training for LLaDA (e.g., O1-like systems [ 99 , 100 ] ) is needed to further unlock the potential of diffusion language models.
## Acknowledgements
This work was supported by the National Natural Science Foundation of China (No. 92470118); Beijing Natural Science Foundation (No. L247030); Beijing Nova Program (No. 20220484044); and Ant Group Research Fund.
## Appendix A Formulation of Masked Diffusion Models
### A.1 Training
MDMs [ 16 , 17 , 18 , 19 , 20 ] define the model distribution p θ ( x 0 ) p_{\theta}(x_{0}) in a manner distinct from autoregressive models.
These models introduce a forward process { x t } \{x_{t}\} indexed by a time t ∈ [ 0 , 1 ] t\in[0,1] . This process gradually and independently masks all tokens in the sequence x 0 x_{0} . At time t = 0 t=0 , the data point x 0 x_{0} is fully observed with no masks, while for t ∈ ( 0 , 1 ] t\in(0,1] , x t x_{t} represents latent variables with varying mask ratios in expectation.
Formally, the conditional distribution of x t x_{t} given x 0 x_{0} is defined by a fully factorized form: q t | 0 ( x t | x 0 ) = ∏ i = 1 L q t | 0 ( x t i | x 0 i ) , \displaystyle q_{t|0}(x_{t}|x_{0})=\prod_{i=1}^{L}q_{t|0}(x_{t}^{i}|x_{0}^{i}), (7) where the conditional distribution for each token is given by: q t | 0 ( x t i | x 0 i ) = { 1 − t , x t i = x 0 i , t , x t i = M . \displaystyle q_{t|0}(x_{t}^{i}|x_{0}^{i})=\begin{cases}1-t,&x_{t}^{i}=x_{0}^{i},\\ t,&x_{t}^{i}=\textrm{M}.\end{cases} (8)
Here, M denotes the mask token. Intuitively, each token either remains unchanged or is masked, with the probability of being masked increasing linearly as t t progresses from 0 0 to 1 1 . At t = 1 t=1 , all tokens are guaranteed to be masked, meaning that x 1 x_{1} follows a Dirac distribution concentrated on a sequence of fully masked tokens. Notably, the linear masking probability is analogous to but distinct from, the noise schedule in continuous diffusion models [ 38 , 39 , 40 ] . This linearity is motivated by the assumption that the information in the text is proportional to the number of tokens on average, making it reasonable to lose information linearly during the forward process.
The forward process is not only reversible but also corresponds to a reverse process that is fully factorized across all tokens. The reverse process, from time t = 1 t=1 to 0 0 , generates new data from sequences of fully masked tokens. The conditional distribution for the reverse process, for 0 ≤ s < t ≤ 1 0\leq s<t\leq 1 , is factorized as: q s | t ( x s | x t ) = ∏ i = 1 L q s | t ( x s i | x t ) , \displaystyle q_{s|t}(x_{s}|x_{t})=\prod_{i=1}^{L}q_{s|t}(x_{s}^{i}|x_{t}), (9) where the conditional distribution for each token is: q s | t ( x s i | x t ) = { 1 , x t i ≠ M , x s i = x t i , s t , x t i = M , x s i = M , t − s t q 0 | t ( x s i | x t ) , x t i = M , x s i ≠ M , 0 , otherwise . \displaystyle q_{s|t}(x_{s}^{i}|x_{t})=\begin{cases}1,&x_{t}^{i}\neq\textrm{M},\,x_{s}^{i}=x_{t}^{i},\\ \frac{s}{t},&x_{t}^{i}=\textrm{M},\,x_{s}^{i}=\textrm{M},\\ \frac{t-s}{t}q_{0|t}(x_{s}^{i}|x_{t}),&x_{t}^{i}=\textrm{M},\,x_{s}^{i}\neq\textrm{M},\\ 0,&\textrm{otherwise}.\end{cases} (10) Thus, the key function to estimate is the conditional distribution q 0 | t ( x s i | x t ) q_{0|t}(x_{s}^{i}|x_{t}) , which predicts the original token if it is masked in the input x t x_{t} . This is analogous to the data prediction form in continuous diffusion models.
As proven in [ 20 ] , an equivalent yet time-free parameterization can be derived as: q 0 | t ( x s i | x t ) = p data ( x 0 i | x t UM ) , ∀ i such that x t i = M , \displaystyle q_{0|t}(x_{s}^{i}|x_{t})=p_{\textrm{data}}(x_{0}^{i}|x_{t}^{\textrm{UM}}),\quad\forall i\textrm{ such that }x_{t}^{i}=\textrm{M}, (11) where x t UM x_{t}^{\textrm{UM}} denotes the collection of unmasked tokens in x t x_{t} , which is identical to the corresponding tokens in the original data x 0 x_{0} since unmasked tokens are solely determined by x 0 x_{0} and are independent of time t t . Intuitively, this implies that estimating the data prediction function is equivalent to estimating the conditional distributions on clean data, which is time-invariant. Consequently, the time t t need not be provided as input to the parametric model.
Although the development of masked diffusion is nontrivial, the implementation is straightforward. We first introduce the mask predictor , a parametric model p θ ( ⋅ | x t ) p_{\theta}(\cdot|x_{t}) (e.g., a Transformer without causal mask), which takes x t x_{t} for any t t as input and predict all masked tokens simultaneously. Then, we define the model distribution p θ ( x 0 ) p_{\theta}(x_{0}) as follows: starting with x 1 x_{1} as a sequence of fully masked tokens, we simulate an approximate reverse process parameterized by p θ ( ⋅ | x t ) p_{\theta}(\cdot|x_{t}) from t = 1 t=1 to 0 0 . The marginal distribution induced at t = 0 t=0 then represents the model distribution p θ ( x 0 ) p_{\theta}(x_{0}) .
Formally, the mask predictor is trained using a cross-entropy loss with masking: ℒ ( θ ) ≜ − 𝔼 t , x 0 , x t [ 1 t ∑ i = 1 L 1 [ x t i = M ] log p θ ( x 0 i | x t ) ] , \displaystyle\mathcal{L}(\theta)\triangleq-\mathbb{E}_{t,x_{0},x_{t}}\left[\frac{1}{t}\sum_{i=1}^{L}\textbf{1}[x_{t}^{i}=\textrm{M}]\log p_{\theta}(x_{0}^{i}|x_{t})\right], (12) where x 0 x_{0} is sampled from the training data, t t is sampled uniformly from [ 0 , 1 ] [0,1] , and x t x_{t} is sampled from q t | 0 ( x t | x 0 ) q_{t|0}(x_{t}|x_{0}) . The indicator function 1 [ ⋅ ] \textbf{1}[\cdot] ensures that the cross-entropy loss is computed only for masked tokens. In Shi et al. [18] , Sahoo et al. [19] , Ou et al. [20] , it has been proven that the loss function ℒ ( θ ) \mathcal{L}(\theta) is an upper bound on the negative log-likelihood of the model distribution: − 𝔼 x 0 ∼ p data ( x 0 ) [ log p θ ( x 0 ) ] ≤ ℒ ( θ ) . \displaystyle-\mathbb{E}_{x_{0}\sim p_{\textrm{data}(x_{0})}}\left[\log p_{\theta}(x_{0})\right]\leq\mathcal{L}(\theta). (13) In summary, this principled approach trains a generative model by progressively masking tokens during a forward process and learning to recover the data distribution during a reverse process, all under the (approximate) maximum likelihood estimation framework.
### A.2 Inference
The cross-entropy loss in Eq. ( 12 ) has several equivalent forms [ 20 ] . The first one is given by − 𝔼 l ∼ { 1 , 2 , … , L } , x 0 , x l [ L l ∑ i = 1 L 1 [ x l i = M ] log p θ ( x 0 i | x l ) ] , \displaystyle-\mathbb{E}_{l\sim\{1,2,\dots,L\},x_{0},x_{l}}\left[\frac{L}{l}\sum_{i=1}^{L}\textbf{1}[x_{l}^{i}=\textrm{M}]\log p_{\theta}(x_{0}^{i}|x_{l})\right], (14) where l l is uniformly sampled from { 1 , 2 , … , L } \{1,2,\dots,L\} , and x l x_{l} is obtained by uniformly sampling l l tokens from x 0 x_{0} without replacement for masking. Despite masking exactly l l tokens is different from masking each token independently with probability t t , these two masking methods lead to equivalent results in expectation [ 20 ] .
While Eq. ( 12 ) and Eq. ( 14 ) share the same expectation, their variances differ. Intuitively, in Eq. ( 12 ), we expect x t x_{t} to have a fraction of t t tokens masked. However, the randomness of the forward process (i.e., Eq. ( 7 )) often causes deviations, especially when x t x_{t} contains few tokens. In contrast, in Eq. ( 14 ), the fraction of masked tokens in x l x_{l} is deterministically l L \frac{l}{L} . While a theoretical analysis depends on the data distribution, empirical results show that Eq. ( 12 ) requires over 1000 Monte Carlo estimates for stable results, whereas Eq. ( 14 ) achieves stability with only 128 estimates. In addition, we can simply modify Eq. ( 14 ) to its conditional version (i.e., Eq. ( 6 )) based on Eq. ( 5 ).
Any-order autoregressive models (AO-ARM) [ 59 , 101 , 102 ] characterize the joint distribution autoregressively for all possible orders π \pi of the L L variables. To learn such a distribution, an AO-ARM utilizes a weight-sharing neural network to model all univariate conditionals and employs mask tokens to represent absent variables. During training, the expected negative log-likelihood over the uniform distribution of all orders U π U_{\pi} is minimized: − 𝔼 x 0 , π ∼ U π [ ∑ i = 1 L log p θ ( x 0 π ( i ) | x 0 π ( < i ) ; π ) ] . \displaystyle-\mathbb{E}_{x_{0},\pi\sim U_{\pi}}\left[\sum_{i=1}^{L}\log p_{\theta}(x_{0}^{\pi(i)}|x_{0}^{\pi(<i)};\pi)\right]. (15) Intuitively, x 0 π ( < i ) x_{0}^{\pi(<i)} can be understood as a masked token x t x_{t} with index in π ( ≥ i ) {\pi(\geq i)} being masked. It can be further proved that Eq. ( 15 ) is equivalent to Eq. ( 12 ). This connection explains the bidirectional reasoning capabilities of LLaDA, even though it was never used explicitly in the inference procedure.
In addition, Nie et al. [27] introduces unsupervised classifier-free guidance (CFG), a plug-and-play technique that balances alignment with prompts and text diversity. Specifically, unsupervised CFG employs the following modified mask predictor for inference: p ~ θ ( r 0 | p 0 , r t ) ∝ p θ ( r 0 | p 0 , r t ) 1 + w p θ ( r 0 | m , r t ) w , \displaystyle\tilde{p}_{\theta}(r_{0}|p_{0},r_{t})\propto\frac{p_{\theta}(r_{0}|p_{0},r_{t})^{1+w}}{p_{\theta}(r_{0}|m,r_{t})^{w}}, (16) where m m is a mask sequence of the same length as p 0 p_{0} and w w is a tunable hyperparameter that controls the strength of p 0 p_{0} . To ensure a fair comparison with ARMs, we do not apply CFG to LLaDA in the main text. However, as demonstrated in Appendix B.3 , LLaDA is fully compatible with CFG and consistently exhibits improved performance when it is applied.
### A.3 Algorithms
In this section, we present the training and inference algorithms. Specifically, we introduce the pre-training and supervised fine-tuning algorithms in Algorithm 1 and Algorithm 2 , respectively. In addition, the likelihood evaluation algorithm is provided in Algorithm 3 . Finally, we present the reverse generation process in Algorithm 4 and Algorithm 5 , which correspond to the random remasking and the low-confidence [ 23 ] remasking strategy, respectively.
## Appendix B Experiments
### B.1 Data Collection and Preprocessing
In this section, we first introduce the data collection and filtering processes for both pre-training and SFT. We then describe how LLaDA leverages these datasets during training.
Our pre-training corpus is constructed from diverse publicly available sources, including web data, books, academic papers, social media, encyclopedias, mathematics, and code, with approximately 11% Chinese, 61% English, and 28% code. The data cleaning process involves PDF text extraction, deduplication, and harmful content filtering. To further ensure quality, we fine-tune a BERT [ 22 ] model for automated data quality annotation, enabling the selection of higher-quality samples. Our SFT dataset consists of 1 million human-annotated samples and 3.5 million synthetic samples, generated using methods similar to those proposed in Xu et al. [103] , Wei et al. [104] .