Large Language Diffusion Models — part 2/5

Cleaned-text code points [15520, 30597). Bibliography omitted here; source.md includes it.

We begin with the reverse generation process. As illustrated in Fig. 2 (c), given a prompt p 0 p_{0} , we discretize the reverse process to sample from the model distribution p θ ​ ( r 0 | p 0 ) p_{\theta}(r_{0}|p_{0}) , starting from a fully masked response. The total number of sampling steps is a hyperparameter, which naturally provides LLaDA with a trade-off between efficiency and sample quality, as analyzed in Sec. 3.3 . We employ uniformly distributed timesteps by default. In addition, the generation length is also treated as a hyperparameter, specifying the length of the fully masked sentence at the beginning of the sampling process. After generation, tokens appearing after the | EOS | |\text{EOS}| token are discarded. As detailed in Appendix B.5 , since both pre-training and SFT are conducted using datasets with variable lengths, the final results are insensitive to this length hyperparameter.

At an intermediate step from time t ∈ ( 0 , 1 ] t\in(0,1] to s ∈ [ 0 , t ) s\in[0,t) , we feed both p 0 p_{0} and r t r_{t} into the mask predictor and predict all masked tokens simultaneously. Subsequently, we remask s t \frac{s}{t} of the predicted tokens in expectation to obtain r s r_{s} , ensuring that the transition of the reverse process aligns with the forward process for accurate sampling [ 18 , 19 , 20 ] . In principle, the remasking strategy should be purely random. However, inspired by the annealing tricks of sampling in LLMs [ 4 , 30 ] , we adopt a low-confidence remasking strategy, where s t \frac{s}{t} of predicted tokens with the lowest confidence are remarked based on the predictions, same as the approach of Chang et al. [23] .

We mention that LLaDA enables flexible sampling. In particular, it supports autoregressive and block diffusion [ 31 ] sampling directly after the pre-training or SFT processes described above, without requiring any further modifications or training. We provide a detailed analysis in Appendix B.4 . Nevertheless, the diffusion sampling (i.e., the reverse generation process) yields the best performance and is adopted as the default throughout this paper, especially for all experiments presented in Sec. 3 .

For conditional likelihood evaluation, we can naturally utilize the upper bound in Eq. ( 5 ). However, we find that the following equivalent form [ 20 ] exhibits lower variance and is more stable: − 𝔼 l , r 0 , r l [ L l ∑ i = 1 L 1 [ r l i = M ] log p θ ( r 0 i | p 0 , r l ) ] , \displaystyle-\mathbb{E}_{l,r_{0},r_{l}}\left[\frac{L}{l}\sum_{i=1}^{L}\textbf{1}[r_{l}^{i}=\textrm{M}]\log p_{\theta}(r_{0}^{i}|p_{0},r_{l})\right], (6) where L L is the sequence length of r 0 r_{0} , l l is uniformly sampled from { 1 , 2 , … , L } \{1,2,\dots,L\} , and r l r_{l} is obtained by uniformly sampling l l tokens from r 0 r_{0} without replacement for masking.

We present the training and inference algorithms, along with theoretical details, in Appendix A .

## 3 Experiments

We evaluate the scalability, instruction-following, and in-context learning capabilities of LLaDA on standard benchmarks, followed by analyses and case studies to provide a comprehensive assessment.

### 3.1 Scalability of LLaDA on Language Tasks

We first investigate the scalability of LLaDA on downstream tasks in comparison with the ARM baselines we constructed. Specifically, at the 1B scale, we ensured that LLaDA and ARM shared the same architecture, data, and all other configurations. At larger scales, we also report results for LLaDA and ARM models of slightly different sizes trained on the same data due to resource limitations. Please refer to Appendix B.2 for more details. We use the pre-training computational cost as a unified scaling metric. For evaluation, we focused on six standard and diverse tasks.

Fig. 3 shows that LLaDA demonstrates impressive scalability, with its overall trend highly competitive with ARMs. Notably, on tasks such as MMLU and GSM8K, LLaDA exhibits even stronger scalability. Even on relatively weaker tasks like PIQA, the performance gap with ARMs narrows as scale increases. To account for the influence of outliers, we opted not to fit quantitative curves, avoiding potential misinterpretation. Nevertheless, the results clearly demonstrate the scalability of LLaDA.

Considering LLaDA’s advantages on certain benchmarks, we hypothesize that this performance gain stems from a key architectural difference: while autoregressive models optimize only left-to-right conditional probabilities, LLaDA is trained to consider multiple conditioning directions, as detailed in Appendix A.2 , which may offer greater flexibility and lead to better generalization. This hypothesis is motivated by LLaDA’s strong performance on reversal reasoning in Sec. 3.3 and the ablation studies on sampling strategies in Appendix B.4 .

Nie et al. [27] suggests that MDM requires 16 times more computation than ARM to achieve the same likelihood. However, key differences make our findings more broadly applicable. In particular, likelihood is a relatively indirect metric for downstream task performance, and diffusion optimizes a bound of the likelihood, making it not directly comparable to ARM. Additionally, we extended the scaling range from 10 18 ∼ 10 20 10^{18}\sim 10^{20} FLOPs in Nie et al. [27] to 10 20 ∼ 10 23 10^{20}\sim 10^{23} FLOPs in this work.

### 3.2 Benchmark Results

To comprehensively evaluate the in-context learning and instruction-following capabilities of LLaDA 8B, we conducted detailed comparisons with existing LLMs [ 6 , 21 , 25 , 26 , 32 , 33 ] of similar scale. Task selection and evaluation protocols followed existing studies, covering popular benchmarks in general tasks, mathematics, code, and Chinese. Further details are provided in Appendix B.6 . For a more direct comparison, we re-evaluated representative LLMs [ 6 , 21 ] in our implementation.

As shown in Tab. 1 , after pretraining on 2.3T tokens, LLaDA 8B Base demonstrates remarkable performance, surpassing LLaMA2 7B Base on nearly all tasks, and is overall competitive with LLaMA3 8B Base. LLaDA shows advantages in math and Chinese tasks. We conjecture that the strengths stem from the same factors as its relatively weaker performance in some tasks—differences in data quality and distribution, largely due to the closed-source situation of LLM datasets.

LLaDA 8B ∗ LLaMA3 8B ∗ LLaMA2 7B ∗ Qwen2 7B † Qwen2.5 7B † Mistral 7B † Deepseek 7B ¶ Model Diffusion AR AR AR AR AR AR Training tokens 2.3T 15T 2T 7T 18T - 2T General Tasks MMLU 65.9 (5) 65.4 (5) 45.9 (5) 70.3 (5) 74.2 (5) 64.2 (5) 48.2 (5) BBH 49.7 (3) 62.1 (3) 39.4 (3) 62.3 (3) 70.4 (3) 56.1 (3) 39.5 (3) ARC-C 45.9 (0) 53.1 (0) 46.3 (0) 60.6 (25) 63.7 (25) 60.0 (25) 48.1 (0) Hellaswag 70.5 (0) 79.1 (0) 76.0 (0) 80.7 (10) 80.2 (10) 83.3 (10) 75.4 (0) TruthfulQA 46.1 (0) 44.0 (0) 39.0 (0) 54.2 (0) 56.4 (0) 42.2 (0) - WinoGrande 74.8 (5) 77.3 (5) 72.5 (5) 77.0 (5) 75.9 (5) 78.4 (5) 70.5 (0) PIQA 73.6 (0) 80.6 (0) 79.1 (0) - - - 79.2 (0) Mathematics & Science GSM8K 70.3 (4) 48.7 (4) 13.1 (4) 80.2 (4) 85.4 (4) 36.2 (4) 17.4 (8) Math 31.4 (4) 16.0 (4) 4.3 (4) 43.5 (4) 49.8 (4) 10.2 (4) 6.0 (4) GPQA 25.2 (5) 25.9 (5) 25.7 (5) 30.8 (5) 36.4 (5) 24.7 (5) - Code HumanEval 35.4 (0) 34.8 (0) 12.8 (0) 51.2 (0) 57.9 (0) 29.3 (0) 26.2 (0) HumanEval-FIM 73.8 (2) 73.3 (2) 26.9 (2) - - - - MBPP 40.0 (4) 48.8 (4) 23.2 (4) 64.2 (0) 74.9 (0) 51.1 (0) 39.0 (3) Chinese CMMLU 69.9 (5) 50.7 (5) 32.5 (5) 83.9 (5) - - 47.2 (5) C-Eval 70.5 (5) 51.7 (5) 34.0 (5) 83.2 (5) - - 45.0 (5)

LLaDA 8B ∗ LLaMA3 8B ∗ LLaMA2 7B ∗ Qwen2 7B † Qwen2.5 7B † Gemma2 9B † Deepseek 7B ¶ Model Diffusion AR AR AR AR AR AR Training tokens 2.3T 15T 2T 7T 18T 8T 2T Post-training SFT SFT+RL SFT+RL SFT+RL SFT+RL SFT+RL SFT+RL Alignment pairs 4.5M - - 0.5M + - 1M + 0.15M - 1.5M + - General Tasks MMLU 65.5 (5) 68.4 (5) 44.1 (5) - - - 49.4 (0) MMLU-pro 37.0 (0) 41.9 (0) 4.6 (0) 44.1 (5) 56.3 (5) 52.1 (5) - Hellaswag 74.6 (0) 75.5 (0) 51.5 (0) - - - 68.5 (-) ARC-C 88.5 (0) 82.4 (0) 57.3 (0) - - - 49.4 (-) Mathematics & Science GSM8K 69.4 (4) 78.3 (4) 29.0 (4) 85.7 (0) 91.6 (0) 76.7 (0) 63.0 (0) Math 31.9 (0) 29.6 (0) 3.8 (0) 52.9 (0) 75.5 (0) 44.3 (0) 15.8 (0) GPQA 33.3 (5) 31.9 (5) 28.4 (5) 34.3 (0) 36.4 (0) 32.8 (0) - Code HumanEval 49.4 (0) 59.8 (0) 16.5 (0) 79.9 (0) 84.8 (0) 68.9 (0) 48.2 (-) MBPP 41.0 (4) 57.6 (4) 20.6 (4) 67.2 (0) 79.2 (0) 74.9 (0) 35.2 (-)

Notably, we have carefully ruled out the possibility of data leakage by taking GSM8K as an example. First, as shown in Fig. 3 , LLaDA outperformed ARM baselines regarding GSM8K. Moreover, the conclusion remains on a fully unseen GSM8K-like task [ 34 ] in Appendix B.8 .

Further, Tab. 2 compares the performance of LLaDA 8B Instruct with existing LLMs. SFT improved LLaDA’s performance on most downstream tasks. A few metrics, such as MMLU, showed declines, possibly due to the suboptimal quality of the SFT data. Overall, since we did not perform alignment with reinforcement learning (RL), our results are slightly behind LLaMA3 8B Instruct, though the gaps in many metrics remain small. Notably, even with only SFT, LLaDA demonstrates impressive instruction-following abilities, as detailed in Sec. 3.4 . We leave RL-based alignment for future work.

All results in Sec. 3 are based on pure diffusion methods, as they achieve better overall performance than approaches incorporating autoregressive components. Specifically, we use Eq. ( 6 ) for conditional likelihood estimation and apply low-confidence remasking for sampling. For LLaDA 8B Instruct, block diffusion style sampling performs better on GSM8K and Math, with scores of 78.6 and 42.2, compared to 69.4 and 31.9 in Tab. 2 . This gain is due to extensive | EOS | |\text{EOS}| token padding in the SFT data, causing early termination in low-confidence remasking. Please refer to Appendix B.4 for details.

Overall, despite the lack of data transparency, we have made every effort to adopt standardized procedures and introduce diverse tasks, we believe they sufficiently demonstrate the extraordinary capabilities of LLaDA, which is the only competitive non-autoregressive model to our knowledge.

### 3.3 Reversal Reasoning and Analyses

Sampling Process User Lily can run 12 kilometers per hour for 4 hours. After that, she runs 6 kilometers per hour. How many kilometers can she run in 8 hours? LLaDA Multi-round Dialogue Case User Could you please share with me what the initial two lines of the renowned poem ‘The Road Not Taken’? LLaDA Certainly! The initial two lines of the renowned poem “The Road Not Taken" by Robert Frost are: “Two roads diverged in a yellow wood, And sorry I could not travel both." These lines set the stage for the poem, introducing the idea of a speaker facing a choice between two paths. User Please help me translate into Chinese. LLaDA Sure! The first two lines of “The Road Not Taken" by Robert Frost can be translated into Chinese as: “两条路分岔在黄色的树林中，遗憾我不能同时走” User Please translate into German. LLaDA Sure! The first two lines of “The Road Not Taken" by Robert Frost can be translated into German as: “Zwei Wege trennten sich im gelben Wald, und leider konnte ich nicht beide tun." User Please also write a poem about life choices. I require 5 sentences, and the first word of each sentence starts with C as the first letter. LLaDA Certainly! Here’s a poem about life choices, with each sentence starting with the letter “C": Choices cascade through life’s journey, Creating connections with each decision, Careful consideration leads to better paths, Courageous choices shape our destiny, Choosing wisely guides us through the years.

To quantify the reversal reasoning [ 15 ] ability of models, we follow the protocol established in Allen-Zhu and Li [35] . Specifically, we construct a dataset of 496 famous Chinese poem sentence pairs. Given a sentence from a poem, models are tasked with generating the subsequent line (forward) or the preceding line (reversal) without additional fine-tuning. Examples can be found in Section B.9 . This setting provides a straightforward and more realistic evaluation compared to previous studies [ 27 , 36 ] .

As shown in Tab. 4 , LLaDA effectively addresses the reversal curse [ 15 ] , demonstrating consistent zero-shot performance across both forward and reversal tasks. In contrast, both Qwen 2.5 and GPT-4o exhibit a significant gap between the two. The results on forward generation confirm that both ARMs are strong, benefiting from significantly larger datasets and greater computational resources than LLaDA. However, LLaDA outperforms both by a large margin in the reversal task.

Forward Reversal GPT-4o (2024 - 08 - 06) 82.7 34.3 Qwen2.5 - 7B Instruct 75.9 38.0 LLaDA - 8B Instruct 51.8 45.6

We did not design anything special for reversal tasks. Intuitively, LLaDA treats tokens uniformly without inductive bias, leading to balanced performance. See Appendix A.2 for details.

We also analyze the effect of different sampling strategies for LLaDA, including autoregressive sampling, block diffusion [ 31 ] sampling, and pure diffusion sampling, showing that pure diffusion sampling achieves the best overall performance, as detailed in Appendix B.4 .

In addition, we examine LLaDA’s sampling speed and memory consumption, showing that it enables a flexible trade-off between generation quality and speed. See Appendix B.7 for more details.

Classifier-free guidance (CFG) [ 37 , 27 ] is a widely used technique in diffusion models to improve generation quality. To ensure a fair comparison with ARMs, we do not apply CFG to LLaDA in the main text. However, we show that LLaDA is compatible with CFG and consistently benefits from its application. See Appendix B.3 for more details.

### 3.4 Case Studies

We present samples generated by LLaDA 8B Instruct in Tab. 3 , showcasing its instruction-following capabilities. First, the table illustrates LLaDA’s ability to generate coherent, fluent, and extended text in a non-autoregressive manner. Second, it highlights the model’s multi-turn dialogue capability, effectively retaining conversation history and producing contextually appropriate responses across multiple languages. Such chat capabilities of LLaDA are impressive, as it departs from conventional ARMs for the first time, to the best of our knowledge. See more case studies in Appendix B.10 .

## 4 Related Work

Diffusion models [ 38 , 39 , 40 ] have achieved remarkable success in visual domains but remain unverified for large-scale (e.g., models trained with over 10 23 10^{23} FLOPs) language modeling, despite growing interest and extensive research efforts.

A simple approach is to continuousize text data and apply continuous diffusion models directly [ 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 ] . Alternatively, some methods model continuous parameters of discrete distributions instead [ 52 , 53 , 54 , 55 , 56 ] . However, scalability remains a significant challenge for these approaches. For instance, a 1B model may require 64 times the compute of an ARM to achieve comparable performance [ 57 ] .