Large Language Diffusion Models — part 4/5

Cleaned-text code points [46531, 62139). Bibliography omitted here; source.md includes it.

We concatenate the collected documents in the pre-training corpus and segment the text into fixed-length sequences according to the predefined sequence length.

For SFT, a dynamic sequence length strategy is employed, where | EOS | |\text{EOS}| tokens are appended to the end of shorter pairs to ensure uniform sequence lengths across all samples within each mini-batch. Notably, the padding | EOS | |\text{EOS}| tokens are treated as part of the response, i.e., masked and included in the training objective. The | EOS | |\text{EOS}| tokens are removed from the generated outputs during sampling. This strategy ensures that the model learns to control the length of its responses by generating | EOS | |\text{EOS}| , enabling the response length to align effectively with the given prompt.

In addition, for n n -turn dialogues ( p 0 0 , r 0 0 , p 0 1 , r 0 1 , … , p 0 n − 1 , r 0 n − 1 ) (p_{0}^{0},r_{0}^{0},p_{0}^{1},r_{0}^{1},\dots,p_{0}^{n-1},r_{0}^{n-1}) , we treat it as n n single-turn dialogue pairs, i.e., ( p 0 0 , r 0 0 ) , ( p 0 0 ​ r 0 0 ​ p 0 1 , r 0 1 ) , … , ( p 0 0 ​ r 0 0 ​ p 0 1 ​ r 0 1 ​ … ​ p 0 n − 1 , r 0 n − 1 ) (p_{0}^{0},r_{0}^{0}),(p_{0}^{0}r_{0}^{0}p_{0}^{1},r_{0}^{1}),\dots,(p_{0}^{0}r_{0}^{0}p_{0}^{1}r_{0}^{1}\dots p_{0}^{n-1},r_{0}^{n-1}) and randomly sample one. This data partitioning strategy not only equips LLaDA with multi-turn dialogue capabilities but also aligns with the above | EOS | |\text{EOS}| padding strategy.

### B.2 Details about Model Training

This section provides the training details of LLaDA and the corresponding ARM baselines.

Firstly, for efficiency, we trained an ARM and an MDM, both with 1.5B parameters and identical architectures. Additionally, we scaled the MDM to 8B parameters. Due to computational resource constraints, we did not train an 8B autoregressive model with the same architecture. Instead, we utilized our previously trained 7B autoregressive model for comparison. These four models are utilized in the scalability analysis in Sec. 3.1 .

We adopted a Transformer architecture similar to LLaMA [ 6 , 21 ] for the ARMs and MDMs we trained. Specifically, we employ RMSNorm [ 105 ] to stabilize training, use SwiGLU [ 106 ] as the activation function to enhance non-linearity, and integrate RoPE [ 107 ] for more expressive positional encoding. Tab. 5 provides an overview of the model architectures.

For the 1B and 7B ARM baselines, as well as the 1B and 8B LLaDA models, we utilized the AdamW optimizer [ 29 ] with a weight decay of 0.1 and adopted the Warmup-Stable-Decay [ 28 ] learning rate scheduler. The learning rate was linearly increased from 0 to the maximum value over the first 2000 iterations and then held constant. For LLaDA 8B, to ensure stable training, the learning rate was reduced once during pre-training, as detailed in Sec. 2.2 . For the 1B ARM baseline and both the 1B and 8B LLaDA models, the maximum learning rate is set to 4 × 10 − 4 4\times 10^{-4} with a batch size of 1280, without any hyperparameter tuning. For the 7B ARM baseline, the maximum learning rate is set to 4.2 × 10 − 4 4.2\times 10^{-4} with a batch size of 4224, both selected via grid search.

Additionally, we employ the widely used 6 ​ N ​ D 6ND formulation [ 108 , 109 ] to calculate the training FLOPs in Fig. 3 , where N N represents the number of non-embedding parameters, and D D denotes the total number of training tokens. The detailed results corresponding to Fig. 3 are provided in Tab. 18 and Tab. 19 .

Our ARM Baseline 1B LLaDA 1B Our ARM Baseline 7B LLaDA 8B LLaMA3 8B Layers 22 22 28 32 32 Model dimension 2048 2048 4096 4096 4096 Attention heads 32 32 32 32 32 Vocabulary size 126,464 126,464 126,464 126,464 128,000 FFN dimension 5634 5634 13,440 12,288 14,336 Key/Value heads 4 4 8 32 8 Total parameters 1.49 B 1.49 B 6.83 B 8.02 B 8.03 B Non-embedding parameters 0.97 B 0.97 B 5.80 B 6.98 B 6.98 B

### B.3 Ablation on Classifier-free Guidance

This section presents an ablation study on classifier-free guidance (CFG). Theoretical details about CFG can be found in the Appendix A.2 .

For simplicity, we select six representative benchmarks, including ARC-C, HellaSwag, TruthfulQA, WinoGrande, PIQA, and GPQA, and conduct experiments using LLaDA 8B Base. We search the CFG scale in { 0.5 , 1 , 1.5 , 2 } \{0.5,1,1.5,2\} for each task and report the best result. As shown in Tab. 6 , CFG consistently improves the performance of LLaDA. We emphasize that, to ensure a fair comparison with ARMs, CFG is not used in the main results reported in the paper.

ARC-C Hellaswag TruthfulQA WinoGrande GPQA PIQA w/o CFG 45.9 70.5 46.1 74.8 25.2 73.6 w/ CFG 47.9 72.5 46.4 74.8 26.1 74.4

### B.4 Details and Ablation on Sampling Strategies

In this section, we first introduce the different sampling strategies supported by LLaDA. We then present ablation studies to evaluate the performance of these sampling strategies.

BBH GSM8K Math HumanEval MBPP Autoregressive 38.1 63.1 23.6 18.3 33.4 Block Difusion L ′ = 2 L^{\prime}=2 37.3 62.6 25.2 14.6 33.6 L ′ = 4 L^{\prime}=4 40.0 65.7 26.6 15.9 36.0 L ′ = 8 L^{\prime}=8 42.0 68.2 27.7 19.5 39.2 L ′ = 32 L^{\prime}=32 45.7 68.6 29.7 29.9 37.4 Block Diffusion LLaDA L ′ = 2 L^{\prime}=2 48.0 70.0 30.8 26.2 40.0 L ′ = 4 L^{\prime}=4 48.5 70.3 31.3 27.4 38.8 L ′ = 8 L^{\prime}=8 48.6 70.2 30.9 31.1 39.0 L ′ = 32 L^{\prime}=32 48.3 70.3 31.2 32.3 40.0 Pure Diffusion 49.7 70.3 31.4 35.4 40.0

GSM8K Math HumanEval MBPP GPQA MMLU-Pro ARC-C Autoregressive 0 9.5 0 0 0 0 84.4 Block Diffusion 24.6 23.5 17.1 21.2 29.3 32.5 88.1 Block Difusion LLaDA 77.5 42.2 46.3 34.2 31.3 34.8 85.4 Pure Diffusion 69.4 31.9 49.4 41.0 33.3 37.0 88.5

Flexible Sampling Strategies. In Sec. 2.4 , Fig. 2 (c) illustrates the reverse generation process of LLaDA. As shown in Fig. 4 , in addition to the reverse generation process, LLaDA also supports autoregressive and block diffusion [ 31 ] sampling directly after the pre-training or SFT stages, without requiring any further modifications or retraining. Block diffusion sampling applies the origin reverse process within each block and the autoregressive sampling across blocks. In the original block diffusion process, the sequence length varies dynamically. As shown in Fig. 4 (c), LLaDA can also adopt a fixed-length block diffusion strategy, which we refer to as block diffusion LLaDA, also known as semi-autoregressive remasking.

Experimental Setup. We evaluate different sampling strategies using both LLaDA 8B Base and LLaDA 8B Instruct for comprehensive analysis. For LLaDA 8B Base, we use the five benchmarks in Tab. 1 that are evaluated based on sampling rather than likelihood estimation. For LLaDA 8B Instruct, we use the seven metrics in Tab. 2 , excluding MMLU and HellaSwag, since these two tasks only require the model to generate a single token (i.e., A, B, C, or D). In all settings, one token is generated per sampling step. For autoregressive and block diffusion sampling, generation terminates when the | EOS | |\text{EOS}| token is produced. For block diffusion LLaDA (i.e., semi-autoregressive remasking) and pure diffusion sampling, the generation length is fixed at 1024 for LLaDA 8B Base, while for LLaDA 8B Instruct, it is tuned from {64, 256, 512} to balance efficiency and performance. Low-confidence remasking is applied to intra-block diffusion sampling in both block diffusion and block diffusion LLaDA, as well as to pure diffusion sampling. We also test different block lengths for LLaDA 8B Base. For LLaDA 8B Instruct, we only evaluate block length 32 for efficiency, as it yields the best results on LLaDA 8B Base.

Additionally, for LLaDA 8B Instruct, due to heavy padding of | EOS | |\text{EOS}| tokens in the SFT data (as detailed in Sec. B.1 ), we observe that under pure diffusion sampling, the proportion of | EOS | |\text{EOS}| tokens in model outputs becomes very high. This leads to extremely short generations and degrades model performance. To mitigate this issue, for HumanEval, MBPP, GSM8K, Math, and GPQA, we set the confidence score of the | EOS | |\text{EOS}| token to zero during pure diffusion sampling. This adjustment helps maintain an appropriate ratio of | EOS | |\text{EOS}| tokens during generation.

Finally, we conduct ablation studies to analyze the effects of random and low-confidence remasking strategies using the pure diffusion sampling. For efficiency, we use LLaDA 8B Base with generation length and sampling steps set to 256 in this analysis.

Results. As shown in Tab. 7 , for block diffusion sampling, overall performance improves as the block length increases. Moreover, both Tab. 7 and Tab. 8 show that block diffusion sampling consistently outperforms autoregressive sampling, and block diffusion LLaDA sampling further improves upon standard block diffusion sampling. Finally, pure diffusion sampling achieves the best overall performance.

In addition, Tab. 9 shows that the low-confidence remasking strategy consistently outperforms the random remasking strategy. We hypothesize that low-confidence remasking functions similarly to the annealed sampling method used by default in ARMs, improving accuracy by reducing the diversity of generated sentences.

Length BBH GSM8K Math HumanEval MBPP Random Remasking 32.1 21.3 9.2 11.6 21.0 Low-confidence Remasking 45.0 70.0 30.3 32.9 40.2

We discover that autoregressive sampling leads to very poor performance for LLaDA 8B Instruct. This is because each SFT data is a complete sentence, so given a sequence length, LLaDA 8B Instruct tends to generate a full sentence within that length. In contrast, LLaDA 8B Base does not suffer from this issue, as the pre-training data consists of truncated documents (as detailed in Appendix B.1 ) and the model is trained with random sequence lengths (as detailed in Sec. 2.2 ). As a result, when given a short sequence length, LLaDA 8B Base tends to generate only part of a sentence, which can then be used as a prefix to continue generation.

Setting the block length to 8 in Tab. 8 further improves the GSM8K score from 77.5 to 78.6.

### B.5 Ablation on Generated Length

Length BBH GSM8K Math HumanEval MBPP 256 45.0 70.0 30.3 32.9 40.2 512 50.4 70.8 30.9 32.9 39.2 1024 49.7 70.3 31.4 35.4 40.0

In this section, we conduct ablation studies on the generated length.

To ensure fairness, for each setting, we set the number of sampling steps equal to the generated length, ensuring that in each sampling step, just one tokens are transferred from the mask to the text. We conduct experiments using LLaDA 8B Base.

As reported in Tab. 10 , the results are not sensitive to the length hyperparameter.

### B.6 Standard Benchmarks and Evaluation Details

In this section, we introduce the benchmarks we used and present the details of our evaluation process.

Following standard LLM [ 25 , 26 ] evaluation practices, we assess LLaDA across four dimensions:

General ability: MMLU [ 110 ] , BBH [ 111 ] , ARC-C [ 112 ] , Hellaswag [ 113 ] , TruthfulQA [ 114 ] , WinoGrande [ 115 ] and PIQA [ 116 ] .

Math and science ability: GSM8K [ 117 ] , Math [ 118 ] and GPQA [ 119 ] .

Code generation: HumanEval [ 120 ] , HumanEval-FIM [ 121 ] and MBPP [ 122 ] .

Chinese understanding: CMMLU [ 123 ] and C-Eval [ 124 ] .

For all the aforementioned benchmarks, we follow the widely adopted evaluation process [ 125 ] used in LLM assessments, primarily employing conditional likelihood estimation and conditional generation. Specifically, for certain benchmarks, a prompt and multiple candidate answers are provided, and the model is required to compute each candidate’s conditional likelihood. The candidate with the highest likelihood is then selected as the model’s final answer, and accuracy is used as the evaluation metric. For the remaining benchmarks, the model generates responses based on the given prompt, and performance is evaluated using metrics such as exact match and other relevant criteria.

For the base model, we use conditional likelihood estimation for MMLU, CMMLU, C-Eval, ARC-C, Hellaswag, TruthfulQA, WinoGrande, PIQA, and GPQA, while the remaining benchmarks are evaluated using conditional generation. For the instruct model, we evaluate all benchmarks using conditional generation.

For the base model, we use the widely adopted open-source evaluation framework lm-evaluation-harness [ 125 ] , except for the HumanEval-FIM metric, which is not supported by the framework. For HumanEval-FIM on the base model and all evaluation metrics on the instruct model, we use an internal evaluation library. We choose the internal library as lm-evaluation-harness shows greater deviation from the LLaMA3 results reported by Yang et al. [25] , relative to our internal evaluation.

For benchmarks evaluated via conditional likelihood estimation, we use Monte Carlo estimation to approximate Eq. ( 6 ) for LLaDA. Since MMLU, CMMLU, and C-EVAL only require the likelihood of a single token, a single Monte Carlo estimate is sufficient for these benchmarks. For all other benchmarks, we find that 128 Monte Carlo samples are adequate to produce stable results.

For benchmarks evaluated using conditional generation, we apply pure diffusion sampling with a low-confidence remasking strategy to both LLaDA Base and LLaDA Instruct. For LLaDA Base, we set both the generation length and the number of sampling steps to 1024. For LLaDA Instruct, the number of sampling steps is set equal to the answer length, which is configured as follows: 3 for MMLU and HellaSwag, 64 for GPQA, 256 for MBPP and MMLU-Pro, and 512 for HumanEval, GSM8K, Math, and ARC-C. As detailed in Appendix B.4 , for HumanEval, MBPP, GSM8K, Math, and GPQA, we set the confidence of the | EOS | |\text{EOS}| token to zero during sampling for LLaDA Instruct.

### B.7 Analysis of Sampling Efficiency

In this section, we first analyze the sampling efficiency of LLaDA, including both sampling speed and memory consumption. We then discuss potential optimizations to further improve its efficiency.

We use four representative open-ended generation benchmarks for sampling speed analysis: GSM8K, Math, HumanEval, and MBPP. We use the widely adopted throughput metric to measure generation speed, defined as the number of tokens generated per second. We compare LLaDA 8B Base and LLaMA3 8B Base, both using bfloat16 precision. All experiments in this section were conducted on a single A100-80GB GPU with a batch size of 1. For LLaDA, the output length is fixed to 256 tokens across all four benchmarks.

Fig. 5 shows that LLaDA enables a flexible trade-off between generation quality and speed by adjusting the number of sampling steps. Specifically, on the GSM8K and Math datasets, LLaDA 8B Base achieves comparable performance to LLaMA3 8B Base while delivering 1.5 and 1.8 times higher throughput, even though LLaMA3 uses KV Cache and LLaDA operates without any inference optimization techniques. For the HumanEval benchmark, LLaDA 8B Base performs comparably to LLaMA3 8B Base when the throughput is matched. On the MBPP benchmark, LLaDA 8B Base lags behind LLaMA3 8B Base.

For LLaMA3, the acceleration benefit provided by KV caching is notably weaker on the HumanEval dataset, which can be attributed to its relatively short prompt lengths. Specifically, the average prompt lengths for GSM8K, Math, MBPP, and HumanEval are 894, 680, 628, and 132 tokens, respectively.

Input Length Output Length LLaDA 8B LLaMA3 8B w/o KV-Cache LLaMA3 8B w/ KV-Cache 512 512 17.03 16.70 16.32 1024 17.53 17.49 16.43 2048 18.52 20.00 16.73 1024 512 17.53 17.16 16.36 1024 18.01 18.26 16.41 2048 19.02 21.39 16.74