Large Language Diffusion Models — part 5/5

Cleaned-text code points [62139, 76195). Bibliography omitted here; source.md includes it.

Tab. 11 compares of memory consumption between LLaDA 8B Base and LLaMA3 8B Base. To avoid variations in generation length caused by differences in training data, we fix both the input and output token lengths during the memory analysis. For LLaDA, memory usage remains constant regardless of the number of sampling steps. Its memory consumption is comparable to LLaMA3 8B Base without KV cache, but slightly higher than with KV cache.

We emphasize that the goal of this study is not to propose a model that is faster than ARMs. Instead, we aim to show the promise of diffusion models for language modeling at scale and challenge the common assumption that core LLM capabilities such as scalability, in-context learning, and instruction-following are inherently depend on ARMs. A substantial body of research [ 31 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 ] has focused on improving the generation efficiency of MDMs through algorithmic or architectural innovations. We leave similar efficiency-oriented exploration for LLaDA to future work.

### B.8 Evaluation on iGSM Dataset

To further assess the mathematical capabilities of LLaDA, we test its performance on iGSM [ 34 ] , an infinite, synthetic GSM8K-like dataset. iGSM is generated via specific rules, with parameters that control the difficulty of problems (i.e., the number of solution steps). For evaluation consistency, we append "#### $answer" to the final solution, adhering to the GSM8K format. Below is an example with solution steps set to 4:

( Question ) The number of each North Star Elementary’s Cultural Studies Classroom equals 1. The number of each Westridge Elementary’s Dance Studio equals 3 times as much as the sum of each North Star Elementary’s Classroom and each North Star Elementary’s Cultural Studies Classroom. How many Dance Studio does Westridge Elementary have? ( Solution ) Define North Star Elementary’s Cultural Studies Classroom as x; so x = 1. Define North Star Elementary’s Classroom as m; so m = x = 1. Define Westridge Elementary’s Dance Studio as n; w = m + x = 1 + 1 = 2; so n = 3 * w = 3 * 2 = 1 #### 1

Since there are slight differences between GSM8K and iGSM (e.g., the use of a mod 5 algorithmic system), we follow [ 34 ] and provide a system prompt along with four-shot question-answer pairs for each problem.

( Prompt ) You’re an expert at solving elementary math problems involving addition, subtraction, and multiplication. You solve all the problems in a uniform format. All calculations are done modulo 5. For example, 4 + 4 equals 3, 2 + 4 equals 1, 3 + 3 + 3 equals 4, 3 * 3 equals 4, and 2 * 2 equals 4. When providing your solution, please end with ’#### x.’ where x is your final answer, an integer between 0 and 4. You must solve all the problems using the same solution format. Our scenarios involve up to four categories of objects: schools, classrooms, backpacks and stationeries. Each school may contain classrooms, each classroom may contain backpacks, and each backpack may contain stationeries. We can specify quantities, such as ẗhe number of dance studios at each Lakeshore High.Ässume that every entity with the same name has an identical configuration; for example, each Lakeshore High contains the same number of dance studios. Another guiding principle is that what is not mentioned does not exist: when we refer to classrooms at Lakeshore High, we are only discussing the classrooms explicitly mentioned in our scenario. Furthermore, if Lakeshore High is not even mentioned, any classroom within it is automatically considered to be non-existent (i.e. 0).

For solution steps ranging from 4 to 6, we generate 100 questions for each case and report the corresponding accuracy in Table 12 . As shown in the table, LLaDA 8B Base demonstrates significant and consistent advantages over LLaMA3 8B Base on unseen mathematical problems, aligning with the results in Table 1 .

### B.9 Poem Completion Tasks

In this section, we present examples from our poem completion dataset as follows.

Example 1: Prompt: 窈窕淑女的下一句是什么？直接输出句子即可。 Answer: 君子好逑。

Example 2: Prompt: 不拘一格降人才的上一句是什么？直接输出句子即可。 Answer: 我劝天公重抖擞。

### B.10 More Case Studies

In this section, we present additional case studies of LLaDA 8B Instruct. First, Tab. 13 shows the sampling process of the block diffusion LLaDA sampling, while Tab. 14 depicts the sampling process for multi-turn dialogues with random remasking. Additionally, Tab. 15 and Tab. 16 provide further examples of single-turn and multi-turn dialogues. Finally, Tab. 17 presents examples of poem reversal completions where the LLaDA 8B Instruct model succeeds, in contrast to the failure of GPT-4o.

## Appendix C Impact Statement

Our work shows the promise of diffusion models for language modeling at scale and challenges the common assumption that core LLM capabilities such as scalability, in-context learning, and instruction-following are inherently dependent on ARMs. Our findings open new avenues for exploring alternative probabilistic paradigms in natural language processing, with potential applications in conversational AI, code generation, and complex reasoning tasks.

However, diffusion models, like traditional LLMs, raise similar societal concerns. These include the environmental impact of large-scale training, the potential misuse for generating harmful content, and the amplification of biases present in training data. Addressing these challenges is critical to ensuring the responsible development and deployment of diffusion language models.

User What are the benefits of regular exercise for physical and mental health? LLaDA

User I want to have a pet, but I don’t know whether to get a cat or a dog. Do you have any suggestions? LLaDA User I’m very busy at work, so it seems like a cat would be a more suitable option. Do you have any suggestions for getting a cat? LLaDA

Daily life User What’s a piece of advice you’ve received that you’ve found helpful? LLaDA One helpful piece of advice I’ve received is to always be open to learning and growth. I’ve found that being willing to explore new ideas and experiences has helped me become more resilient and adaptable. By embracing a growth mindset, I’ve been able to identify opportunities for improvement and development, both personally and professionally. Daily life User Please recommend me three famous movies LLaDA Certainly! Here are three famous movies that have achieved widespread popularity and critical acclaim: 1. **The Shawshank Redemption** (1994) - Inspired by a novella and directed by Frank Darabont, this film tells the story of Andy Dufresne, a man convicted of murdering his wife, dealing with his harsh life in Shawshank Prison and forming an unlikely friendship with a fellow inmate. 2. **The Godfather** (1972) - Directed by Francis Ford Coppola, this film is based on Mario Puzo’s novel of the same name. It follows the story of the Corleone family, particularly Michael Corleone, who rises to power and takes over the family’s criminal empire. 3. **The Empire Strikes Back** (1980) - Directed by George Lucas, this film continues the story of a young Skywalker in the Star Wars saga. It follows Luke Skywalker as he helps the Rebel Alliance fight against the Galactic Empire and destroy the powerful weapon, the Death Star. All of these films are celebrated for their cinematic achievements, powerful storytelling, and memorable characters. Translation User Please help me translate into Chinese: ‘What is now proved was once only imagined’ LLaDA 现在被证明的曾经只是想象 Translation User Please help me translate into German: ‘What is now proved was once only imagined’ LLaDA Wer heute bestätigt ist, war einst nur immaginiert. Math User Emma decides to run 4 sprints 5 times a week. She runs 75 meters each sprint. How many total meters does she run a week? LLaDA Emma runs 4 sprints each time, so she runs a total of 4 x 5 = 20 sprints a week. Each sprint is 75 meters long, so Emma runs a total of 20 x 75 = 1500 meters a week. The final result is 1500 Code User I am writing python code, I have an array, please help me sort it from largest to smallest LLaDA Certainly! You can sort an array from the largest to smallest in Python using the ‘sort()’ method or the ‘sorted()‘ function. Here’s an example: ### Using ‘sort()’ method: my_array = [3, 1, 4, 1, 5, 2] my_array.sort(reverse=True) print(my_array)

Daily life User How do I prepare for a job interview? LLaDA To prepare for a job interview, you should research the company and the job, practice common interview questions, and dress appropriately. User What are some common interview questions? LLaDA Some common interview questions include “Tell me about yourself” and “Why do you want to work for us”? Daily life User How do I start learning programming? LLaDA To learn programming, you can start by choosing a programming language that you are interested in. Some popular languages include Python, Java, and C++. User Which programming language should I learn first? LLaDA It depends on your interests and goals. Python is a great language for beginners for its simplicity and versatility. User How long does it take to learn Python? LLaDA It can take anywhere from a few weeks to several months to learn Python. User What are some good Python projects for beginners? LLaDA Some good Python projects for beginners include building a simple calculator, a game, and a web application. Math User Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? LLaDA Natalia sold clips to 48 friends in April. She sold half as many clips in May, so she sold 48/2 = 24 clips in May. Altogether, Natalia sold 48 + 24 = 72 clips in April and May. The final result is 72. User Natalia sold three times as many items in June as she did in May. How many clips did Natalia sell altogether in these three months? LLaDA Natalia sold clips to 48 friends in April. She sold half as many clips in May, so she sold 48/2 = 24 clips in May. Natalia sold three times as many items in June as she did in May, so she sold 24 * 3 = 72 clips in June. Altogether, Natalia sold 48 + 24 + 72 = 144 clips in these three months. The final result is 144.

Question1 User ‘但闻人语响’的上一句是什么? GPT-4o “但闻人语响”的上一句是“无可奈何花落去”,这两句出自唐代词人李清照的（武陵春，春晚: 无可奈何花落去，但闻人语响。这首词表达了作者在春天时节的孤独与感伤，借花落和人语声的描写，反映出词人内心的无亲与失落。 LLaDA “但闻人语响”的上一句是“空山不见人” Question2 User ‘夜静春山空’的上一句是什么? GPT-4o “夜静春山空”的上一句是“空山新雨后”.这两句出自唐代诗人王维的（山居秋膜）:空山新雨后，夜静春山空。这首诗描绘了春天山中雨后的寂静与空灵，表现出自然的宁静与和谐美。 LLaDA “夜静春山空”的上一句是“人闲桂花落”

Model Training Tokens FLOPs MMLU CMMLU ARC-C PIQA GSM8K HumanEval LLaDA 1B 37.75B 2.20e20 25.52 25.95 25.17 59.41 1.82 0.00 LLaDA 1B 88.08B 5.13e20 27.11 26.52 26.96 61.86 3.03 1.83 LLaDA 1B 138.41B 8.06e20 29.32 27.13 30.20 63.38 2.35 0.00 LLaDA 1B 239.08B 1.39e21 31.48 30.77 27.99 63.11 3.26 1.22 LLaDA 1B 352.32B 2.05e21 35.86 34.35 31.31 65.34 3.64 3.05 LLaDA 1B 461.37B 2.69e21 31.86 30.98 30.12 65.51 2.35 0.61 LLaDA 8B 62.91B 2.63e21 32.22 28.5 30.20 63.82 3.87 2.44 LLaDA 8B 125.83B 5.27e21 33.39 33.9 34.64 66.54 8.72 3.66 LLaDA 8B 251.66B 1.05e22 42.84 40.59 40.10 69.04 15.31 3.66 LLaDA 8B 377.49B 1.58e22 45.11 43.99 39.25 68.61 25.40 9.76 LLaDA 8B 503.32B 2.11e22 43.57 41.38 42.06 70.24 27.52 9.76 LLaDA 8B 629.14B 2.63e22 48.80 47.13 42.24 72.09 30.10 12.80 LLaDA 8B 679.48B 2.85e22 49.61 48.19 41.30 70.84 26.31 8.54 LLaDA 8B 792.72B 3.31e22 50.88 49.01 42.58 70.51 31.99 6.10 LLaDA 8B 981.47B 4.11e22 49.47 48.10 40.27 71.38 - 6.10 LLaDA 8B 1107.30B 4.64e22 51.13 47.57 41.13 69.26 36.69 10.37 LLaDA 8B 1233.13B 5.16e22 50.52 49.72 45.05 71.49 38.97 9.76 LLaDA 8B 1358.95B 5.69e22 54.61 53.97 49.40 74.05 48.14 17.68 LLaDA 8B 1547.70B 6.48e22 57.38 56.04 49.49 74.59 53.30 20.73 LLaDA 8B 1975.52B 8.27e22 58.52 57.87 50.68 75.35 - 19.51

Model Training Tokens FLOPs MMLU CMMLU ARC-C PIQA GSM8K HumanEval ARM 1B 37.75B 2.20e20 25.47 25.38 30.20 67.36 2.20 4.88 ARM 1B 88.08B 5.13e20 24.67 25.23 33.96 70.02 7.51 10.37 ARM 1B 138.41B 8.06e20 29.25 27.48 33.45 70.29 8.34 9.76 ARM 7B 17.30B 6.02e20 26.92 25.18 21.02 57.18 1.29 1.22 ARM 7B 34.60B 1.20e21 25.83 25.38 24.07 62.84 1.59 2.44 ARM 7B 86.50B 3.01e21 24.41 24.90 25.42 71.11 2.88 7.93 ARM 7B 173.02B 6.02e21 26.20 24.78 26.10 74.27 6.67 9.15 ARM 7B 207.62B 7.23e21 30.36 28.86 31.86 74.48 8.57 12.80 ARM 7B 224.92B 7.83e21 29.49 32.26 31.19 74.37 8.95 8.54 ARM 7B 242.22B 8.43e21 33.62 31.38 34.92 75.41 10.84 9.15 ARM 7B 259.52B 9.03e21 34.11 34.20 32.88 75.19 9.33 10.98 ARM 7B 311.43B 1.08e22 35.66 35.49 36.61 75.14 11.30 10.37 ARM 7B 363.33B 1.26e22 34.54 37.67 34.58 76.55 12.28 14.02 ARM 7B 415.24B 1.45e22 35.37 38.37 35.25 76.39 14.40 12.80 ARM 7B 449.84B 1.57e22 39.51 39.24 34.92 76.82 14.94 14.63 ARM 7B 519.09B 1.81e22 40.30 40.69 37.29 77.15 14.03 14.63 ARM 7B 778.57B 2.71e22 43.33 43.50 38.31 77.53 17.59 14.63 ARM 7B 1038.09B 3.61e22 45.06 46.12 41.69 77.86 20.02 15.85 ARM 7B 1384.12B 4.82e22 47.63 48.18 47.80 76.93 22.82 15.24 ARM 7B 2076.18B 7.23e22 47.68 50.85 44.07 77.37 24.79 14.63 ARM 7B 2214.59B 7.71e22 49.26 52.08 53.56 77.69 27.37 17.07

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .