DiffusionGemma Technical Report

arxiv:2608.00146

Original source · Paper JSON

LLMs: architectures, context, training · 2026-07-31

Cites (3) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• DiffusionGemma Technical Report
• Abstract
• 1 Introduction
• Open-weights release.
• Outline.
• 2 Generative Text Modeling with Discrete Diffusion
• 2.1 Probability Paths and Denoising
• 2.1.1 The Forward Process
• 2.1.2 The Backward Denoising Process
• 3 The DiffusionGemma Architecture
• 3.1 Block-Autoregressive Generation
• KV cache initialization.
• Canvas denoising conditioned on KV cache.
• 3.2 The Denoising Framework at Inference
• Canvas initialization.
• The decoder forward pass.
• The denoising iteration.
• Multinomial diffusion.
• Remark on interpretability.
• 3.3 The DiffusionGemma Sampler
• Entropy-bounded token refinement.
• Temperature annealing.
• Adaptive stopping heuristic.
• 3.4 Inference Efficiency Metrics
• 3.5 Retained Autoregressive Capability
• 4 Supervised Finetuning
• 5 Sampler Distillation & Reinforcement Learning
• Training setup.
• Training dynamics & implicit curriculum effect.
• Improved speed-to-intelligence Pareto frontier.
• SD ⋅ \cdot RL specialization in the few-step regime.
• Emergent conciseness.
• 6 Inference Optimizations
• Low batch size serving.
• GPU time breakdown.
• Eliminating CPU-GPU synchronization.
• Throughput for batch size of 1 serving.
• Multi-user throughput.
• Toward real traffic throughput.
• 7 Experimental Results
• 8 Open-Source Downstream SFT
• Finetuning procedure.
• LoRA for parameter-efficient finetuning.
• Case study: Sudoku puzzle solving.
• 9 Practical Advantages of Text Diffusion
• 9.1 Bidirectional Reasoning and Self-Correction
• 9.2 Dynamic and Adaptive Computation
• 9.3 Structured and Constrained Outputs
• 10 Limitations & Known Issues
• 11 Conclusion
• Appendix A Related work
• A.1 Continuous Diffusion for Text
• A.2 Speculative Decoding
• Appendix B Samples Before and After SD ⋅ \cdot RL Training
• Appendix C Additional Open-Source Downstream Finetuning Results
• Sudoku solving trace summary.
• Practical recipe summary.
• Case study: PubMedQA.
• Appendix D Prompt Formatting
• D.1 BOS and EOS Special Tokens
• D.2 Conversational Structuring
• D.3 Thinking Channels
• D.4 Tool and Function Calling Serialization
• Appendix E Mercury 2 Speed Estimation
• Estimation data.
• Estimation model.
• Speed estimates.
• Appendix F Denoising trajectory sampled from DiffusionGemma
• Appendix G More on Practical Advantages of Text Diffusion
• G.1 Bidirectional Reasoning and Self-Correction
• G.2 More on Dynamic and Adaptive Computation
• G.3 Structured and Constrained Outputs
• Instructions for reporting errors