DiffusionGemma Technical Report — part 1/6

Cleaned-text code points [0, 15828). Bibliography omitted here; source.md includes it.

\uselogo

# DiffusionGemma Technical Report

###### Abstract

We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model’s total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model’s support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.

## 1 Introduction

Autoregressive (AR) models dominate the current Large Language Model (LLM) landscape, but their strict left-to-right, token-by-token generation creates a problematic memory bottleneck. When serving many requests simultaneously, one can achieve acceptable throughput by batching, but serving single or low-concurrency requests is fundamentally memory-bound: time spent transferring model weights and context KV cache from memory to the accelerator far exceeds time spent on actual computation. This leaves the accelerator’s compute units under-utilized and limits per-user generation speed. Speculative decoding can improve utilization by drafting a candidate sequence, typically 8 tokens, from a small “drafter” model and feeding the draft to the LLM for verification ( Leviathan et al., 2023 ; Chen et al., 2023a ; Xia et al., 2024 ) . With a draft length of 8 this can produce 3-6 tokens per forward pass (TPF) ( Li et al., 2026 ) . However, the draft-then-verify paradigm is still limited: on one hand, AR drafters are bottlenecked by sequential generation; on the other hand, parallel drafters—building on early blockwise decoding ( Stern et al., 2018 ) and lookahead strategies ( Zhao et al., 2024 ) —exhibit declining acceptance rates at later draft positions ( Cheng et al., 2026 ) .

Text diffusion circumvents this bottleneck by predicting entire blocks of tokens simultaneously (Figure 22 ), effectively shifting execution from a memory-bound regime toward a compute-bound one. Recently, there has been a surge of interest in text diffusion ( Deschenaux and Gulcehre, 2024 ) , with models like Gemini Diffusion ( Google DeepMind, 2025 ) , Mercury ( Inception Labs et al., 2025 ) , LLaDA ( Nie et al., 2026 ; Bie et al., 2025 ) , Seed Diffusion ( Song et al., 2025 ) , and Nemotron-Labs-Diffusion ( Fu et al., 2026 ) . However, the current landscape forces a stark compromise between speed, intelligence, and accessibility. Some models, e.g., Gemini Diffusion and Mercury, are locked behind proprietary APIs. On the other hand, existing open-weights alternatives either exhibit limited reasoning capabilities and multimodal understanding, or fail to deliver on the extreme latency benefits promised by the diffusion technology, or both. Until now, there has been no text diffusion model that is highly intelligent, exceptionally fast, and openly accessible.

We introduce DiffusionGemma to bridge this gap. A finetuned text diffusion variant of the Gemma 4 26B A4B mixture-of-experts (MoE) model ( Gemma Team et al., 2026 ) , the model establishes a new Pareto frontier in the intelligence-to-speed trade-off for generative text modeling (Figure 1 ). Notably, it extends the speed frontier beyond both prior text diffusion models and the Gemma 4 AR family across all parameter scales (from E2B up to 31B), even when the AR models are equipped with multi-token prediction ( Google DeepMind, 2026 ; DeepSeek-AI, 2024 , MTP,) , a state-of-the-art speculative decoding technique. To unlock this new speed-to-intelligence frontier, DiffusionGemma maximizes algorithmic efficiency by outputting blocks of 256 tokens simultaneously in around 12 forward passes. Put differently, DiffusionGemma generates on average 20 TPF—a step-change improvement over the ∼ \sim 3-6 TPF that can be achieved by state-of-the-art speculative decoding methods. This massive reduction in total forward passes directly offsets the computational cost of the diffusion model’s heavier individual forward passes and as a result, DiffusionGemma yields generation speeds of around 1,500 tokens per second (TPS) on a single NVIDIA H100 GPU. Furthermore, early benchmarking by third-party inference providers such as Unsloth (2026) demonstrates these speeds can scale up to 2,000 TPS on the NVIDIA RTX 6000.

To avoid the prohibitive computational cost of pretraining it is common practice to initialize text diffusion models from pretrained AR models ( Han et al., 2024 ; Gong et al., 2025 ) . We warm-start DiffusionGemma from the final post-trained, publicly released weights of the Gemma 4 26B A4B MoE model ( Gemma Team et al., 2026 ) . By adopting the same transformer backbone, we efficiently repurpose the AR weights to support both causal attention for context encoding and bidirectional attention for the diffusion process while inheriting their full capabilities. We employ a two-stage training pipeline (see Figure 2 ) that utilizes less than 10% of the AR model’s total training tokens:

topsep=0pt Supervised fine-tuning (SFT) : We first run an SFT phase to adapt the model to attend to a context of clean tokens as well as denoising a block of 256 noisy tokens (as dictated by the diffusion process) with bidirectional attention across the block.

topsep=0pt Sampler distillation and reinforcement learning ( SD ⋅ \cdot RL ) : We then apply an online learning phase that simultaneously improves generation quality (by maximizing rewards) and unlocks ultra-low latency (by substantially reducing the number of forward passes).

By preserving features of its AR starting point, DiffusionGemma exhibits strong multimodal understanding, long-context capabilities, and thinking mode, enabling it to generate reasoning traces prior to responding. These are hallmarks of current frontier systems ( Gemini Team, 2025 ) . Although adapting the model to the diffusion regime introduces a performance penalty compared to the original AR baseline, the massive gains in inference speed offer an entirely new operating point for latency-critical applications. DiffusionGemma retains the ability to generate text autoregressively. This dual mode opens up the possibility to route requests dynamically based on latency constraints and task complexity, as well as to employ hybrid decoding approaches.

##### Open-weights release.

We release DiffusionGemma with open weights under a permissive Apache 2.0 license, aiming to democratize access to state-of-the-art text diffusion technology. By providing full, unrestricted access to the model parameters, we hope to empower a diverse ecosystem of researchers, developers, and practitioners. For the research community, this transparent release provides a robust, white-box baseline to deeply probe the underlying mechanics of discrete diffusion, accelerating foundational research and pushing the theoretical frontier of text generation. For developers, the liberal licensing removes friction, ensuring seamless integration into both experimental prototypes and commercial applications without restrictive usage barriers.

Crucially, DiffusionGemma’s highly efficient architecture makes it an ideal foundation for rapid, domain-specific adaptation. By dramatically lowering the computational barrier to entry, the community can easily finetune ultra-fast, task-specific models tailored to their own unique use cases, even in environments with constrained compute budgets. This capacity for lightweight, rapid iteration has catalyzed immediate community adoption. Despite the model being available for only a few weeks at the time of writing, it is already serving as the engine for specialized downstream applications. These rapidly emerging use cases showcase the model’s versatility across highly diverse and demanding domains, spanning from multilingual automatic speech recognition ( Interfaze, 2026 ) to interactive radiology report drafting in healthcare ( Van Puyvelde et al., 2026 ) .

##### Outline.

The remainder of this technical report is organized as follows. Section 2 formalizes the discrete diffusion framework and the theoretical foundations of our approach. Section 3 details the DiffusionGemma architecture, including the bidirectional decoding mechanism and sampling algorithm. We then describe the two stages of our training pipeline: the SFT phase in Section 4 , followed by our online learning phase combining sampler distillation and reinforcement learning in Section 5 . Section 6 breaks down our low-level inference optimizations. Section 7 presents experimental results. Section 8 provides instructions and a practical example of finetuning DiffusionGemma for downstream applications. In Section 9 , we showcase some practical advantages of text diffusion. Finally, we discuss limitations and known issues in Section 10 , before concluding in Section 11 .

## 2 Generative Text Modeling with Discrete Diffusion

Historically, LLMs have treated text generation as a strictly sequential process ( Bengio et al., 2003 ; Mikolov et al., 2010 ; Graves, 2013 ; Sutskever et al., 2014 ; Vaswani et al., 2017 ; Gu et al., 2018 ) . Classical AR language modeling factorizes the joint probability of a sequence x x of length L L into an exact product of conditional probabilities: p ⁡ ( x ) = ∏ i = 1 L p ⁡ ( x i ∣ x < i ) p(x)=\prod_{i=1}^{L}p(x^{i}\mid x^{<i}) , where i i indicates the token position in the sequence. While theoretically rigorous, this single-token factorization fundamentally limits generation speed on modern hardware accelerators due to severe memory-bandwidth constraints as well as strictly preventing the model from bidirectionally revising tokens based on future tokens ( Leviathan et al., 2023 ; Savinov et al., 2022 ) .

Diffusion models approximate the joint distribution p ⁡ ( x ) p(x) by factorizing the generative process over a sequence of noise levels (a Markov chain) rather than strictly left-to-right spatial positions ( Sohl-Dickstein et al., 2015 ; Ho et al., 2020 ; Song et al., 2021c ; Holderrieth and Erives, 2025 ) , building upon foundational work in score-matching and continuous-time flow models ( Song and Ermon, 2019 ; Song et al., 2021a ; Song et al., 2021b ; Meng et al., 2022 ; Lipman et al., 2023 ; De Bortoli et al., 2021 ) . During training, a forward process gradually corrupts clean data into random noise; a neural network is trained to learn the reverse denoising process. During inference, the model generates full sequences of data by iteratively refining the output sequence in parallel, starting from random noise.

Adapting diffusion to text requires handling the discrete nature of language ( Yi et al., 2024 ) . Early approaches adapt standard Gaussian diffusion to text by mapping discrete tokens into a continuous embedding space. These models differ primarily in how they map back to text: architectures like Diffusion-LM ( Li et al., 2022 ) and SED ( Strudel et al., 2022 ) produce continuous vectors that have to be forcibly projected or “rounded” back to discrete tokens, whereas methods like CDCD ( Dieleman et al., 2022 ) maintain continuous processes but predict categorical logits directly to sample tokens. Regardless of the decoding mechanism, these formulations face theoretical and geometric challenges that ultimately limit their effectiveness compared to native discrete diffusion models. In particular, hard rounding operations fundamentally break exactness of diffusion likelihood bounds established by continuous-time and variational formulations ( Kingma et al., 2021 ) and the valid vocabulary tokens occupy an infinitesimally small fraction of a high-dimensional latent space. As a result, the reverse generative process often drifts into empty, “meaningless” regions of the space; when rounded to nearest token, these degenerate embeddings map to arbitrary, unrelated tokens and produce incoherent text ( Shabalin et al., 2025 ; Meshchaninov et al., 2025 ) . To combat this spatial drift, methods have been developed to rely on per-step discretization mechanisms, but these generally underutilize the continuous space and restrict generative flexibility ( Hu et al., 2026 ) .

Discrete diffusion has emerged as a highly effective alternative ( Austin et al., 2021 ; Sahoo et al., 2024 ; Lou et al., 2024 ; Zhao et al., 2025 ) . This paradigm generalizes the single-step corruption heuristics of early bidirectional models into formal, multi-step Markov processes. The probabilistic transitions defined between categorical states in modern diffusion, such as absorbing mask states and multinomial distributions across the vocabulary, are direct descendants of masking and random token swapping introduced by BERT ( Devlin et al., 2019 ; Liu et al., 2019 ; Sanh et al., 2019 ) , as well as the plausible token replacements of ELECTRA ( Clark et al., 2020 ) . By operating directly on discrete states rather than continuous vectors, discrete diffusion avoids embedding projection mismatches entirely and offers better theoretical grounding for categorical transitions ( Gat et al., 2024 ) . DiffusionGemma builds upon this lineage of discrete diffusion to ensure high-fidelity token generation.

### 2.1 Probability Paths and Denoising

To formalize our discrete diffusion framework, we adopt the continuous-time Markov chain (CTMC) approach established in recent literature ( Campbell et al., 2022 ; Campbell et al., 2024 ; Gat et al., 2024 ) . Let 𝒱 \mathcal{V} define a categorical token vocabulary of size V V . We refer to the sequence of tokens undergoing iterative refinement as the canvas , with a length of C C . A realization of this canvas at time t ∈ [ 0 , 1 ] t\in[0,1] is denoted by the vector x t ∈ 𝒱 C x_{t}\in\mathcal{V}^{C} . We construct a marginal probability path to smoothly interpolate between a clean data distribution p p at time t = 0 t=0 and a fully corrupted source canvas at time t = 1 t=1 . At the start of this path, the clean tokens are defined as x 0 ∼ p ⁡ ( ⋅ ) x_{0}\sim p(\cdot) , which ultimately transition into the fully corrupted state x 1 ∼ Unif ⁡ ( 𝒱 C ) x_{1}\sim\operatorname{Unif}(\mathcal{V}^{C}) . Here, Unif ⁡ ( 𝒱 C ) \operatorname{Unif}(\mathcal{V}^{C}) denotes the uniform prior over the joint state space, where each of the C C tokens is independently uniformly distributed over 𝒱 \mathcal{V} . This corruption process parallels the explicit transition matrices—such as uniform noise or absorbing masking—commonly utilized in order-agnostic discrete diffusion models ( Hoogeboom et al., 2022 ) . By operating in continuous time, this framework bypasses the sequential, left-to-right decoding bottleneck of traditional AR models, allowing tokens across the entire canvas to be processed and refined in parallel.

#### 2.1.1 The Forward Process