Continuous Autoregressive Language Models — part 4/6

Cleaned-text code points [46620, 62528). Bibliography omitted here; source.md includes it.

Latent Generative Modeling. A prominent paradigm in generative modeling involves a two-stage process: first learning a compressed latent representation of the data, and then training a generative model within that latent space. This approach often begins with a Variational Autoencoder ( Kingma & Welling, 2014 ) , which learns a mapping from a high-dimensional data space into a compact, continuous latent space. This principle enables modern architectures, such as latent diffusion models ( Rombach et al., 2022 ; Liu et al., 2023 ) , to efficiently generate high-dimensional data from a continuous latent representation. An alternative path, the Vector Quantized VAE ( van den Oord et al., 2017 , VQ-VAE,) , learns a discrete latent space by mapping inputs to a finite, learned codebook. This approach has been foundational to the autoregressive generation of continuous data like images ( Razavi et al., 2019 ; Esser et al., 2021 ; Ramesh et al., 2021 ; Sun et al., 2024a ) and audio ( Dhariwal et al., 2020 ; Zeghidour et al., 2021 ; Défossez et al., 2023 ) . Our approach introduces a distinct way by performing a discrete-to-continuous mapping. Driven by the pursuit of efficiency, it significantly reduces the number of autoregressive steps required for language generation.

Text Compression. Compressing long text into compact vector representations is a foundational concept in sequence modeling. For instance, Recurrent Neural Networks can be viewed as implicitly compressing the entire history of a sequence into a single hidden state vector ( Elman, 1990 ; Hochreiter & Schmidhuber, 1997 ) . In the era of LLMs, this concept has been revitalized, with a focus on prompt compression to improve inference efficiency. For example, Mu et al. (2023) designed a modified attention mechanism to distill prompt information into a few memory tokens. Chevalier et al. (2023) ; Ge et al. (2024) ; Gao et al. (2024) further introduced explicit reconstruction objectives to promote high fidelity compression. Recently, Li et al. (2025) ; Kuratov et al. (2025) ; Mezentsev & Oseledets (2025) pushed the limits of compression to a ratio up to 1568x, underscoring the inherent sparsity of discrete text representations. More recently, DeepSeek-OCR ( Wei et al., 2025 ) demonstrated compressing text into continuous image tokens, showing promise for applications like long-context compression. The primary focus of these methods on prompt compression places a greater emphasis on reconstruction fidelity than on the robustness of the resulting representation. Our work, by contrast, prioritizes the creation of a robust and smooth latent manifold, which is a critical prerequisite for stable downstream generative modeling.

### 6.2 Likelihood-Free Language Modeling

Continuous Autoregressive Generation. Autoregressive generation over continuous vectors is an emerging research frontier, with notable successes in domains such as image ( Tschannen et al., 2023 ; Li et al., 2024 ; Shao et al., 2025b ; Fan et al., 2025 ; Team, 2025 ) , video ( Chen et al., 2024 ; Deng et al., 2025 ) , and audio synthesis ( Turetzky et al., 2024 ; Sun et al., 2024b ; Ma et al., 2025 ) . GIVT ( Tschannen et al., 2023 ) pioneered this direction by fitting the distribution of the target vector with a Gaussian Mixture Model. However, the expressive power of GIVT is confined to the pre-defined family of Gaussian mixtures, a constraint that limits its ability to capture complex distributions. Li et al. (2024) overcomes this limitation by employing a lightweight diffusion head to model the vector distribution. While being more expressive, this method comes at the cost of inference efficiency due to its iterative sampling process. More recently, Shao et al. (2025b) introduced a general framework based on strictly proper scoring rules. The Energy Transformer was presented as a concrete and powerful instance of this framework, capable of high-quality, single-step generation. Our work adopts the core Energy Transformer framework but introduce several key improvements to the generative head architecture, the energy loss, and the model’s input structure, to further enhance its performance and stability for the specific challenges of language modeling.

Parallel Token Prediction. The goal of predicting multiple tokens in parallel to overcome the sequential bottleneck of autoregressive models is a long-standing pursuit in sequence modeling. Early efforts in this area were pioneered by non-autoregressive machine translation ( Gu et al., 2018 ; Gu & Kong, 2021 ; Shao et al., 2021 ; Shao & Feng, 2022 ; Huang et al., 2022 ; Gui et al., 2023 ) , which aims to generate an entire target sentence in a single step. While effective for highly constrained conditional tasks like translation, these methods often struggle with the inherent multi-modality of open-ended language generation. A different line of work uses multi-token prediction to enrich training signals ( Gloeckle et al., 2024 ; Shao et al., 2025a ) or provide candidates for speculative decoding ( Stern et al., 2018 ; Leviathan et al., 2023 ) , while the underlying generation remains single-token autoregressive. A more direct approach involves hierarchical modeling, where a global model predicts large semantic chunks, which are then decoded by a local model ( Lee et al., 2022 ; YU et al., 2023 ; Ho et al., 2024 ; team et al., 2024 ; Pagnoni et al., 2025 ; Neitemeier et al., 2025 ) . For instance, MegaByte ( YU et al., 2023 ) uses a global Transformer to predict blocks of tokens, but still relies on a local autoregressive model to generate tokens sequentially within each block. Conceptually closer to our work, Large Concept Models ( team et al., 2024 ) also adopt a hierarchical structure, where their global model autoregressively predicts continuous sentence embeddings. However, this approach faces several challenges that our CALM framework is designed to address: its SONAR autoencoder ( Duquenne et al., 2023 ) is computationally heavy and fragile, and its reliance on a diffusion-based generative process introduces a iterative inference bottleneck. Finally, another paradigm for parallel generation is diffusion models for text, which iteratively refine a sequence of tokens from noise, either at the full sentence ( Austin et al., 2021 ; Li et al., 2022 ; Lou et al., 2024 ) or block level ( Han et al., 2023 ; Arriola et al., 2025 ) . These models, which currently operate in the challenging discrete token space, could potentially benefit from the robust continuous space our autoencoder provides.

### 6.3 Likelihood-Free LM Evaluation

LM metrics. The evaluation of language models is split into two distinct paradigms, which reflects the separation between assessing the quality of generated output and the fidelity of the learned distribution. On one hand, likelihood-based metrics, such as Perplexity, offer a principled way to evaluate the learned distribution, but they are limited to models where likelihoods are tractable. On the other hand, a diverse family of sample-based metrics focuses on the generated output. Classic methods like BLEU ( Papineni et al., 2002 ) and ROUGE ( Lin, 2004 ) assess the quality of generated text by comparing it to reference outputs. More recent approaches such as MAUVE ( Pillutla et al., 2021 ) or LLM-as-a-judge ( Zheng et al., 2023 ) allows for reference-free evaluation, but they rely on heuristics or black-box models and lack the formal guarantees of scoring rules. Our proposed metric, BrierLM, is designed to bridge this gap by combining the advantages of both paradigms: it operates exclusively on model samples, yet as a strictly proper scoring rule, it offers a faithful assessment of the model’s predictive quality, akin to perplexity.

Brier Score. The Brier Score was originally proposed by Brier (1950) for the evaluation of probabilistic weather forecasts. It is a classic example of a strictly proper scoring rules, theoretically guaranteeing that a model is incentivized to report its true belief to achieve the optimal score ( Gneiting & Raftery, 2007 ) . Consequently, it has been widely adopted in classification tasks, primarily for evaluating the quality of probabilistic forecasts ( Sanders, 1963 ; fer, 2009 ; Hui & Belkin, 2021 ) and assessing model calibration ( Lakshminarayanan et al., 2017 ; Ovadia et al., 2019 ; Gruber & Buettner, 2022 ) . The innovation of our work is twofold: first, we introduce a method to unbiasedly estimate the Brier score in a likelihood-free manner; second, we generalize its application from a metric for simple classification tasks to one capable of assessing language modeling capabilities.

### 6.4 Likelihood-Free Temperature Sampling

Bernoulli Factory. The temperature sampling problem is conceptually related to the classic problem of the Bernoulli Factory ( Keane & O’Brien, 1994 ; Occil, 2020 ) , which addresses the challenge of simulating a new coin with a success probability of f ⁡ ( p ) f(p) given only a coin with an unknown success probability p p . This mirrors our challenge of achieving a target probability proportional to P ​ ( x ) 1 / T P(x)^{1/T} using only a base sampler for the implicit distribution P ⁡ ( x ) P(x) . A key distinction is that the Bernoulli Factory problem assumes a binary outcome, whereas we operate over a large discrete sample space. Our two-stage algorithm elegantly bridges this gap. The first stage isolates a single candidate x ∗ x^{*} and reduces the problem to a binary one, and the second stage directly applies an existing Bernoulli Factory algorithm ( Mendo, 2019 ) to construct an event with probability P ​ ( x ∗ ) α P(x^{*})^{\alpha} .

Controlled Generation. While many generative models lack the explicit probabilistic controls for temperature sampling, they have developed alternative strategies to navigate the trade-off between sample quality and diversity. For instance, VAEs and normalizing flows ( Kingma & Welling, 2014 ; Rezende & Mohamed, 2015 ) often achieve this by adjusting the variance of their prior latent distribution ( Kingma & Dhariwal, 2018 ) . In Generative Adversarial Networks ( Goodfellow et al., 2014 ) , the truncation trick restricts sampling to a high-density region of the latent space ( Brock et al., 2019 ) . Similarly, diffusion models can control stochasticity by altering the noise variance during the reverse sampling process ( Song et al., 2021 ) . These techniques, however, are fundamentally heuristic, as it is generally intractable to characterize the shape of the modified output distribution, and they all require white-box access to model internals like the latent space. Our work, in contrast, proposes a universal, black-box algorithm for temperature sampling from implicit models over discrete spaces, offering a provably exact method for this broad class of models.

## 7 Experiments

### 7.1 Settings

Datasets. We train our models on the Pile uncopyrighted dataset ( Gao et al., 2020 ) . The raw text is processed with the Llama 3 tokenizer ( Grattafiori et al., 2024 ) , resulting in a training set of ∼ \sim 230B tokens. We evaluate model performance on the WikiText-103 benchmark ( Merity et al., 2017 ) .

Model. Our models are built upon a standard Transformer backbone. We adopt most of the architecture designs from the LLaMA family ( Touvron et al., 2023 ) , including RMSNorm ( Zhang & Sennrich, 2019 ) , SwiGLU activation ( Shazeer, 2020a ) , and rotary positional embeddings ( Su et al., 2021 ) . We experiment with four scales: S (12 layers, hidden_size=768, intermediate_size=2048), M (16 layers, hidden_size=1024, intermediate_size=2752), L (16 layers, hidden_size=1536, intermediate_size=4096), and XL (16 layers, hidden_size=2560, intermediate_size=6880).

Training Details. The training process for our CALM framework is two-staged. We first train a suite of autoencoders on a 15B token subset of the Pile to map token chunks of size K ∈ { 1 , 2 , 4 , 8 } K\in\{1,2,4,8\} into continuous vectors. These autoencoders use a hidden size of 512, a latent dimension of 32 K K , have approximately 75M parameters, and are trained for 30k steps with a batch size of 512k tokens. Following this, the CALM models are trained on the remaining data for 250k steps with a batch size of 2 million tokens. The context length is set to 2048 steps; for CALM, this corresponds to 2048 K K tokens. All models are optimized using the AdamW optimizer ( Loshchilov & Hutter, 2019 ) with β 1 = 0.9 , β 2 = 0.95 , ϵ = 1 ​ e − 8 \beta_{1}=0.9,\beta_{2}=0.95,\epsilon=1e-8 . We use a learning rate of 3 × 10 − 4 3\times 10^{-4} with a constant schedule and a warmup of 2000 steps, a weight decay of 0.1, and gradient clipping of 1.0.

### 7.2 Main Results

We present the primary results of our comparison between the standard Transformer baselines and our CALM framework (with a fixed chunk size of K=4) in Table 1 . The results demonstrate that CALM establishes a new, more efficient performance-compute frontier for language modeling. By increasing the semantic bandwidth of each autoregressive step, CALM is allowed to be substantially larger in parameter count while demanding fewer FLOPs for both training and inference. For instance, our 371M parameter CALM-M model achieves a BrierLM score comparable to the 281M Transformer-S baseline, yet requires 44% fewer training FLOPs and 34% fewer inference FLOPs. Furthermore, the results confirm that CALM benefits from scaling just as effectively as traditional Transformers, allowing performance to be consistently improved by increasing model size.

In addition to scaling model size, our framework introduces the semantic bandwidth K K as a new lever for navigating the performance-compute landscape. Figure 5 illustrates this by plotting the performance of CALM-L with varying K K against the standard Transformer scaling curve. Notably, CALM-L with K = 1 K=1 operates at a significant disadvantage, demanding more FLOPs for lower performance compared to its discrete counterpart. The gap arises because the model undertakes the more challenging continuous prediction task, which in turn highlights the significant room for future architectural and algorithmic optimizations. The advantages of CALM become apparent as we increase K K . Moving from K = 1 K=1 to K = 2 K=2 nearly halves the cost with only a marginal drop in performance, and at K = 4 K=4 , the CALM model surpasses the baseline performance-compute frontier. This finding validates our central hypothesis: scaling the semantic bandwidth of each generative step provides a new and highly effective axis for optimizing the performance-compute trade-off in language models. Further increasing the chunk size to K = 8 K=8 leads to a larger performance drop, which is likely a model capacity limitation. We hypothesize that larger models may be required to leverage the benefits of higher semantic bandwidths.

To further investigate the learning dynamics of our framework, we plot the training curves of CALM-XL against the Transformer baselines in Figure 5 . The baseline Transformer models exhibit rapid initial gains before their performance gradually begins to saturate. In contrast, CALM-XL displays a more patient but ultimately steeper learning curve, initially trailing Transformer-M but progressively closing the performance gap with the Transformer-L model. We attribute this phenomenon to the different nature of the predictive task. While the baseline models learn the relatively simple task of predicting a single, low-information discrete token, our CALM model must learn to model the complex, high-dimensional distribution of continuous vectors, which explains the slower initial progress. However, once this ability is established, the model can unlock the potential of its large parameter count, entering a phase of more significant and sustained improvements.

### 7.3 Effect of Autoencoder