Continuous Autoregressive Language Models — part 1/6

Cleaned-text code points [0, 15982). Bibliography omitted here; source.md includes it.
# Continuous Autoregressive Language Models

###### Abstract

The efficiency of large language models (LLMs) is fundamentally limited by their sequential, token-by-token generation process. We argue that overcoming this bottleneck requires a new design axis for LLM scaling: increasing the semantic bandwidth of each generative step. To this end, we introduce Continuous Autoregressive Language Models (CALM), a paradigm shift from discrete next-token prediction to continuous next-vector prediction. CALM uses a high-fidelity autoencoder to compress a chunk of K tokens into a single continuous vector, from which the original tokens can be reconstructed with over 99.9% accuracy. This allows us to model language as a sequence of continuous vectors instead of discrete tokens, which reduces the number of generative steps by a factor of K. The paradigm shift necessitates a new modeling toolkit; therefore, we develop a comprehensive likelihood-free framework that enables robust training, evaluation, and controllable sampling in the continuous domain. Experiments show that CALM significantly improves the performance-compute trade-off, achieving the performance of strong discrete baselines at a significantly lower computational cost. More importantly, these findings establish next-vector prediction as a powerful and scalable pathway towards ultra-efficient language models.

Code: https://github.com/shaochenze/calm

Project: https://shaochenze.github.io/blog/2025/CALM

## 1 Introduction

Large Language Models (LLMs) have revolutionized the field of artificial intelligence, demonstrating unprecedented capabilities in understanding, generating, and reasoning with human language ( Achiam et al., 2023 ; Google, 2025 ; DeepSeek-AI, 2025 ) . However, this remarkable success is shadowed by a critical challenge: their immense computational demands. The training and inference of state-of-the-art LLMs demand massive computational resources, leading to prohibitive expenses and significant environmental concerns ( Strubell et al., 2019 ; Bender et al., 2021 ) . At the heart of this inefficiency lies the foundational paradigm of these models: an autoregressive generation process that operates on a sequence of discrete tokens. Because the computational cost scales with the length of the sequence, generating long-form text or processing extensive contexts remains a fundamental bottleneck, limiting the scalability and accessibility of these powerful models.

The now-ubiquitous use of discrete tokens in LLMs is the result of a pivotal evolution from earlier modeling paradigms. Initially, models that operated at the character level struggled with the computational burden of extremely long sequences ( Sutskever et al., 2011 ; Kim et al., 2016 ) . The subsequent shift to modern subword tokenization ( Sennrich et al., 2016 ) was driven by a crucial insight: increasing the information density of each text unit reduces sequence length and dramatically boosts model efficiency. This historical success suggests a clear path for unlocking the next order of magnitude in efficiency: continue to increase the semantic bandwidth of each predictive unit.

We argue, however, that this path has reached a fundamental limit, constrained by the very nature of discrete representation. With typical vocabularies in modern LLMs ranging from approximately 32,000 to 256,000 entries, each token carries a surprisingly small amount of information—merely 15 to 18 bits (e.g., log 2 ⁡ ( 32768 ) = 15 \log_{2}(32768)=15 ). To increase this capacity—for instance, to represent a whole phrase—the vocabulary size would need to grow exponentially, making the final softmax computation over this vocabulary an untenable bottleneck. This reveals a critical limitation: the information density of discrete tokens is not scalable. Consequently, a profound mismatch has emerged: while model capacity has scaled to unprecedented levels, the task itself—predicting low-information discrete units one at a time—has not evolved. We are now deploying models of immense representational power on a task that fundamentally limits their throughput, forcing them to laboriously predict simple, low-information tokens one by one.

In this work, we confront this limitation directly by introducing a paradigm shift from discrete tokens to a continuous-domain representation. Central to our approach is an autoencoder trained to compress a chunk of K tokens into a single, dense continuous vector and, crucially, reconstruct the original tokens from this vector with high fidelity. Unlike the discrete paradigm, where increasing information density requires an exponential growth in vocabulary size, our continuous representation offers a scalable path forward: the vector’s information capacity can be gracefully expanded by simply increasing its dimensionality to accommodate a larger K. This design directly reduces the number of autoregressive steps by a factor of K. Ultimately, it allows us to reframe language modeling from a task of next-token prediction on discrete token sequences to next-vector prediction on continuous vector sequences, as conceptually illustrated in Figure 1 .

However, shifting to the continuous domain introduces a significant challenge: without a finite vocabulary, a model cannot compute an explicit probability distribution over all possible outcomes using a standard softmax layer. To address this, we develop a comprehensive, likelihood-free framework for our Continuous Autoregressive Language Models (CALM). Our primary contributions, which structure the remainder of this paper, are as follows: • A Powerful and Lightweight Autoencoder (Section 2 ): We first introduce an efficient autoencoder architecture designed to produce robust vector representations. We demonstrate that this model can be both compact and powerful, ensuring high-fidelity reconstruction of the original tokens, which is a prerequisite for the downstream language modeling task.

• Likelihood-Free Language Modeling (Section 3 ): To perform generative modeling in the continuous vector space, we employ a lightweight generative head that conditions on the last hidden state to generate the output vector. While the generative head can be any continuous generative model, options like Diffusion ( Ho et al., 2020 ; Li et al., 2024 ) or Flow Matching ( Lipman et al., 2023 ) rely on an iterative sampling process, re-introducing a significant inference bottleneck. Our framework therefore specifically adopts the Energy Transformer ( Shao et al., 2025b ) , a recent architecture designed for efficient, single-step generation of continuous vectors, while empirically demonstrating superior generation quality.

• Likelihood-Free LM Evaluation (Section 4 ): The absence of explicit likelihoods makes traditional metrics like Perplexity inapplicable. We address this by proposing BrierLM, a novel metric for language modeling based on the Brier score ( Brier, 1950 ) . We show that BrierLM is strictly proper, theoretically ensuring a fair comparison of model capabilities. Crucially, BrierLM can be estimated unbiasedly by only drawing samples from the model, making it perfectly suited for CALM where likelihoods are intractable.

• Likelihood-Free Temperature Sampling (Section 5 ): Controlled generation via temperature sampling is an indispensable feature of modern LLMs, yet it relies on the explicit manipulation of a probability distribution. We introduce a principled, likelihood-free sampling algorithm that can, in theory, draw samples from the exact temperature distribution, and we accompany it with a highly efficient batch approximation.

We empirically validate our CALM framework on standard language modeling benchmarks, which demonstrates a superior performance-compute trade-off. For instance, a CALM grouping K=4 tokens delivers performance comparable to strong discrete baselines, but at a significantly lower computational cost. This findings highlight a new design axis for language models: rather than solely scaling parameters and data for performance, one can now scale the information capacity of each step as a powerful new lever for computational efficiency.

## 2 Autoencoder

### 2.1 High-Fidelity Reconstruction

The foundational component of our CALM framework is an autoencoder tasked with learning a bijective mapping between a chunk of K K discrete tokens and a continuous vector. Formally, we seek an encoder f e  n  c : 𝒱 K → ℝ l f_{enc}:\mathcal{V}^{K}\to\mathbb{R}^{l} and a decoder g d  e  c : ℝ l → 𝒱 K g_{dec}:\mathbb{R}^{l}\to\mathcal{V}^{K} , where 𝒱 \mathcal{V} is the vocabulary, such that for a given token sequence 𝐱 1 : K = ( x 1 , … , x K ) \mathbf{x}_{1:K}=(x_{1},\dots,x_{K}) , the reconstruction g d  e  c ( f e  n  c ( 𝐱 1 : K ) ) g_{dec}(f_{enc}(\mathbf{x}_{1:K})) closely approximates 𝐱 1 : K \mathbf{x}_{1:K} . For simplicity and computational efficiency, we design our autoencoder to be context-free , meaning it processes each token chunk independently of its surrounding sequence. A context-aware autoencoder that also conditions on previous vector representations is a natural and promising next step, which we leave for future exploration.

The encoder begins by mapping the input sequence 𝐱 1 : K \mathbf{x}_{1:K} to K K embeddings. Each embedding is independently processed by a position-wise feed-forward network (FFN). The resulting K K hidden states are then flattened and compressed by a linear layer : ℝ K  d → ℝ d :\mathbb{R}^{Kd}\to\mathbb{R}^{d} . This unified representation is passed through a second FFN and a linear projection to produce the l l -dimensional latent vector 𝐳 \mathbf{z} .

The decoder architecture mirrors the encoder. It first transforms 𝐳 \mathbf{z} using a linear layer and an FFN to obtain a d d -dimensional hidden state, which is then expanded by another linear layer to dimension K  d Kd and reshaped into a sequence of K K hidden states. Each of these states is passed through a second FFN, followed by a projection to vocabulary logits using the tied input embedding matrix. Finally, the tokens are reconstructed by applying an argmax operation over these logits.

The autoencoder is trained to minimize the reconstruction error by optimizing the standard cross-entropy loss across all K K token positions: ℒ ae ( 𝐱 1 : K ) = − ∑ i = 1 K log p d  e  c ( x i | 𝐳 = f enc ( 𝐱 1 : K ) ) . \mathcal{L}_{\text{ae}}(\mathbf{x}_{1:K})=-\sum_{i=1}^{K}\log p_{dec}(x_{i}|\mathbf{z}=f_{\text{enc}}(\mathbf{x}_{1:K})). (1)

We empirically validate this architecture and find it to be both highly effective and efficient. For instance, when grouping K = 4 K=4 tokens, a latent vector of just l = 10 l=10 dimensions is sufficient to achieve high-fidelity reconstruction, with a token-level accuracy of over 99.9%. Moreover, the autoencoder is exceptionally lightweight; with a shallow architecture and a modest hidden dimension of d = 512 d=512 , its computational overhead is nearly negligible compared to that of language model.

### 2.2 Robust Vector Representation

While the autoencoder described above achieves near-perfect reconstruction, we found that it is practically impossible to effectively train a continuous language model based on the vector space it produces. The root cause of this challenge is that an autoencoder optimized solely for reconstruction learns an exceptionally brittle representation. Lacking any incentive to form a smooth latent manifold, the encoder learns to pack information with maximum efficiency, creating a highly irregular mapping. In such a space, a minor perturbation to a latent vector 𝐳 \mathbf{z} —such as the small, inevitable errors made by a generative model can cause the decoder to reconstruct a completely unrelated token sequence. Therefore, for our CALM framework to be viable, the autoencoder must satisfy another critical objective: its vector representation should be robust.

Variational Regularization. To build a robust latent space, our primary strategy is to smooth the latent manifold by moving from a deterministic autoencoder to a variational one ( Kingma & Welling, 2014 ) , aligning our approach with prominent generative models ( Rombach et al., 2022 ; Liu et al., 2023 ) that operate within a smooth and structured latent space. Instead of mapping an input chunk directly to a vector 𝐳 \mathbf{z} , the encoder now outputs the parameters of a diagonal Gaussian distribution, 𝝁 \bm{\mu} and 𝝈 \bm{\sigma} , from which the latent vector is sampled: 𝐳 ∼ 𝒩 ⁡ ( 𝝁 , 𝝈 2  𝐈 ) \mathbf{z}\sim\mathcal{N}(\bm{\mu},\bm{\sigma}^{2}\mathbf{I}) . This change is accompanied by a new objective term, a KL divergence loss that penalizes the deviation of the encoded distribution from a standard normal prior, 𝒩 ⁡ ( 0 , 𝐈 ) \mathcal{N}(0,\mathbf{I}) . The total loss function is thus a weighted sum of the reconstruction and regularization terms: ℒ total = ℒ ae + β ⋅ ℒ KL , \mathcal{L}_{\text{total}}=\mathcal{L}_{\text{ae}}+\beta\cdot\mathcal{L}_{\text{KL}}, (2) where β \beta is a hyperparameter balancing the two objectives (we set β = 0.001 \beta=0.001 ), and ℒ KL \mathcal{L}_{\text{KL}} is the KL divergence, defined as: ℒ KL ( p E ( 𝐳 | 𝐱 1 : K ) ∥ 𝒩 ( 0 , 𝐈 ) ) = − 1 2 ∑ i = 1 l ( 1 + log σ i 2 − σ i 2 − μ i 2 ) . \mathcal{L}_{\text{KL}}(p_{E}(\mathbf{z}|\mathbf{x}_{1:K})\|\mathcal{N}(0,\mathbf{I}))=-\frac{1}{2}\sum_{i=1}^{l}(1+\log\sigma_{i}^{2}-\sigma_{i}^{2}-\mu_{i}^{2}). (3) This variational objective discourages the encoder from relying on arbitrarily precise or large-magnitude values in 𝐳 \mathbf{z} , thereby promoting a smoother and more regularized latent manifold that is more amenable to generative modeling.

Preventing Posterior Collapse. A significant challenge in training VAEs is posterior collapse. This issue manifested in our model as a tendency for some latent dimensions to fully collapse to the standard normal prior. While collapsing a dimension drives its KL divergence to zero, it renders that dimension uninformative for reconstruction. More critically, these pure noise dimensions introduce a chaotic signal that interferes with the training of the downstream language model, destabilizing the learning process. To mitigate this, we adopt the KL clipping strategy from Kingma et al. (2016) , which modifies the objective by clipping each dimension’s KL loss at a constant floor: ℒ KL c  l  i  p = ∑ i = 1 l max ⁡ ( λ K  L , ℒ KL , i ) , \mathcal{L}_{\text{KL}}^{clip}=\sum_{i=1}^{l}\max(\lambda_{KL},\mathcal{L}_{\text{KL},i}), (4) where ℒ KL , i \mathcal{L}_{\text{KL},i} is the KL divergence for the i i -th dimension and λ K  L \lambda_{KL} is the threshold (we use λ K  L = 0.5 \lambda_{KL}=0.5 ). This technique ensures that every dimension is encouraged to actively participate in reconstruction, thus preventing collapse and fostering a dense, structured representation.

Dropout for Enhanced Robustness. Beyond structuring the latent space with variational methods, we further enhance its robustness by injecting noise during training using two complementary forms of dropout. First, we apply dropout with a rate of p = 0.15 p=0.15 to the latent vector 𝐳 \mathbf{z} before it is passed to the decoder. This forces the autoencoder to learn a redundant representation, making it robust to minor prediction errors from the downstream generative model. Second, we apply dropout to input tokens by randomly masking a fraction ( p = 0.15 p=0.15 ) of tokens. Analogous to the Continuous Bag-of-Words (CBOW) method ( Mikolov et al., 2013 ) , this compels the autoencoder to infer masked tokens from their context, thereby enriching the latent vector with the chunk’s semantic context rather than just performing a simple token-index compression. Crucially, these dropout techniques are employed exclusively during the autoencoder’s training phase to build a robust latent representation; they are disabled during the subsequent training and inference of the continuous language model.