Continuous Autoregressive Language Models — part 2/6
Cleaned-text code points [15982, 31747). Bibliography omitted here; source.md includes it.
The synthesis of these techniques produces a powerful and robust autoencoder. For a chunk of K = 4 K=4 tokens, we now employ a latent vector of l = 128 l=128 dimensions, providing the necessary capacity to encode information redundantly. The encoder learns a posterior distribution where the standard deviations, σ i \sigma_{i} , converge to approximately 0.3. This means that sampling the latent vector 𝐳 \mathbf{z} effectively perturbs the predicted mean 𝝁 \bm{\mu} with a substantial Gaussian noise 𝝈 ≈ 0.3 𝐈 \bm{\sigma}\approx 0.3\mathbf{I} . Despite this significant latent perturbation, the decoder still maintains a token-level accuracy exceeding 99.9%. This vector representation, which combines high fidelity with high robustness, lays a solid foundation for the subsequent learning of Continuous Autoregressive Language Models (CALM).
## 3 Likelihood-Free Language Modeling
### 3.1 Next-Vector Prediction
The autoencoder developed in Section 2 establishes a robust and high-fidelity mapping between a chunk of K K discrete tokens and a single continuous vector, which allow us to reframe language modeling from a task of next-token prediction on discrete token sequences to next-vector prediction on continuous vector sequences. Specifically, a sequence of T T tokens, 𝐗 = ( x 1 , … , x T ) \mathbf{X}=(x_{1},\dots,x_{T}) , is first grouped into L = T / K L=T/K non-overlapping chunks. The encoder, f enc f_{\text{enc}} , then transforms the original sequence into a new, more compact sequence of continuous vectors: 𝐙 = ( 𝐳 1 , 𝐳 2 , … , 𝐳 L ) , where 𝐳 i = f enc ( x ( i − 1 ) K + 1 , … , x i K ) . \mathbf{Z}=(\mathbf{z}_{1},\mathbf{z}_{2},\dots,\mathbf{z}_{L}),\quad\text{where}\quad\mathbf{z}_{i}=f_{\text{enc}}(x_{(i-1)K+1},\dots,x_{iK}). (5) Consequently, the autoregressive objective evolves to predicting the next vector in the sequence: p ( 𝐙 ) = ∏ i = 1 L p ( 𝐳 i | 𝐳 < i ) . p(\mathbf{Z})=\prod_{i=1}^{L}p(\mathbf{z}_{i}|\mathbf{z}_{<i}). (6)
While this autoregressive structure is preserved, the underlying mechanism for predicting the next element must be redesigned. Unlike standard language models, which rely on a softmax layer to compute a probability distribution over a finite vocabulary, our model must predict a vector within the infinite space ℝ l \mathbb{R}^{l} . The softmax function is not applicable over this uncountable set, rendering the explicit probability density p ( 𝐳 i | 𝐳 < i ) p(\mathbf{z}_{i}|\mathbf{z}_{<i}) intractable. This introduces two critical challenges: • Training: The likelihood p ( 𝐳 i | 𝐳 < i ) p(\mathbf{z}_{i}|\mathbf{z}_{<i}) becomes intractable, precluding the use of maximum likelihood estimation (i.e., minimizing cross-entropy loss) for training.
• Evaluation: Standard evaluation metrics like Perplexity, which are derived directly from the model’s likelihood, can no longer be computed to measure model performance.
We address both of these challenges in turn. For the training problem, we introduce our approach to likelihood-free language modeling in the remainder of this section. For the evaluation problem, we propose a likelihood-free evaluation methodology in Section 4 .
### 3.2 Generative Head
Generative modeling of continuous data ( Kingma & Welling, 2014 ; Goodfellow et al., 2014 ; Ho et al., 2020 ) is a well-established field, foundational to domains such as image and audio synthesis where data is inherently continuous. A promising recent paradigm ( Tschannen et al., 2023 ; Li et al., 2024 ; Shao et al., 2025b ) combines these approaches with autoregressive models: a Transformer backbone predicts a conditioning hidden state, which is used by a subsequent generative model to produce the continuous output for each step. Our Continuous Autoregressive Language Models (CALM) adapts this paradigm, but with a critical focus on computational efficiency that constrains the design of this generative component. We therefore conceptualize this component as a lightweight generative head . Formally, the generative head is a stochastic function that takes the Transformer’s hidden state, 𝐡 i − 1 ∈ ℝ d \mathbf{h}_{i-1}\in\mathbb{R}^{d} , and draws a sample 𝐳 i ∈ ℝ l \mathbf{z}_{i}\in\mathbb{R}^{l} from the conditional distribution: 𝐡 i − 1 = Transformer ( 𝐳 1 : i − 1 ) , 𝐳 i ∼ p ( ⋅ | 𝐡 i − 1 ) . \mathbf{h}_{i-1}=\text{Transformer}(\mathbf{z}_{1:i-1}),\quad\mathbf{z}_{i}\sim p(\cdot|\mathbf{h}_{i-1}). (7)
While the generative head can be any continuous generative model, prominent options like Diffusion ( Ho et al., 2020 ; Li et al., 2024 ; Fan et al., 2025 ) or Flow Matching ( Lipman et al., 2023 ; Ren et al., 2025a ; Ren et al., 2025b ) are misaligned with our goal of efficiency. These models rely on an iterative sampling process—requiring dozens or even hundreds of network evaluations to produce a single vector—which directly counteracts the speedup gained from reducing the number of autoregressive steps. The CALM architecture therefore demands a generative head capable of high-quality, single-step generation, a challenge we address next with an energy-based objective.
### 3.3 Energy Transformer
#### 3.3.1 Strictly Proper Scoring Rules
To meet the demand for a generative head capable of high-quality, single-step generation, we draw inspiration from Shao et al. (2024) ; Shao et al. (2025b) , which frames the generative task as the optimization of strictly proper scoring rules ( Gneiting & Raftery, 2007 ) . Formally, a scoring rule S ( P , y ) S(P,y) assigns a numerical score to a predictive distribution P P upon observing an outcome y y , where higher scores are better. The quality of a forecast P P against the true data-generating distribution Q Q is measured by its expected score, defined as S ( P , Q ) = 𝔼 y ∼ Q [ S ( P , y ) ] S(P,Q)=\mathbb{E}_{y\sim Q}[S(P,y)] . A scoring rule is considered proper if the expected score is maximized when the predictive distribution P P matches the data distribution Q Q : S ( P , Q ) ≤ S ( Q , Q ) for all distributions P . S(P,Q)\leq S(Q,Q)\quad\text{for all distributions }P. (8) This property ensures that the scoring rule does not incentivize the model to predict a biased or distorted distribution. Furthermore, a scoring rule is strictly proper if equality holds only when P = Q P=Q , meaning that the optimal score can only be achieved by reporting the true distribution.
The use of a strictly proper scoring rule as a training objective is therefore a powerful and principled approach for training our generative head, as maximizing the expected score is equivalent to driving the model’s predictive distribution to match the true distribution. This principle offers a direct generalization of maximum likelihood estimation, where the negative log-likelihood is a special case corresponding to the logarithmic score ( Good, 1952 ) . While the likelihood is intractable in the continuous domain, the theory of scoring rules provides a rich family of alternatives.
#### 3.3.2 Energy Loss
We build our training objective using the Energy Score ( Székely, 2003 ) , a strictly proper scoring rule that has proven effective across a range of generative applications ( Gritsenko et al., 2020 ; Vahidi et al., 2024 ; Pacchiardi et al., 2024 ; Shao et al., 2025b ; Ma et al., 2025 ) . The energy score is entirely likelihood-free; rather than evaluating probability densities, it measures the alignment between the prediction and the observation via sample distances. For a predictive distribution P P and a ground truth observation 𝐲 \mathbf{y} , the energy score is defined as: S ( P , 𝐲 ) = 𝔼 𝐱 ′ , 𝐱 ′′ ∼ P [ ‖ 𝐱 ′ − 𝐱 ′′ ‖ α ] − 2 𝔼 𝐱 ∼ P [ ‖ 𝐱 − 𝐲 ‖ α ] , S(P,\mathbf{y})=\mathbb{E}_{\mathbf{x}^{\prime},\mathbf{x}^{\prime\prime}\sim P}[\|\mathbf{x}^{\prime}-\mathbf{x}^{\prime\prime}\|^{\alpha}]-2\ \mathbb{E}_{\mathbf{x}\sim P}[\|\mathbf{x}-\mathbf{y}\|^{\alpha}], (9) where 𝐱 \mathbf{x} , 𝐱 ′ \mathbf{x}^{\prime} and 𝐱 ′′ \mathbf{x}^{\prime\prime} are independent samples drawn from P P . The score is strictly proper for any α ∈ ( 0 , 2 ) \alpha\in(0,2) . Typically, α \alpha is set to 1. The first term encourages diversity, penalizing the model for producing collapsed or overly confident predictions where all samples are identical. The second term encourages fidelity, driving the model’s predictions to be close to the ground truth observation.
While the expectations in Equation 9 make the energy score intractable to compute exactly, we can construct an unbiased Monte Carlo estimator to serve as a practical loss function, which we term the energy loss . To do this, we draw N N candidate samples, { 𝐳 ~ i , 1 , … , 𝐳 ~ i , N } \{\tilde{\mathbf{z}}_{i,1},\dots,\tilde{\mathbf{z}}_{i,N}\} , from the generative head at each step i i . Furthermore, we leverage a unique property of our setup: our autoencoder does not map a token chunk to a fixed point, but rather to a conditional Gaussian posterior 𝐳 i ∼ q ( ⋅ | 𝐱 ( i − 1 ) K + 1 : i K ) \mathbf{z}_{i}\sim q(\cdot|\mathbf{x}_{(i-1)K+1:iK}) . Relying on a single sample 𝐳 i \mathbf{z}_{i} as ground-truth can introduce high variance into the energy loss. To mitigate this and stabilize training, we draw M M target samples, { 𝐳 i , 1 , … , 𝐳 i , M } \{\mathbf{z}_{i,1},\dots,\mathbf{z}_{i,M}\} , from this posterior. Combining these sample sets, the final energy loss is formulated as: ℒ energy = ∑ i = 1 L ( 2 N M ∑ n = 1 N ∑ m = 1 M ‖ 𝐳 i , m − 𝐳 ~ i , n ‖ − 1 N ( N − 1 ) ∑ n ≠ k ‖ 𝐳 ~ i , n − 𝐳 ~ i , k ‖ ) . \mathcal{L}_{\text{energy}}=\sum_{i=1}^{L}(\frac{2}{NM}\sum_{n=1}^{N}\sum_{m=1}^{M}\|\mathbf{z}_{i,m}-\tilde{\mathbf{z}}_{i,n}\|-\frac{1}{N(N-1)}\sum_{n\neq k}\|\tilde{\mathbf{z}}_{i,n}-\tilde{\mathbf{z}}_{i,k}\|). (10)
In practice, we set N = 8 N\!=\!8 and M = 100 M\!=\!100 . The number of model samples N N directly scales the training cost, as each sample requires an evaluation of the generative head; we therefore use a small N N to maintain high training efficiency. The overhead of drawing target vectors from a known Gaussian posterior is almost negligible, which allows us to use a large M M to reduce the variance of loss.
A key advantage of this likelihood-free training objective is its flexibility: it only requires the ability to draw samples from the generative head, placing minimal constraints on its internal architecture and allowing for the simple and efficient designs we explore next.
#### 3.3.3 Model Architecture
We now detail our model architecture. We use a standard Transformer backbone, with modifications focused on the output-side generative head and the input-side adaptation.
Energy-Based Generative Head . The inputs to the generative head are twofold: the hidden state 𝐡 i − 1 \mathbf{h}_{i-1} from the Transformer backbone, which provides the conditional context, and a random noise vector 𝜺 ∈ ℝ d noise \bm{\varepsilon}\in\mathbb{R}^{d_{\text{noise}}} , which provides the necessary stochasticity for sampling. Each dimension of 𝜺 \bm{\varepsilon} is drawn independently from a uniform distribution 𝒰 [ − 0.5 , 0.5 ] \mathcal{U}[-0.5,0.5] . Both the hidden state 𝐡 i − 1 \mathbf{h}_{i-1} and the noise vector 𝜺 \bm{\varepsilon} are projected by independent linear layers to match the head’s internal dimension, which we set to match the Transformer’s hidden dimension d d .
The core of the generative head is a stack of L L residual MLP blocks that progressively refine the initial noise representation 𝜺 0 = 𝜺 \bm{\varepsilon}_{0}=\bm{\varepsilon} into the final output vector. As illustrated in Figure 2 , each MLP block first fuses the current representation 𝜺 l \bm{\varepsilon}_{l} with the hidden state via two linear layers. This is followed by a SwiGLU layer ( Shazeer, 2020b ) with an intermediate dimension of d d . A residual connection then adds the block’s input to its output. This process concludes with a final linear layer that projects the representation to the target dimension l l , producing the output vector 𝐳 i \mathbf{z}_{i} .
A single MLP block contains approximately 6 d 2 6d^{2} parameters. We set the number of blocks to a quarter of the number of Transformer layers; the entire generative head therefore accounts for only about 10 % 10\% of the total model parameters, making its computational overhead minimal.
Discrete Token Input . An intuitive approach for the model’s input would be to embed the predicted latent vectors 𝐳 i − 1 \mathbf{z}_{i-1} from the previous step into the Transformer’s hidden dimension d d using a linear projection. However, we empirically found that using these latent vectors as input for the Transformer leads to a noticeable degradation in performance, as the model struggles to unpack the semantic information from such a compact input representation.
To circumvent this, we ground the model’s autoregressive process in the discrete token space. During training, the input for each step is formed by the K K tokens from the previous step. To maintain efficiency, we use a lightweight input compression module—a two-layer MLP—to map the K embeddings into a single input representation. The inference process unfolds as follows: 1. Input Processing: At step i i , the previously generated chunk of K K tokens are embedded and compressed into a single input representation and fed into the Transformer.
2. Continuous Prediction: The Transformer outputs the hidden state 𝐡 i − 1 \mathbf{h}_{i-1} , which our energy-based generative head then uses to predict the next continuous vector, 𝐳 i \mathbf{z}_{i} .
3. Discrete Feedback Loop: The predicted vector 𝐳 i \mathbf{z}_{i} is immediately passed through the frozen decoder of our pre-trained autoencoder, g dec g_{\text{dec}} , to reconstruct the next K K discrete tokens.
The complete architecture of CALM is illustrated in Figure 2 .
## 4 Likelihood-Free LM Evaluation
### 4.1 Principles of LM Evaluation
The CALM framework operates as an implicit generative model, whose predictive probability distribution is defined by its sampling process. Consequently, standard LM evaluation metrics like Perplexity, which are defined in terms of explicit likelihoods, can no longer be employed to measure model performance. Furthermore, the energy loss used for training is itself unsuitable for evaluation, as its magnitude is subjective to the specific latent space shaped by the autoencoder. This necessitates the development of a model-agnostic evaluation metric, one that can faithfully assess language modeling capabilities in a principled, yet entirely likelihood-free, manner.
The goal of a evaluation metric is to quantify the divergence between the model’s predictive distribution, P P , and the true data distribution, Q Q . This principle is formalized by the property that the metric is uniquely optimized when the model accurately recovers the data distribution ( P = Q P=Q ). This ensures the evaluation is fair and cannot be hacked by a model that systematically distorts its predictions. For instance, the conventional metric of Perplexity serves as a prime example of this principle. It is grounded in the expected negative log-likelihood, which can be decomposed into the sum of the KL divergence and data entropy: 𝔼 y ∼ Q [ − log P ( y ) ] = 𝔼 y ∼ Q [ log Q ( y ) P ( y ) ] + 𝔼 y ∼ Q [ − log Q ( y ) ] = D K L ( Q ∥ P ) ⏟ Minimized at P = Q + H ( Q ) ⏟ Constant . \mathbb{E}_{y\sim Q}[-\log P(y)]=\mathbb{E}_{y\sim Q}\left[\log\frac{Q(y)}{P(y)}\right]+\mathbb{E}_{y\sim Q}[-\log Q(y)]=\underbrace{D_{KL}(Q\|P)}_{\text{Minimized at }P=Q}+\underbrace{H(Q)}_{\text{Constant}}. (11) This property establishes Perplexity as a theoretically sound measure of a model’s capability to capture the true distribution, which is uniquely minimized when P = Q P=Q .