Continuous Autoregressive Language Models — part 5/6

Cleaned-text code points [62528, 78318). Bibliography omitted here; source.md includes it.

In this section, we study the effect of the autoencoder’s design choices on the final performance of the CALM framework. The autoencoder is a critical component, as it defines the latent space in which the continuous language model operates. To isolate its effects, we hold the downstream language model fixed across all experiments in this section: an Energy Transformer with a hidden size of 768, 12 hidden layers, 16 attention heads, an FFN intermediate size of 2048, and a generative head with 3 MLP blocks. Each model configuration is trained for 50,000 steps. Unless otherwise specified, the autoencoder uses the default parameters as described in Section 7.1 . We begin with a comprehensive ablation study to validate the contribution of each proposed technique, followed by a detailed analysis of the effect of several key hyperparameters.

We first validate the design choices for the autoencoder, with results detailed in Table 2 . While a standard, reconstruction-only autoencoder provides a reasonable baseline, naively incorporating a variational objective leads to a significant drop in performance. This degradation is traced to a severe instance of posterior collapse, where we found that 71 of the 128 latent dimensions had collapsed to the standard normal prior. The introduction of the KL clipping strategy proves to be the crucial remedy, which effectively prevents dimensional collapse and leads to a notable performance improvement. Furthermore, applying dropout regularization to both the input tokens and the latent vector yields considerable, orthogonal performance benefits, confirming that each technique contributes uniquely to shaping a high-fidelity and robust latent space.

KL weight. We next examine the model’s sensitivity to the KL divergence weight, β \beta , which governs the trade-off between reconstruction fidelity and latent space regularization. We varied β \beta across several orders of magnitude and present the results in Figure 7 . Starting from a baseline with no KL regularization ( β = 0 \beta=0 ), we observe that introducing a small amount of variational regularization significantly improves the final BrierLM score, which confirms our hypothesis: a moderate regularization effectively smooths the latent manifold, making it more learnable for the Energy Transformer, while leaving reconstruction accuracy almost unaffected. However, this trend reverses as the regularization becomes overly aggressive. At β = 0.1 \beta=0.1 , the BrierLM score drops sharply, a decline directly linked to the autoencoder’s compromised reconstruction fidelity, which falls to ∼ \sim 99%. Based on these findings, we selected β = 0.001 \beta=0.001 to train our autoencoder.

Latent Demension. We next examine the influence of the latent dimension, l l , which functions as the information bottleneck of the autoencoder. We evaluated latent dimensions of 32, 64, 128, and 256, and respectively scaled the dropout rate to 0.05, 0.1, 0.15, and 0.2. The results are illustrated in Figure 7 . As seen, the reconstruction accuracy remains consistently high across all configurations, but the downstream performance varies, peaking at l = 128 l=128 . This suggests a trade-off in selecting the optimal dimension. A latent space that is too small, such as with l = 32 l=32 , forces the autoencoder to learn an overly compact and brittle representation. Conversely, a large latent dimension may lead the autoencoder to encode noisy or irrelevant features from the input tokens. This forces the Energy Transformer to expend its finite capacity modeling this noise, making it more challenging to discern the underlying data manifold. A dimension of l = 128 l=128 thus appears to strike an optimal balance, providing sufficient capacity for a robust representation while maintaining a structured and learnable latent space for the downstream generative model.

Scale. Finally, we examine the impact of scaling the autoencoder. We explored several axes of scaling: doubling the number of layers in both the encoder and decoder to 4, doubling the hidden dimension to 1024, and expanding the training dataset to 100B tokens. Interestingly, none of these scaling efforts resulted in a significant improvement in the final BrierLM score. This finding suggests that the autoencoder’s task is inherently simple and does not benefit from the aggressive scaling. A lightweight architecture, trained on a relatively modest amount of data, is sufficient to learn the high-fidelity and robust representation required for our framework. This is a desirable property, as it allows the autoencoder to be a computationally negligible component of the overall system.

### 7.4 Effect of Model Architecture

In this section, we conduct ablation studies on the CALM model architecture to investigate the impact of different design choices on model performance. Unless otherwise specified, all experiments are conducted using a base configuration with a hidden size of 768, 12 hidden layers, 16 attention heads, and an FFN intermediate size of 2048. The generative head consists of 3 MLP blocks, and all models are trained for 50,000 steps.

Diffusion and Flow Matching. Since the generative head can be any continuous generative model, we also evaluate two prominent choices as alternatives: diffusion ( Ho et al., 2020 ) and flow matching ( Lipman et al., 2023 ) . For these experiments, we adopt an architecture consistent with the diffusion head used in Li et al. (2024) . To ensure a fair comparison, we replicate the input hidden state N = 8 N=8 times for both models, mirroring the multi-sample approach used for our energy loss and promoting stable learning. During inference, we use 100 iterative steps by default. For the Flow Matching model, we use a midpoint sampler. Figure 9 compares the performance of the diffusion, flow matching, and energy-based generative heads. The results show that both flow matching and our energy-based head outperform the diffusion model, exhibiting a noticeable performance gap. Between the two, flow matching exhibits faster initial convergence, whereas our energy-based head reaches a higher performance ceiling.

Figure 9 further compares the models’ performance across different numbers of inference iterations. For the flow matching model, we tested both the Euler and midpoint samplers. As shown, the diffusion model requires a large number of iterations to generate valid results. In contrast, the flow matching model is significantly more efficient; the midpoint sampler, in particular, achieves decent quality in just 2 steps and reaches its near-optimal performance within 4 steps. Our energy-based generative head achieves the best of both worlds: it delivers superior performance while completely eliminating the need for iterative decoding, making it a compelling choice for the CALM framework.

Energy Loss. We now analyze the impact of the energy loss formulation on model performance. Our energy loss (Equation 10 ) involves two sampling hyperparameters: the number of model-generated samples, N N , and the number of target samples, M M . Larger values for N N and M M provide a better estimation of the true energy score, but also increase the computational cost. Our default configuration is N = 8 N=8 and M = 100 M=100 . Table 3 shows the results of varying N N and M M , which reveal a clear trade-off between performance and computational cost. As expected, increasing the number of samples consistently improves the BrierLM score, but this comes at a nearly linear increase in training cost. Our default setting of N = 8 N=8 and M = 100 M=100 is thus justified as a balanced configuration, leveraging a moderately sized N N for a robust gradient signal and a large M M to stabilize training.

We also investigate the effect of the exponent α \alpha in the energy score (Equation 9 ), which is guaranteed to be strictly proper for any α ∈ ( 0 , 2 ) \alpha\in(0,2) . As shown in Table 4 , our empirical results align with this theoretical property. We observe that training fails for α < 1 \alpha<1 (e.g., α = 0.75 \alpha=0.75 ), a phenomenon previously analyzed by Shao et al. (2025b) and attributed to gradient explosion issues. For values of α \alpha within the range of [ 1 , 2 ) [1,2) , the model achieves decent performance, with the best empirical results obtained at our default setting of α = 1 \alpha=1 . The model’s BrierLM score drops to 0 0 at α = 2 \alpha=2 . This is expected, as the energy score is only proper but not strictly proper when α = 2 \alpha=2 . Consequently, the energy loss can no longer guide the model to uniquely match the true data distribution, leading to a collapse in modeling capability.

Model Input. A critical design choice in our CALM framework is the input representation fed into the Transformer backbone at each autoregressive step. We evaluate three distinct input schemes: (1) Discrete input, which first decodes the previously generated vector 𝐳 i − 1 \mathbf{z}_{i-1} into K K discrete tokens, then passes them through an embedding layer and an input compression MLP to form the next step input; (2) Continuous input, a more direct alternative where the vector 𝐳 i − 1 \mathbf{z}_{i-1} is directly projected to the Transformer’s hidden dimension via a single linear layer; (3) Combined input, which fuses the representations from the discrete and continuous methods through element-wise addition.

As summarized in Table 5 , the results clearly favor the discrete input strategy. The combined input offers no advantage and slightly degrades performance, while the purely continuous input leads to a substantial performance drop. This confirms our hypothesis: although the continuous vector theoretically contains all the information of its corresponding discrete tokens, its highly compact and brittle nature makes it challenging for the model to unpack the underlying semantic information. Grounding the autoregressive process in the discrete token space provides a more structured and stable input signal, which is therefore critical for achieving optimal performance.

### 7.5 Temperature Sampling

In this section, we conduct a fine-grained analysis to characterize the practical behavior of our approximate temperature sampling algorithm (Algorithm 2 ), i.e., how the algorithm navigates the trade-off between predictive accuracy and generative diversity. To quantify them, we decompose the Brier score estimator, Brier ( P , y ) ≈ 𝕀 { x 1 = y } + 𝕀 { x 2 = y } − 𝕀 { x 1 = x 2 } \text{Brier}(P,y)\approx\mathbb{I}\{x_{1}=y\}+\mathbb{I}\{x_{2}=y\}-\mathbb{I}\{x_{1}=x_{2}\} , into two metrics: • Accuracy 𝔼 [ 𝕀 { x = y } ] \mathbb{E}[\mathbb{I}\{x=y\}] : This metric measures the probability that a single sample drawn from the model matches the ground truth. It directly reflects the model’s accuracy.

• Collision Rate 𝔼 [ 𝕀 { x 1 = x 2 } ] \mathbb{E}[\mathbb{I}\{x_{1}=x_{2}\}] : This metric measures the collision probability that two independent samples drawn from the model are identical. It serves as an inverse proxy for diversity, where a higher collision rate indicates that the output distribution is less diverse.

Consistent with our primary BrierLM metric, we report both accuracy and collision rate as the geometric mean of scores over n-grams from n=1 to 4. We first investigate how the two key hyperparameters of our algorithm—temperature T T and batch size N N —influence these metrics. Specifically, we conduct two sets of experiments: for a fixed temperature T = 1 / 3 T=1/3 , we vary the batch size N ∈ { 1 , 10 , 20 , 50 , 100 , 200 , 500 , 1000 } N\in\{1,10,20,50,100,200,500,1000\} ; for a fixed batch size N = 500 N=500 , we vary the the temperature T ∈ { 1 / 2 , 1 / 3 , 1 / 4 , 1 / 5 , 1 / 6 } T\in\{1/2,1/3,1/4,1/5,1/6\} .

As shown in Figure 11 , both increasing the batch size N N and decreasing the temperature T T sharpen the output distribution, achieving higher accuracy at the cost of reduced diversity (i.e., a higher collision rate). A key observation, however, is the dominant role of the batch size N N , which covers a substantially greater range of this trade-off than the temperature T T . Intuitively, a larger batch provides a clearer statistical picture of the true distribution, making it easier to indentify high-probability candidates and confidently output them. In contrast, the effectiveness of temperature T T is capped by the information available within the finite batch. Thus, while temperature serves its conventional purpose, these empirical results suggest that the batch size N N is a more effective tool for navigating the accuracy-diversity frontier in our likelihood-free framework.

Finally, we compare the behavior of our sampling algorithm against that of a traditional Transformer. We ensure a fair comparison by selecting model checkpoints with nearly identical BrierLM scores. For CALM, we fix the temperature at T = 1 / 3 T=1/3 while varying the batch size N ∈ { 1 , 10 , 20 , 50 , 100 , 200 , 500 , 1000 } N\in\{1,10,20,50,100,200,500,1000\} ; for the Transformer baseline, we adjust its softmax temperature across T ∈ { 1 , 0.9 , … , 0.4 } T\in\{1,0.9,\dots,0.4\} .

The results, plotted in Figure 11 , are compelling: the accuracy-diversity trajectory traced by tuning N N in CALM is nearly identical to the one produced by tuning T T in the traditional Transformer. This alignment shows that we can accurately replicate the generative behavior of traditional models across a wide spectrum of temperatures. For instance, matching T = 0.6 T=0.6 requires a batch size of approximately N = 100 N=100 , while simulating a lower temperature of T = 0.5 T=0.5 necessitates a larger batch of N = 200 N=200 . This suggests a clear and predictable trade-off: the ability to simulate lower-temperature, higher-fidelity generation comes at the cost of an increased number of samples.

## 8 Conclusion and Future Work

In this work, we challenge the inefficient, token-by-token paradigm of LLMs by introducing Continuous Autoregressive Language Models (CALM), a framework that shifts generation from discrete tokens to a continuous vector space where a single vector represents K tokens. To support this approach, we develop a comprehensive likelihood-free toolkit: a robust and high-fidelity autoencoder, the energy loss for generative modeling, the BrierLM metric for LM evaluation, and a new suite of algorithms for temperature sampling. Empirical results show that CALM achieves a superior performance-compute trade-off, highlighting a new scaling axis for language modeling: scaling the semantic bandwidth of each generative step to push the performance-compute frontier of LLMs.

Despite these promising results, CALM still has significant room for future architectural and algorithmic optimizations, as indicated by the performance gap between CALM at K=1 and a standard Transformer baseline. We identify several key areas with great potential for future research: • Autoencoder. The autoencoder is the cornerstone of the CALM framework, which directly governs the quality of the latent space. One key limitation of the current autoencoder is its primary focus on reconstruction, with less emphasis on semantic structure. A promising direction is to design an autoencoder that learns a semantically grounded latent space, where proximity in the latent space corresponds to semantic similarity. We note that building such semantically rich latent spaces has been a recent trend in the vision domain ( Zheng et al., 2025 ) . This could provide a powerful inductive bias for the downstream generative model. Another direction is the development of more powerful architectures. Designs that are context-aware or autoregressive, for example, could offer superior robustness and more reliable reconstruction.