Continuous Autoregressive Language Models — part 3/6

Cleaned-text code points [31747, 46620). Bibliography omitted here; source.md includes it.

In contrast, a naive metric like the raw likelihood of the observed outcome, P ⁡ ( y ) P(y) , fails this principle. The expected score under this metric, 𝔼 y ∼ Q ​ [ P ​ ( y ) ] \mathbb{E}_{y\sim Q}[P(y)] , is maximized by a deterministic prediction that assigns a probability of 1 to the single most frequent outcome, i.e., P ⁡ ( arg ⁡ max y ⁡ Q ⁡ ( y ) ) = 1 P(\arg\max_{y}Q(y))=1 . Such a metric would therefore incorrectly favor an overconfident model that fails to capture the underlying data uncertainty. This highlights a critical distinction: a principled metric must balance rewarding accuracy with correctly representing the predictive uncertainty. The naive likelihood P ⁡ ( y ) P(y) only addresses the former, making it an inadequate measure of a model’s predictive quality.

### 4.2 BrierLM: Brier for Language Modeling

For a principled and likelihood-free evaluation, we turn to the Brier score ( Brier, 1950 ) , a classic strictly proper scoring rule now widely used to assess the calibration of modern neural networks ( Lakshminarayanan et al., 2017 ; Ovadia et al., 2019 ; Gruber & Buettner, 2022 ) . For a predictive distribution P P and a ground-truth outcome y y , the Brier score is defined as: Brier ​ ( P , y ) = 2 ​ P ​ ( y ) − ∑ x P ​ ( x ) 2 . \text{Brier}(P,y)=2P(y)-\sum_{x}P(x)^{2}. (12) Unlike the raw likelihood P ⁡ ( y ) P(y) , which solely measures accuracy, the Brier score incorporates an additional term, ∑ x P ​ ( x ) 2 \sum_{x}P(x)^{2} , to quantify predictive uncertainty. This structure balances two competing objectives, which ultimately rewards a well-calibrated prediction. This property is revealed by the following decomposition of the expected Brier score: 𝔼 y ∼ Q ​ [ Brier ​ ( P , y ) ] = − ∑ x ( P ⁡ ( x ) − Q ⁡ ( x ) ) 2 ⏟ Squared Error (minimized at ​ P = Q ​ ) + ∑ x Q ​ ( x ) 2 ⏟ Data Variance (constant) . \mathbb{E}_{y\sim Q}[\text{Brier}(P,y)]=-\underbrace{\sum_{x}(P(x)-Q(x))^{2}}_{\text{Squared Error (minimized at }P=Q\text{)}}+\underbrace{\sum_{x}Q(x)^{2}}_{\text{Data Variance (constant)}}. (13)

While the Brier score is theoretically sound, its direct computation remains intractable for CALM, as it requires knowledge of the full predictive distribution P P . We find, however, that an unbiased Monte Carlo estimator for the Brier score can be constructed in an entirely likelihood-free manner, using only samples drawn from the model. Specifically, the uncertainty term, ∑ x P ​ ( x ) 2 \sum_{x}P(x)^{2} , can be interpreted as the collision probability of two independent samples. Therefore, its unbiased estimator is simply the indicator function 𝕀 { x 1 = x 2 } \mathbb{I}{\{x_{1}=x_{2}\}} , where x 1 , x 2 ∼ P x_{1},x_{2}\sim P . Similarly, the accuracy term P ⁡ ( y ) P(y) can be estimated by 𝕀 { x = y } \mathbb{I}{\{x={y}\}} using a single sample x ∼ P x\sim P . Combining these, we construct a practical, unbiased estimator for the Brier score using two samples drawn from the model: Brier ( P , y ) ≈ 𝕀 { x 1 = y } + 𝕀 { x 2 = y } − 𝕀 { x 1 = x 2 } , x 1 , x 2 ∼ P . \text{Brier}(P,y)\approx\mathbb{I}\{x_{1}=y\}+\mathbb{I}\{x_{2}=y\}-\mathbb{I}\{x_{1}=x_{2}\},\quad{x}_{1},{x}_{2}\sim P. (14)

This estimator enables a likelihood-free evaluation of CALM’s predictive capabilities. A straightforward approach is to assess next-token prediction performance in a teacher-forcing setting. This would involve generating two latent vectors at each step, decoding them using the frozen autoencoder’s decoder, and computing the Brier score using only the first token of each resulting chunk. However, such an evaluation is insufficient as it ignores the generation quality of the remaining K − 1 K-1 tokens. To address this limitation, we further introduce Brier- n n , a metric that computes the Brier score over entire n-grams. In this formulation, the indicator functions of the estimator treat the n-gram as a single, atomic outcome. Finally, following the convention of established n-gram-based metrics like BLEU ( Papineni et al., 2002 ) , we define our composite metric, BrierLM (Brier for Language Modeling), as the geometric mean of Brier- n n scores for n = 1 n=1 to 4 4 , which we then scale by 100 to place it on a more interpretable 0-100 range: BrierLM = 100 ⋅ ( ∏ n = 1 4 Brier- ​ n ) 0.25 . \text{BrierLM}=100\cdot\left(\prod_{n=1}^{4}\text{Brier-}n\right)^{0.25}. (15)

The utility of BrierLM extends beyond CALM, serving as a universal evaluation protocol that is also applicable to conventional autoregressive models. For such models, the BrierLM estimator can be applied by simply drawing samples from the final softmax distribution, enabling direct and fair comparisons with our likelihood-free framework. To validate this, we evaluated both cross-entropy and BrierLM throughout the training of our baseline autoregressive models (detailed in Section 7.1 ). Figure 3 visualizes the joint distribution of the two metrics. Intriguingly, we find that BrierLM is highly consistent with cross-entropy loss, exhibiting a nearly linear relationship with a Pearson correlation coefficient of -0.966 and a Spearman’s rank correlation of -0.991. This strong monotonic alignment confirms that BrierLM is a reliable measure of language modeling capability, establishing it as a trustworthy likelihood-free alternative to Perplexity.

Furthermore, BrierLM offers a particularly significant advantage for the growing class of implicit generative models, such as diffusion-based language models ( Austin et al., 2021 ; Han et al., 2023 ; Lou et al., 2024 ; Arriola et al., 2025 ) . These models have historically been challenging to evaluate, often relying on the complex and sometimes loose estimation of variational lower bounds (ELBOs) to approximate Perplexity. BrierLM circumvents this entire challenge, offering a direct, unbiased method to faithfully assess their language modeling capabilities and enabling fair comparisons across different model families.

## 5 Likelihood-Free Temperature Sampling

### 5.1 Exact Temperature Sampling via Rejection

Controlled generation via temperature sampling is an indispensable feature of modern LLMs. Conventionally, this technique is implemented by rescaling pre-softmax logits, a mechanism that requires explicit access to the model’s probability distribution. However, this approach is incompatible with our CALM framework, whose generative head is likelihood-free and provides only a sampler. This presents a critical challenge: performing temperature sampling with only a black-box sampler. In this section, we address this challenge by developing an exact algorithm, grounded in the principles of rejection sampling, that provably achieves this goal.

The intuition for our algorithm stems from the relationship between repeated sampling and probability exponentiation. In the context of CALM, a sample x x corresponds to a complete chunk of K K tokens produced at each step. Consider the simple case where the temperature T = 1 / n T=1/n for an integer n n , which makes the target distribution P T ​ ( x ) ∝ P ​ ( x ) n P_{T}(x)\propto P(x)^{n} . The probability of drawing the exact same sample x x in n n independent trials from the sampler is also P ​ ( x ) n P(x)^{n} . This motivates an elegant rejection sampling scheme: we draw n n samples and accept them if and only if all n n samples are identical. Otherwise, we reject the entire set and restart the process. The distribution of accepted samples is thus provably proportional to P ​ ( x ) n P(x)^{n} , providing a foundation for our general algorithm.

To generalize this approach for any arbitrary temperature T ∈ ( 0 , 1 ) T\in(0,1) , we decompose the exponent 1 / T 1/T into its integer part, n = ⌊ 1 / T ⌋ n=\lfloor 1/T\rfloor , and fractional part, α = 1 / T − n \alpha=1/T-n . This decomposition structures our algorithm as a two-stage rejection sampling process. The first stage handles the integer component n n using the repetition-based scheme described above, producing a candidate sample x x only if n n independent draws are identical. The second stage, which handles the fractional exponent α \alpha , requires a more subtle approach. Here, we draw upon the theory of Bernoulli Factory ( Keane & O’Brien, 1994 ; Mendo, 2019 ) to construct an iterative procedure that simulates a biased coin flip with a success probability of P ​ ( x ) α P(x)^{\alpha} . A sample is accepted only if it passes both stages; failure at any point triggers a restart of the entire process. The complete procedure is formally detailed in Algorithm 1 . The following theorem guarantees its correctness.

###### Theorem 1 .

For an implicit discrete distribution P ⁡ ( x ) P(x) with sampler S S and a temperature T ∈ ( 0 , 1 ) T\in(0,1) , Algorithm 1 generates samples distributed as: P T ​ ( x ) = P ​ ( x ) 1 / T Z T , Z T = ∑ x P ​ ( x ) 1 / T . P_{T}(x)=\frac{P(x)^{1/T}}{Z_{T}},\quad Z_{T}=\sum_{x}P(x)^{1/T}.

The proof is provided in Appendix A.1 .

### 5.2 Expected Sampling Cost

While Algorithm 1 provides an exact solution for likelihood-free temperature sampling, its practical viability hinges on its computational efficiency. A central concern is the expected number of samples it requires, as each sampler call involves a forward pass through the generative head and autoencoder. Although these forward passes can be executed in parallel during inference, a prohibitively large number of samples would still create a significant computational bottleneck. The following theorem provides a closed-form expression for this expected number of sampler calls, with Corollary 2.1 offering a more interpretable upper bound. The proof is provided in Appendix 2 .

###### Theorem 2 .

The expected number of calls to the base sampler S S , denoted 𝔼 ⁡ [ N total ] \mathbb{E}[N_{\text{total}}] , required to generate one sample using Algorithm 1 is: 𝔼 ⁡ [ N total ] = n + 𝕀 ⁡ ( α > 0 ) ​ ∑ x P ​ ( x ) 1 / T − 1 Z T \mathbb{E}[N_{\text{total}}]=\frac{n+\mathbb{I}(\alpha>0)\sum_{x}P(x)^{1/T-1}}{Z_{T}} where Z T = ∑ x P ​ ( x ) 1 / T Z_{T}=\sum_{x}P(x)^{1/T} , n = ⌊ 1 / T ⌋ n=\lfloor 1/T\rfloor , α = 1 / T − n \alpha=1/T-n , and 𝕀 ⁡ ( ⋅ ) \mathbb{I}(\cdot) is the indicator function.

###### Corollary 2.1 .

Let | 𝒳 | |\mathcal{X}| be the size of sample space. The expected number of sampler calls 𝔼 ⁡ [ N total ] \mathbb{E}[N_{\text{total}}] at temperature T ∈ ( 0 , 1 ) T\in(0,1) is bounded by: 𝔼 ⁡ [ N total ] ≤ { 1 + n Z T , if ​ 0 < T ≤ 0.5 1 + | 𝒳 | 2 − 1 / T Z T , if ​ 0.5 < T < 1 \mathbb{E}[N_{\text{total}}]\leq\begin{cases}\displaystyle\frac{1+n}{Z_{T}},&\text{if }\ 0<T\leq 0.5\\ \displaystyle\frac{1+|\mathcal{X}|^{2-1/T}}{Z_{T}},&\text{if }\ 0.5<T<1\end{cases} where n = ⌊ 1 / T ⌋ n=\lfloor 1/T\rfloor and Z T = ∑ x P ​ ( x ) 1 / T Z_{T}=\sum_{x}P(x)^{1/T} .

These results highlight that the algorithm’s practicality is highly sensitive to the temperature T T . A potential limitation first emerges for T → 1 T\to 1 , as the cost can scale up to the size of sample space | 𝒳 | = | 𝒱 | K |\mathcal{X}|=|\mathcal{V}|^{K} . It is therefore advisable to avoid using temperatures in this high-temperature regime to prevent a potential computational bottleneck. Conversely, at low temperatures, the integer part n = ⌊ 1 / T ⌋ n=\lfloor 1/T\rfloor becomes large. The algorithm’s success requires drawing n n identical samples, an event with a vanishingly small probability for a large n n that leads to an extremely high rejection rate. A more sample-efficient approximate algorithm is therefore needed to enhance its practical utility.

### 5.3 Batch Approximation

The practical limitations of the exact algorithm become most pronounced in the low-temperature regime, where the requirement of drawing n = ⌊ 1 / T ⌋ n=\lfloor 1/T\rfloor identical samples leads to an extremely high rejection rate that results in poor sample utilization. To address this, we propose an efficient approximate algorithm tailored for low temperatures of the form T = 1 / n T=1/n . The key insight is to shift from a single, high-risk trial to a combinatorial search within a large batch of N N samples ( N ≫ n N\gg n ). This shift allows a single batch to constitute ( N n ) \binom{N}{n} distinct candidates, which dramatically improves sample utilization and increases the probability of finding a successful match in a single round.

For example, to sample at T = 0.5 ​ ( n = 2 ) T=0.5(n=2) , we might draw a batch of N = 10 N=10 samples, such as { A , C , A , D , B , E , A , F , B , G } \{A,C,A,D,B,E,A,F,B,G\} . Here, sample A A appears three times, and sample B B appears twice. The algorithm then counts the number of successful n n -tuple candidates within this batch. For sample A A , there are ( 3 2 ) = 3 \binom{3}{2}=3 successful candidates. For sample B B , there is only ( 2 2 ) = 1 \binom{2}{2}=1 successful candidate. Finally, the output is sampled from the set of valid candidates { A , B } \{A,B\} according to their weighted probabilities, where P ⁡ ( A ) = 3 / 4 P(A)=3/4 and P ⁡ ( B ) = 1 / 4 P(B)=1/4 . In the rare case that no sample appears at least n n times, the candidate set would be empty. To ensure the algorithm always produces an output, we introduce a fallback mechanism that iteratively reduce the matching requirement from n n to n − 1 n-1 , n − 2 n-2 , … \dots , until a non-empty candidate set is found. The detailed process is illustrated in Algorithm 2 .

For any finite batch size N N , the algorithm is biased. This bias arises because the output probability is determined by the ratio of weights calculated within a single stochastic batch, and the expectation of a ratio is generally not equal to the ratio of expectations. However, its key strength is that it is asymptotically unbiased: as the batch size N N approaches infinity, the output distribution converges to the true target distribution. We formalize this crucial property in the following theorem.

###### Theorem 3 .

Let P alg ​ ( x , N ) P_{\text{alg}}(x;N) be the probability of sampling x x using Algorithm 2 with a batch size of N N , and let P T ​ ( x ) = P ​ ( x ) n / Z T P_{T}(x)=P(x)^{n}/Z_{T} be the true target distribution at temperature T = 1 / n T=1/n , where Z T = ∑ x P ​ ( x ) n Z_{T}=\sum_{x}P(x)^{n} . The algorithm is asymptotically unbiased: lim N → ∞ P alg ​ ( x , N ) = P T ​ ( x ) . \lim_{N\to\infty}P_{\text{alg}}(x;N)=P_{T}(x).

The proof is provided in Appendix A.3 . This property of consistency establishes the algorithm as a principled approximation, where the batch size N N serves as a practical lever for the trade-off between efficiency and accuracy. Because the algorithm relies solely on a black-box sampling interface, its utility extends naturally beyond the CALM framework to the entire class of implicit language models. This positions it as a universal toolkit for controlled decoding in discrete spaces.

## 6 Related Work

### 6.1 Autoencoder