Continuous Autoregressive Language Models — part 6/6
Cleaned-text code points [78318, 93139). Bibliography omitted here; source.md includes it.
• Model. The core generative model also presents significant avenues for exploration. For instance, in terms of architecture, our current design employs a Transformer backbone followed by a lightweight generative head; while efficient, an alternative is to explore a more integrated, end-to-end generative Transformer, which may yield stronger generative modeling capabilities. In terms of the training objective, while the energy loss provides a robust foundation, investigating other strictly proper scoring rules or generative models is worthwhile, as they may offer different optimization dynamics and improved sample quality.
• Sampling. While our work introduces a provably exact algorithm for likelihood-free temperature sampling, its reliance on rejection sampling can introduce significant inference overhead. A promising direction for future work is to explore more lightweight, heuristic methods for navigating the diversity-fidelity trade-off at inference time. This could include techniques such as manipulating the scale of the input noise to the generative head, or fine-tuning the model with a modified loss function to steer its generative behavior.
• Scaling. A critical next step is to investigate the scaling properties of CALM. A hypothesis to validate is that larger models possess the requisite capacity to support higher semantic bandwidths. A further pursuit is to establish a new family of scaling laws. While traditional laws ( Kaplan et al., 2020 ) model performance as a function of model and data size, our framework introduces the semantic bandwidth K K as a third variable. Formulating a unified scaling law would enable the principled selection of an optimal K K for any compute budget.
• Algorithmic Toolkit. The paradigm shift from discrete tokens to a continuous domain necessitates a re-evaluation of the standard LLM algorithmic toolkit. For instance, policy optimization methods in reinforcement learning typically update the model by increasing the log-probability of rewarded samples, a quantity that CALM cannot directly compute. Similarly, knowledge distillation often requires minimizing the KL divergence between teacher and student distributions, which is intractable without access to the full probability mass function. How to reformulate these techniques to operate in a sample-based regime is an important question for future research.
## Appendix A Proof
### A.1 Proof of Theorem 1
See 1
###### Proof.
Algorithm 1 implements a rejection sampling scheme where sample x x is accepted with probability P accept ( x ) P_{\text{accept}}(x) . The proof proceeds by showing that the acceptance probability P accept ( x ) = P ( x ) 1 / T P_{\text{accept}}(x)=P(x)^{1/T} , so the rejection sampling procedure yields the desired normalized sample distribution: P T ( x ) = P accept ( x ) ∑ x P accept ( x ) = P ( x ) 1 / T ∑ x P ( x ) 1 / T . P_{T}(x)=\frac{P_{\text{accept}}(x)}{\sum_{x}P_{\text{accept}}(x)}=\frac{P(x)^{1/T}}{\sum_{x}P(x)^{1/T}}. (16)
The inverse temperature is decomposed as 1 / T = n + α 1/T=n+\alpha , where n = ⌊ 1 / T ⌋ n=\lfloor 1/T\rfloor is the integer part and α ∈ [ 0 , 1 ) \alpha\in[0,1) is the fractional part. The acceptance probability is the product of the success probabilities of the two corresponding stages.
Stage 1 (Integer Part): For the algorithm to proceed to stage 2 with a candidate sample x x , it must first draw x x for n n consecutive times in stage 1. As each draw is independent with probability P ( x ) P(x) , the probability of passing stage 1 with candidate x x is P ( x ) n P(x)^{n} .
Stage 2 (Fractional Part): Let p = P ( x ) p=P(x) . The probability of acceptance in stage 2 is the cumulative probability of being accepted at any given iteration i ≥ 1 i\geq 1 : P stage2 \displaystyle P_{\text{stage2}} = P ( accept at i = 1 ) + P ( pass i = 1 , accept at i = 2 ) + … \displaystyle=P(\text{accept at }i=1)+P(\text{pass }i=1,\text{accept at }i=2)+\dots (17) = p + ( 1 − p ) ( 1 − α 1 ) p + ( 1 − p ) 2 ( 1 − α 1 ) ( 1 − α 2 ) p + … \displaystyle=p+(1-p)\left(1-\frac{\alpha}{1}\right)p+(1-p)^{2}\left(1-\frac{\alpha}{1}\right)\left(1-\frac{\alpha}{2}\right)p+\dots = p ∑ k = 0 ∞ ( 1 − p ) k ∏ j = 1 k ( 1 − α j ) \displaystyle=p\sum_{k=0}^{\infty}(1-p)^{k}\prod_{j=1}^{k}\left(1-\frac{\alpha}{j}\right) = p ∑ k = 0 ∞ ( p − 1 ) k ( α − 1 k ) \displaystyle=p\sum_{k=0}^{\infty}(p-1)^{k}\binom{\alpha-1}{k} = p ⋅ ( p − 1 + 1 ) α − 1 \displaystyle=p\cdot(p-1+1)^{\alpha-1} = p α . \displaystyle=p^{\alpha}. The second to last step is due to the generalized binomial theorem.
Total Probability: The total probability of accepting a sample x x in a single trial is the product of the probabilities from the two stages: P accept ( x ) = P ( x ) n ⋅ P ( x ) α = P ( x ) 1 / T , P_{\text{accept}}(x)=P(x)^{n}\cdot P(x)^{\alpha}=P(x)^{1/T}, (18) which completes the proof. ∎
### A.2 Proof of Theorem 2
See 2
###### Proof.
The algorithm conducts series of independent trials, each with identical probability of success, and continues until one trial is successful, thereby following a geometric distribution. Therefore, the expected number of samples is the ratio of the expected number of samples per trial, 𝔼 [ N t r i a l ] \mathbb{E}[N_{trial}] , to the probability of a trial’s success, P ( success ) P(\text{success}) : 𝔼 [ N t o t a l ] = 𝔼 [ N t r i a l ] P ( success ) . \mathbb{E}[N_{total}]=\frac{\mathbb{E}[N_{trial}]}{P(\text{success})}. (19)
Denominator (Success Probability): A trial is successful if any sample x x is accepted. The total success probability is the sum of acceptance probabilities over all possible outcomes: P ( success ) = ∑ x P accept ( x ) = ∑ x P ( x ) 1 / T = Z T . P(\text{success})=\sum_{x}P_{\text{accept}}(x)=\sum_{x}P(x)^{1/T}=Z_{T}. (20)
Numerator (Expected Calls per Trial): Let N t r i a l N_{trial} be the number of sampler calls in a single trial. A trial always involves N 1 N_{1} calls for stage 1 and may involve N 2 N_{2} calls for stage 2. The number of calls in Stage 1 is fixed at N 1 = n N_{1}=n . Thus, 𝔼 [ N 1 ] = n \mathbb{E}[N_{1}]=n .
If α = 0 \alpha=0 , stage 2 is never performed, so 𝔼 [ N 2 ] = 0 \mathbb{E}[N_{2}]=0 . If α > 0 \alpha>0 , stage 2 is only executed if stage 1 succeeds with some candidate x x . Let ℰ x \mathcal{E}_{x} be this event, so P ( ℰ x ) = P ( x ) n P(\mathcal{E}_{x})=P(x)^{n} , and we have: 𝔼 [ N 2 ] = ∑ x P ( ℰ x ) ⋅ 𝔼 [ N 2 | ℰ x ] . \mathbb{E}[N_{2}]=\sum_{x}P(\mathcal{E}_{x})\cdot\mathbb{E}[N_{2}|\mathcal{E}_{x}]. (21) The conditional expectation 𝔼 [ N 2 | ℰ x ] \mathbb{E}[N_{2}|\mathcal{E}_{x}] is the expected number of draws in stage 2 given candidate x x . Using the formula 𝔼 [ X ] = ∑ k = 1 ∞ P ( X ≥ k ) \mathbb{E}[X]=\sum_{k=1}^{\infty}P(X\geq k) , where X X is the number of draws in stage 2: 𝔼 [ N 2 | ℰ x ] \displaystyle\mathbb{E}[N_{2}|\mathcal{E}_{x}] = ∑ k = 1 ∞ P ( stage 2 requires at least k draws ) \displaystyle=\sum_{k=1}^{\infty}P(\text{stage 2 requires at least }k\text{ draws}) (22) = ∑ k = 0 ∞ ( 1 − P ( x ) ) k ∏ j = 1 k ( 1 − α j ) . \displaystyle=\sum_{k=0}^{\infty}(1-P(x))^{k}\prod_{j=1}^{k}\left(1-\frac{\alpha}{j}\right). This is the same sum we evaluated in Equation 17 , which equals P ( x ) α − 1 P(x)^{\alpha-1} . Therefore, if α > 0 \alpha>0 : 𝔼 [ N 2 ] = ∑ x P ( x ) n ⋅ P ( x ) α − 1 = ∑ x P ( x ) n + α − 1 = ∑ x P ( x ) 1 / T − 1 . \mathbb{E}[N_{2}]=\sum_{x}P(x)^{n}\cdot P(x)^{\alpha-1}=\sum_{x}P(x)^{n+\alpha-1}=\sum_{x}P(x)^{1/T-1}. (23) Combining the two cases, the total expected number of calls per trial is: 𝔼 [ N trial ] = 𝔼 [ N 1 ] + 𝔼 [ N 2 ] = n + 𝕀 ( α > 0 ) ∑ x P ( x ) 1 / T − 1 . \mathbb{E}[N_{\text{trial}}]=\mathbb{E}[N_{1}]+\mathbb{E}[N_{2}]=n+\mathbb{I}(\alpha>0)\sum_{x}P(x)^{1/T-1}. (24) Combining the numerator and denominator gives the final result: 𝔼 [ N total ] = n + 𝕀 ( α > 0 ) ∑ x P ( x ) 1 / T − 1 Z T . \mathbb{E}[N_{\text{total}}]=\frac{n+\mathbb{I}(\alpha>0)\sum_{x}P(x)^{1/T-1}}{Z_{T}}. (25) ∎
See 2.1
###### Proof.
The proof is divided into two cases based on the temperature range. We start from the general formula for the expected cost from Theorem 2 : 𝔼 [ N total ] = n + 𝕀 ( α > 0 ) ∑ x P ( x ) 1 / T − 1 Z T . \mathbb{E}[N_{\text{total}}]=\frac{n+\mathbb{I}(\alpha>0)\sum_{x}P(x)^{1/T-1}}{Z_{T}}. (26)
Case 1: Low-Temperature Regime ( 0 < T ≤ 0.5 0<T\leq 0.5 )
In this range, the exponent 1 / T − 1 ≥ 1 1/T-1\geq 1 . Since P ( x ) ∈ [ 0 , 1 ] P(x)\in[0,1] , for any exponent β ≥ 1 \beta\geq 1 , we have P ( x ) β ≤ P ( x ) P(x)^{\beta}\leq P(x) . Thus, by summing over the entire sample space 𝒳 \mathcal{X} : ∑ x ∈ 𝒳 P ( x ) 1 / T − 1 ≤ ∑ x ∈ 𝒳 P ( x ) = 1 . \sum_{x\in\mathcal{X}}P(x)^{1/T-1}\leq\sum_{x\in\mathcal{X}}P(x)=1. (27) The numerator of the cost formula is therefore bounded by n + 1 n+1 : n + 𝕀 ( α > 0 ) ∑ x ∈ 𝒳 P ( x ) 1 / T − 1 ≤ n + 𝕀 ( α > 0 ) ⋅ 1 ≤ n + 1 . n+\mathbb{I}(\alpha>0)\sum_{x\in\mathcal{X}}P(x)^{1/T-1}\leq n+\mathbb{I}(\alpha>0)\cdot 1\leq n+1. (28) This establishes the bound for the low-temperature regime.
Case 2: High-Temperature Regime ( 0.5 < T < 1 0.5<T<1 )
In this range,the exponent β = 1 / T − 1 \beta=1/T-1 is in the interval ( 0 , 1 ) (0,1) . For such an exponent, the function f ( p ) = p β f(p)=p^{\beta} is concave. By Jensen’s inequality, the sum ∑ x ∈ 𝒳 P ( x ) β \sum_{x\in\mathcal{X}}P(x)^{\beta} is maximized when P ( x ) P(x) is a uniform distribution over the sample space, i.e., P ( x ) = 1 / K P(x)=1/K for all x ∈ 𝒳 x\in\mathcal{X} . The bound is: ∑ x ∈ 𝒳 P ( x ) 1 / T − 1 ≤ ∑ x ∈ 𝒳 ( 1 K ) 1 / T − 1 = K ⋅ ( 1 K ) 1 / T − 1 = K 2 − 1 / T . \sum_{x\in\mathcal{X}}P(x)^{1/T-1}\leq\sum_{x\in\mathcal{X}}\left(\frac{1}{K}\right)^{1/T-1}=K\cdot\left(\frac{1}{K}\right)^{1/T-1}=K^{2-1/T}. (29) Substituting this into the cost formula gives the bound for the high-temperature regime. This completes the proof. ∎
### A.3 Proof of Theorem 3
See 3
###### Proof.
Let ℬ = { x 1 , … , x N } \mathcal{B}=\{x_{1},\dots,x_{N}\} be a batch of N N samples drawn i.i.d. from the base distribution P ( x ) P(x) . For any sample x ∈ 𝒳 x\in\mathcal{X} , let C x C_{x} be the random variable for the count of x x in ℬ \mathcal{B} . The weight assigned to x x is W x = ( C x n ) W_{x}=\binom{C_{x}}{n} . Let X N X_{N} be the random variable for the probability of sampling x x from ℬ \mathcal{B} : X N ( x ) = W x ∑ z ∈ 𝒳 W z = ( C x n ) ∑ z ∈ 𝒳 ( C z n ) . X_{N}(x)=\frac{W_{x}}{\sum_{z\in\mathcal{X}}W_{z}}=\frac{\binom{C_{x}}{n}}{\sum_{z\in\mathcal{X}}\binom{C_{z}}{n}}. (30)
The overall probability we seek to analyze is the expectation of this random variable: P alg ( x , N ) = 𝔼 [ X N ] P_{\text{alg}}(x;N)=\mathbb{E}[X_{N}] . Our goal is to show that lim N → ∞ 𝔼 [ X N ] = P T ( x ) \lim_{N\to\infty}\mathbb{E}[X_{N}]=P_{T}(x) . The proof proceeds in two main steps: first, we show that the random variable X N X_{N} converges in probability to P T ( x ) P_{T}(x) ; second, we use the Bounded Convergence Theorem to show that this implies the convergence of its expectation.
1. Convergence in Probability. By the Weak Law of Large Numbers, the proportion of occurrences of any sample x x converges in probability to its true probability P ( x ) P(x) : C x N → 𝑝 P ( x ) as N → ∞ . \frac{C_{x}}{N}\xrightarrow{p}P(x)\quad\text{as }N\to\infty. (31) The weight W x W_{x} can be written as a polynomial in C x C_{x} , W x = 1 n ! C x ( C x − 1 ) … ( C x − n + 1 ) W_{x}=\frac{1}{n!}C_{x}(C_{x}-1)\dots(C_{x}-n+1) . We normalize this by N n N^{n} : W x N n = 1 n ! ( C x N ) ( C x − 1 N ) … ( C x − n + 1 N ) . \frac{W_{x}}{N^{n}}=\frac{1}{n!}\left(\frac{C_{x}}{N}\right)\left(\frac{C_{x}-1}{N}\right)\dots\left(\frac{C_{x}-n+1}{N}\right). (32) Since C x N → 𝑝 P ( x ) \frac{C_{x}}{N}\xrightarrow{p}P(x) and k N → 0 \frac{k}{N}\to 0 for any constant k k , each term in the product converges to P ( x ) P(x) . By the Continuous Mapping Theorem, the entire expression converges in probability: W x N n → 𝑝 1 n ! P ( x ) n . \frac{W_{x}}{N^{n}}\xrightarrow{p}\frac{1}{n!}P(x)^{n}. (33) Now, we analyze the random variable X N X_{N} by dividing its numerator and denominator by N n N^{n} : X N = W x / N n ∑ z ∈ 𝒳 ( W z / N n ) . X_{N}=\frac{W_{x}/N^{n}}{\sum_{z\in\mathcal{X}}(W_{z}/N^{n})}. (34) Applying the Continuous Mapping Theorem again for the ratio, we show that the random variable X N X_{N} converges in probability to P T ( x ) P_{T}(x) : X N → 𝑝 1 n ! P ( x ) n ∑ z ∈ 𝒳 1 n ! P ( z ) n = P ( x ) n ∑ z ∈ 𝒳 P ( z ) n = P T ( x ) . X_{N}\xrightarrow{p}\frac{\frac{1}{n!}P(x)^{n}}{\sum_{z\in\mathcal{X}}\frac{1}{n!}P(z)^{n}}=\frac{P(x)^{n}}{\sum_{z\in\mathcal{X}}P(z)^{n}}=P_{T}(x). (35)
2. Convergence of Expectation. We have established that the random variable X N X_{N} converges in probability to P T ( x ) P_{T}(x) . Besides, we have that X N X_{N} is inherently bounded: 0 ≤ X N = W x ∑ z ∈ 𝒳 W z ≤ 1 . 0\leq X_{N}=\frac{W_{x}}{\sum_{z\in\mathcal{X}}W_{z}}\leq 1. (36)
We can now invoke the Bounded Convergence Theorem, which states that if a sequence of random variables X N X_{N} converges in probability to X X , and | X N | ≤ M |X_{N}|\leq M for all N N for some constant M M , then lim N → ∞ 𝔼 [ X N ] = 𝔼 [ lim N → ∞ X N ] \lim_{N\to\infty}\mathbb{E}[X_{N}]=\mathbb{E}[\lim_{N\to\infty}X_{N}] . Applying this theorem to our case: lim N → ∞ P alg ( x , N ) = lim N → ∞ 𝔼 [ X N ] = 𝔼 [ lim N → ∞ X N ] = P T ( x ) . \lim_{N\to\infty}P_{\text{alg}}(x;N)=\lim_{N\to\infty}\mathbb{E}[X_{N}]=\mathbb{E}\left[\lim_{N\to\infty}X_{N}\right]=P_{T}(x). (37) This completes the proof that the algorithm is asymptotically unbiased.
∎
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .