Continuous Autoregressive Language Models

arxiv:2510.27688

Original source · Paper JSON

LLMs: architectures, context, training · 2025-10-31

Cites (5) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• Continuous Autoregressive Language Models
• Abstract
• 1 Introduction
• 2 Autoencoder
• 2.1 High-Fidelity Reconstruction
• 2.2 Robust Vector Representation
• 3 Likelihood-Free Language Modeling
• 3.1 Next-Vector Prediction
• 3.2 Generative Head
• 3.3 Energy Transformer
• 3.3.1 Strictly Proper Scoring Rules
• 3.3.2 Energy Loss
• 3.3.3 Model Architecture
• 4 Likelihood-Free LM Evaluation
• 4.1 Principles of LM Evaluation
• 4.2 BrierLM: Brier for Language Modeling
• 5 Likelihood-Free Temperature Sampling
• 5.1 Exact Temperature Sampling via Rejection
• Theorem 1 .
• 5.2 Expected Sampling Cost
• Theorem 2 .
• Corollary 2.1 .
• 5.3 Batch Approximation
• Theorem 3 .
• 6 Related Work
• 6.1 Autoencoder
• 6.2 Likelihood-Free Language Modeling
• 6.3 Likelihood-Free LM Evaluation
• 6.4 Likelihood-Free Temperature Sampling
• 7 Experiments
• 7.1 Settings
• 7.2 Main Results
• 7.3 Effect of Autoencoder
• 7.4 Effect of Model Architecture
• 7.5 Temperature Sampling
• 8 Conclusion and Future Work
• Appendix A Proof
• A.1 Proof of Theorem 1
• Proof.
• A.2 Proof of Theorem 2
• Proof.
• Proof.
• A.3 Proof of Theorem 3
• Proof.
• Instructions for reporting errors