In-Context Algebra

arxiv:2512.16902

Original source · Paper JSON

Reasoning and the "physics" of language models · 2025-12-18

Cites (7) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• In-Context Algebra
• Abstract
• 1 Introduction
• 2 Task Description
• 3 Can transformers learn in-context algebra?
• 4 Hypothesizing Model Mechanisms
• 4.1 Measuring Coverage and Performance on Targeted Distributions
• 5 Causal Verification of Learned Mechanisms
• 5.1 Causal Interventions
• 5.2 Copying and Commutative Copying
• 5.3 Identity Recognition
• 5.4 Closure-Based Cancellation
• 6 Phase Transitions Correspond to Learning of Discrete Skills
• 7 Related Work
• 8 Conclusion
• Ethics Statement
• Acknowledgments
• Appendix A Group Theory
• Appendix B Discussion: Contextual Reasoning is Richer Than Just Copying
• Appendix C Model Architecture and Training
• C.1 Model Architecture Hyperparameters
• C.2 Group Sampling Probability and Task Diversity
• C.3 Performance on Groups and Non-Group Algebraic Structures
• C.4 Additional Performance on Data Subsets
• Appendix D Additional Results on Copying
• Appendix E Identity Recognition Details
• Appendix F Closure and Elimination Subspaces
• F.1 How are Closure and Cancellation Sets Computed?
• Cancellation Set Construction.
• F.2 Subspace Construction
• F.2.1 Subspace Training Details
• F.2.2 Probing the Closure and Elimination Subspaces
• Appendix G Data Coverage Pseudocode
• Appendix H Use of Large Language Models
• Instructions for reporting errors