Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics

arxiv:2606.03982

Original source · Paper JSON

Reasoning and the "physics" of language models · 2026-06-02

Cites (5) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics
• Abstract
• 1 Introduction
• 2 Task Formulation and Overall Experimental Setup
• 2.1 Task Formulation
• 2.2 Overall Experimental Setup
• 3 Behavioral Observation: LMs Become Less Accurate Near the Boundary
• 3.1 Experimental Setup
• 3.2 Results
• 4 Cue-Combination Hypothesis: Quantity Decisions Depend on Cue Agreement
• 5 Surrogate Analysis: Number and Unit Differences Best Predict LM Behavior
• 5.1 Experimental Setup
• Implementation details.
• 5.2 Results
• 6 Causal Analysis: Number and Unit Differences Steer LM Decisions
• 6.1 Experimental Setup
• 6.2 Results
• 7 Related Work
• 8 Conclusion
• Limitations
• Ethical Considerations
• Appendix A Task Formulation and Overall Experimental Setup
• A.1 Prompt Templates
• A.2 Choice of Prompt for Main Analyses
• A.3 Generation and Evaluation
• A.4 Effects of Number Formatting, Unit Notation, and Tokenization
• A.5 Motivation for Analyzing the Quantity Margin in Log Space
• Appendix B On the Terms “Heuristic”, “Heuristic Aggregation”, and “Bag of Heuristics”
• Heuristic.
• Heuristic Aggregation.
• Appendix C Behavioral Observation: LMs Become Less Accurate Near the Boundary
• C.1 Results Across LMs
• C.2 Effects of Prompt Templates
• C.3 Effects of Unit Notation
• C.4 Generated Outputs from Reasoning LMs
• Appendix D Cue-Combination Hypothesis: Quantity Decisions Depend on Cue Agreement
• Appendix E Surrogate Analysis: Number and Unit Differences Best Predict LM Behavior
• E.1 Surrogate Feature Design
• E.2 Additional Surrogate Results
• Appendix F Causal Analysis: Number and Unit Differences Steer LM Decisions
• F.1 Distributed Alignment Search
• F.2 Robustness Across LMs, Unit Settings, Prompts, and training random seeds
• F.3 Robustness to Intervention Dimensionality
• Instructions for reporting errors