Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics — part 3/4
Cleaned-text code points [31617, 47530). Bibliography omitted here; source.md includes it.
While our work shows that language models compare quantities with measurement units using heuristics based on numerical difference and unit-scale difference, it has several limitations.
First, our study focuses on controlled, single-step comparisons between two quantities with measurement units. This design allows us to analyze LM behavior in a controlled setting, but it does not cover more realistic multi-step quantitative reasoning, where unit conversion may be combined with retrieval, arithmetic, or reasoning over longer textual contexts. Extending the analysis to such settings is an important direction for future work.
Second, our surrogate analysis uses linear models to approximate LM behavior from candidate cue features. This choice makes the analysis interpretable and allows us to compare feature families directly, but it may miss nonlinear interactions among cues or more complex heuristic strategies. Similarly, our DAS analysis tests whether selected variables are represented in linear subspaces, rather than fully identifying the circuit components that implement these computations.
Third, extending our analysis to reasoning-oriented LMs or long chain-of-thought outputs is nontrivial. Such LMs may distribute comparison-relevant information over many generated tokens, making it less clear where and when unit-comparison information is encoded and used. Our current analysis focuses on short answer generation, where the relevant comparison information is localized around the input tokens. Future work could adapt the proposed analyses to reasoning LMs and multi-token reasoning traces.
## Ethical Considerations
All data created and/or used in this work was synthetically generated. All language models used in this study are publicly available. We strictly adhered to the terms and conditions of each model’s license. During the development of code and the writing of this paper, we made use of AI assistants, including large language models. All code snippets and textual content generated with the assistance of such tools were carefully reviewed and revised by the authors to ensure scientific integrity, accuracy, and ethical compliance.
## Appendix A Task Formulation and Overall Experimental Setup
This section provides additional details on the task formulation and experimental setup shared across our behavioral, surrogate, and causal analyses. We first describe the prompt templates and unit notations used in the behavioral analysis, and then explain the prompt and notation choices used in the surrogate and DAS analyses.
### A.1 Prompt Templates
For the behavioral analysis in § 3 , we evaluate LMs under a broad set of prompt designs. The prompts vary along two axes. First, we vary the comparison direction, asking either for the larger quantity or the smaller quantity. Second, we vary the position of the two quantities relative to the comparison phrase. In the postposed design, the two quantities appear after the comparison phrase, as in “Which is larger, q 1 q_{1} or q 2 q_{2} ?”. In the proposed design, the two quantities appear before the comparison phrase, as in “Between q 1 q_{1} and q 2 q_{2} , which is larger?”. For mass comparisons, we use “heavier” and “lighter” instead of “larger” and “smaller”. The full set of prompt templates is shown in Tbl. 4 .
We also vary the surface form of unit expressions. For each unit, we test both a short notation, such as “cm”, and a long notation, such as “centimeter”. The unit notations are listed in Tbl. 5 . This allows us to test whether the observed behavioral patterns depend on a particular surface form of measurement units.
Furthermore, to enable analysis of instruction-tuned LMs, we append “Answer only with one of the two quantities.” to the prompt templates in Tbl. 4 , ensuring that the LMs output one of the two quantities immediately after the prompt.
### A.2 Choice of Prompt for Main Analyses
In the behavioral analysis, we observe some variation across prompt templates and unit notations. In particular, the postposed design generally yields higher accuracy than the preposed design, and the surface form of the unit notation leads to small differences in performance. However, the main qualitative pattern is stable across these variations. Across prompt templates and unit notations, accuracy changes systematically with the Quantity Margin, and performance decreases near the comparison boundary. Based on this observation, the surrogate analysis in § 5 and the DAS analysis in § 6 use the first larger-comparison prompt in Tbl. 4 , with the postposed design and short unit notation. For Qwen3-4B-Base, we also confirm that the second larger-comparison prompt shows the same qualitative pattern.
### A.3 Generation and Evaluation
All generations are performed with greedy decoding. For the prompt templates in Tbl. 4 , the evaluated LMs consistently output either q 1 q_{1} or q 2 q_{2} . This property allows us to use a unified evaluation protocol across the three analyses. For the behavioral analysis, we evaluate predictions by exact match against the correct quantity. For the surrogate analysis, we compare the LM’s average log-probability of generating the two candidate answer strings, q 1 q_{1} and q 2 q_{2} , as continuations of the same prompt. For the DAS analysis, the same binary output structure allows us to train interventions using the correct candidate string as the supervision signal.
### A.4 Effects of Number Formatting, Unit Notation, and Tokenization
First of all, we simply used the default tokenizers provided with each LM. All values were rendered in ordinary decimal notation; scientific notation was not used. The fractional part always had three digits, while the integer part had zero to four digits depending on the value.
In Qwen3, digits and decimal points are mostly tokenized character by character. In Olmo3, integer and fractional parts are mainly tokenized in three-digit groups, with the decimal point as a separate token.
We also examined the short and long unit forms in Tbl. 5 . The long forms corresponding to mm, cm, km, mg, and kg are split into two tokens; most other short and long forms are single tokens. This pattern is shared by Qwen3 and Olmo3.
These surface differences may affect absolute accuracy. § C.3 indeed reports a tendency for long notation to outperform short notation. However, they are unlikely to drive the main findings for the following reasons: • Qwen3 and Olmo3 exhibit consistent qualitative patterns despite differences in number tokenization.
• Similar results are observed across multiple unit systems and unit notations.
• The surrogate analysis uses the average token log-probability of each answer string, reducing the direct effect of answer-token length on the preference margin.
### A.5 Motivation for Analyzing the Quantity Margin in Log Space
Our main empirical observation is that LM accuracy decreases near the comparison decision boundary. § 3 establishes this phenomenon, § 4 proposes an explanation based on the cue-combination hypothesis, and § 5 and § 6 test this hypothesis. This line of analysis requires a quantity that measures how close a comparison is to the decision boundary, motivating the introduction of the Quantity Margin.
One motivation for defining the Quantity Margin in log space comes from prior work, which suggests that LMs represent numerical scale logarithmically ( AlQuabeh et al., 2026 ) .
A second motivation comes from our own preliminary experiments, in which the log-scale Quantity Margin could be decoded substantially more accurately from the hidden state than the corresponding linear-scale margin. Specifically, in the metric-length setting, we used the hidden state of the last token of u 2 u_{2} at Layer 18 of Qwen3-4B-Base when prompted with the template defined in Eq. 1 . Using this representation as input, we separately trained PLS regressions to predict the log-scale Quantity Margin defined in Eq. 2 and the corresponding linear-scale margin. The linear-scale formulation is given by Q M ( q 1 , q 2 ) linear = r 1 s D ( u 1 ) − r 2 s D ( u 2 ) . QM(q_{1},q_{2})_{\mathrm{linear}}=r_{1}s_{D}(u_{1})-r_{2}s_{D}(u_{2}). (18)
The regressions were trained on 5,000 examples and evaluated on 1,000 held-out examples. The number of PLS components was varied from 10 to 20. Performance was evaluated using both R 2 R^{2} and the mean absolute error (MAE).
The results are shown in Fig. 6 . The log-scale Quantity Margin is substantially easier to decode than the corresponding linear-scale margin across all evaluated numbers of PLS components. These results provide empirical support for our choice to formulate the Quantity Margin in log space.
## Appendix B On the Terms “Heuristic”, “Heuristic Aggregation”, and “Bag of Heuristics”
In this section, we define the terms “Heuristic,” “Heuristic Aggregation,” and “Bag of Heuristics” as used in this paper.
Our methods (surrogate analysis and DAS) are intended to provide a simpler, more interpretable, high-level account of otherwise opaque LM predictions, following the framework of causal abstraction proposed by Geiger et al. (2025) . Consequently, neither our methods nor the resulting findings uniquely identify the computational mechanism used by LMs for quantities with measurement units.
One concrete form of such a high-level account is to explain LM behavior through combinations of simple, interpretable heuristics, as in the “bag of heuristics” analysis of Nikankin et al. (2025) . Specifically, Nikankin et al. (2025) use the term “bag of heuristics” to describe an account that explains LM behavior through a combination of multiple heuristics. Our use of the term follows this interpretation.
In this paper, we define a heuristic as a comparison cue that is predictive of the LM’s answer without fully implementing the correct comparison rule. For example, the Quantity Margin fully implements the correct comparison rule and is therefore not itself a heuristic. We further use the term heuristic aggregation to denote a weighted combination of such heuristics. We provide the following formal definitions for clarity.
#### Heuristic.
For quantities q 1 q_{1} and q 2 q_{2} , a heuristic is a comparison rule based on simple, interpretable cues, such as numerical difference, unit-scale difference, unit identity, or thresholds, that does not implement the correct comparison rule g g over the full input domain. Formally, a comparison rule f f is a heuristic if ∃ ( q 1 , q 2 ) f ( q 1 , q 2 ) ≠ g ( q 1 , q 2 ) . \exists(q_{1},q_{2})\quad f(q_{1},q_{2})\neq g(q_{1},q_{2}). For example, a rule based only on the numerical values or only on the unit scales is a heuristic. By contrast, the sign of the Quantity Margin agrees with the correct comparison rule over the full domain and is therefore not a heuristic.
#### Heuristic Aggregation.
Let c 1 , … , c m c_{1},\ldots,c_{m} denote heuristic cues. Heuristic aggregation is a high-level account that combines these cues using weights w j w_{j} , A ( q 1 , q 2 ) = ∑ j = 1 m w j c j ( q 1 , q 2 ) , A(q_{1},q_{2})=\sum_{j=1}^{m}w_{j}\,c_{j}(q_{1},q_{2}), to explain the LM’s preference or output margin. This abstraction does not uniquely identify the LM’s underlying computational mechanism. It explains complex behavior through a weighted combination of simple, interpretable cues.
## Appendix C Behavioral Observation: LMs Become Less Accurate Near the Boundary
In § 3 , we showed that LM accuracy in quantity comparison is strongly organized by the Quantity Margin. Accuracy remains high when the absolute margin is large, but decreases as the comparison approaches the decision boundary. In this section, we examine the robustness of this gradual margin-dependent pattern and the additional differences that arise across LMs, prompt templates, and unit notations.
### C.1 Results Across LMs
Fig. 7 shows the results for Qwen3-4B-Base, Qwen3-8B-Base, Qwen3-14B-Base, Olmo-3-1025-7B, and Olmo-3-1125-32B. In addition, Fig. 8 reports the corresponding results for the instruct LMs Qwen3-4B and Qwen3-8B.
Across all five base LMs and two instruction-tuned LMs, accuracy changes systematically with the Quantity Margin. Comparisons are easy when the absolute margin is large and become gradually more difficult as the margin approaches zero. This indicates that the boundary-related degradation observed in § 3 is not specific to a single LM.
Fig. 7 also shows differences across unit settings. Heterogeneous metric-imperial comparisons tend to be less accurate than comparisons within a single unit system. For length comparisons, metric comparisons are generally easier than imperial comparisons. Overall, the same margin-dependent pattern is preserved across LMs and unit settings.
### C.2 Effects of Prompt Templates
We next examine the effect of prompt design. As described in Tbl. 4 , we vary both the comparison direction and the position of the quantities relative to the comparison phrase. For the comparison direction, we ask either for the larger quantity or for the smaller quantity. For the phrase position, we compare a postposed design, where the quantities q 1 q_{1} and q 2 q_{2} appear after the comparison phrase, with a preposed design, where the quantities appear before the comparison phrase. For example, the postposed design uses prompts such as “Which is larger, q 1 q_{1} or q 2 q_{2} ?”, whereas the preposed design uses prompts such as “Between q 1 q_{1} and q 2 q_{2} , which is larger?”. Fig. 9 shows the results across these prompt templates for Qwen3-4B-Base. The main margin-dependent pattern is stable across prompt designs. For all prompt templates, comparisons become gradually more difficult as the Quantity Margin approaches zero. At the same time, the prompt wording affects the absolute accuracy. Although the trend is not perfectly consistent across all unit settings, the postposed prompts, shown by solid curves, tend to achieve higher accuracy than the preposed prompts, shown by dashed curves. Similarly, prompts asking for the larger quantity tend to perform better than prompts asking for the smaller quantity, although this effect is also not fully consistent across all settings.
### C.3 Effects of Unit Notation
We also examine whether the surface form of units affects LM behavior. As described in Tbl. 5 , we compare short unit notation, such as cm , with long unit notation, such as centimeter . This allows us to test whether the observed behavioral patterns depend on the surface form used to express measurement units. Fig. 10 shows the results for Qwen3-4B-Base. The margin-dependent pattern again remains stable across unit notations. For both short and long unit forms, accuracy decreases as the Quantity Margin approaches zero. However, the choice of notation affects the absolute accuracy. Although the trend is not perfectly consistent across all settings, long-unit notation, shown by red curves, tends to achieve higher accuracy than short-unit notation, shown by blue curves. These results suggest that unit surface forms can influence performance, while the overall boundary-related degradation remains robust.
### C.4 Generated Outputs from Reasoning LMs
We conducted a behavioral analysis of representative generated outputs from Qwen3-4B-Thinking-2507 ( Yang et al., 2025 ) . Representative outputs are provided in Tbl. 6 . Analysis of these outputs shows that, at the output level, the model often performs correct unit conversion explicitly. It sometimes considers multiple solution paths, including conversions in both directions. However, for easy comparisons with a large Quantity Margin, the model may still rely on heuristic statements, such as noting that one quantity is sufficiently larger than the other.
## Appendix D Cue-Combination Hypothesis: Quantity Decisions Depend on Cue Agreement