Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics — part 1/4
Cleaned-text code points [0, 15700). Bibliography omitted here; source.md includes it.
# Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics
###### Abstract
Quantities with measurement units, such as 110 cm and 1.2 m , require language models (LMs) to combine a numeral with a symbolic unit scale. Here, we study how LMs compare such quantities in controlled settings spanning several unit systems. We find that accuracy degrades near the comparison boundary, where small changes in value determine the correct answer. The resulting errors are systematic: linear surrogate models predict LM preferences from numerical-difference and unit-scale-difference cues, and causal interventions on subspaces aligned with these variables shift LM’s output. The results suggest that LM behavior is better explained by a bag of heuristics account over numeral and unit-scale cues than by an account based solely on first converting both expressions into exact shared-scale representations.
## 1 Introduction
Prior work has shown that language models (LMs) encode and compare numerical information in an interpretable manner. Behavioral studies have found magnitude-comparison effects in LM representations ( Shah et al., 2023 ) , while mechanistic work has analyzed how LMs compute greater-than judgments and compare numeric properties ( Hanna et al., 2023 ; El-Shangiti et al., 2025 ) . Other work has studied how numerical values or numeric properties are represented internally ( Heinzerling and Inui, 2024 ; Levy and Geva, 2025 ) . However, quantities in text often appear with measurement units, such as 110 cm , 1.2 m , or 3 kg . Comparing such quantities requires the LM to combine the numeral with the unit scale: 110 cm has a larger numeral than 1.2 m , but denotes a smaller length.
Here, we study comparison of quantities with measurement units as a setting where numerical comparison must be performed over heterogeneous expressions. Unlike comparison of bare numbers, each expression contains both a numeral and a symbolic unit, and the larger numeral need not correspond to the larger quantity. Quantity comparison also differs from prior interpretability work on fixed arithmetic functions over two numerals, including the four basic arithmetic operations such as addition, subtraction, multiplication, and division ( Stolfo et al., 2023 ; Quirke and Barez, 2024 ; Quirke et al., 2025 ; Zhang et al., 2024 ; Zhou et al., 2024 ) , as well as modular addition ( Zhong et al., 2023 ; Nanda et al., 2023 ; Ding et al., 2024 ) . Here, the relevant operation is not a fixed arithmetic function over two numerals: the LM must combine each numeral with a symbolic unit and compare the resulting magnitudes on a shared scale. The setting, therefore, gives a controlled test case for extending interpretability work from numerical operations to quantity representations in text.
Our results suggest that exact shared-scale conversion alone does not fully account for LM behavior. From a causal abstraction perspective ( Geiger et al., 2025 ) , the behavior is better explained by a high-level bag-of-heuristics account over numerical and unit-scale cues. Across LMs and unit systems, accuracy degrades near the comparison boundary. The resulting errors are systematic: linear surrogate models predict LM preferences from numerical-difference and unit-scale-difference features, and causal interventions on subspaces aligned with these variables shift the LM’s output in the corresponding direction. This parallels recent evidence that LMs solve arithmetic through a bag of heuristics ( Nikankin et al., 2025 ) . While the relevant heuristics for the four basic arithmetic operations include diverse and nontrivial task-solving cues, such as operand ranges, modulo patterns, and result ranges, numerical difference and unit-scale difference emerge as particularly important heuristic cues in our quantity-comparison setting.
By heuristics, we mean comparison cues that are predictive of the LM’s answer without fully implementing the correct comparison rule (formal definitions are provided in App. B ). In this setting, such cues include whether the left numeral is larger than the right numeral, whether the left unit has a larger scale than the right unit, and thresholded versions of these differences. Aggregating heuristics then means that the LM’s preference can be predicted from a weighted combination of such cues. This account differs from exact unit conversion: it can explain correct answers when the cues agree, but also predicts systematic errors when numeral and unit-scale cues point in different directions.
We test this account in three steps, summarized in Fig. 1 . First, we measure behavioral accuracy across controlled quantity comparisons and show that performance degrades near the comparison boundary. Second, we fit surrogate models over candidate cues and find that numerical-difference and unit-scale-difference features predict LM preferences better than exact shared-scale quantities. Third, we use causal interventions to show that these variables are represented in activation space and can shift the LM’s output. Together, these results suggest that LM behavior in quantity comparison is better explained by heuristic aggregation over numerical and unit-scale cues, providing a controlled case study of how numerical and symbolic information are combined in language-model representations.
## 2 Task Formulation and Overall Experimental Setup
To analyze how LMs combine numerical values with symbolic unit information, we study the comparison of quantities with measurement units as a controlled task. This section describes the formulation of the comparison task over quantities with measurement units, as well as the experimental setup shared across this study.
### 2.1 Task Formulation
Let D D denote a physical dimension, such as length or mass, and let U D U_{D} be a chosen set of mutually comparable measurement units for that dimension. A quantity with a measurement unit is specified by a numerical value r ∈ [ r min , r max ] ⊂ ℝ > 0 r\in[r_{\min},r_{\max}]\subset\mathbb{R}_{>0} and a unit u ∈ U D u\in U_{D} , and is written as r u ru . Given two quantities q 1 = r 1 u 1 q_{1}=r_{1}u_{1} and q 2 = r 2 u 2 q_{2}=r_{2}u_{2} , where u 1 , u 2 ∈ U D u_{1},u_{2}\in U_{D} , the LM is asked to decide which is larger: Which is larger, r 1 u 1 or r 2 u 2 ? \text{Which is larger, }r_{1}u_{1}\text{ or }r_{2}u_{2}? (1)
For example, in the comparison “Which is larger, 110 cm or 1.2 m?”, we have D = length D=\mathrm{length} , U D = { mm , cm , m , km } U_{D}=\{\mathrm{mm},\mathrm{cm},\mathrm{m},\mathrm{km}\} , r 1 = 110 r_{1}=110 , u 1 = cm u_{1}=\mathrm{cm} , r 2 = 1.2 r_{2}=1.2 , and u 2 = m u_{2}=\mathrm{m} .
For each comparison q 1 = r 1 u 1 q_{1}=r_{1}u_{1} and q 2 = r 2 u 2 q_{2}=r_{2}u_{2} , the correct answer can be determined by the log-scale difference between the two quantities after accounting for unit scale: Q M ( q 1 , q 2 ) \displaystyle QM(q_{1},q_{2}) = log ( r 1 ) − log ( r 2 ) \displaystyle=\log(r_{1})-\log(r_{2}) (2) + log ( s D ( u 1 ) ) − log ( s D ( u 2 ) ) . \displaystyle+\log(s_{D}(u_{1}))-\log(s_{D}(u_{2})). Here, s D ( u ) s_{D}(u) denotes the scale of unit u u relative to a base unit for dimension D D . We refer to Q M ( q 1 , q 2 ) QM(q_{1},q_{2}) as the Quantity Margin. Throughout this paper, we use the Quantity Margin to quantify how close a comparison is to the decision boundary and to analyze LM behavior systematically across different numerical values and unit pairs. We formulate the Quantity Margin in the log space, following prior work ( AlQuabeh et al., 2026 ) and our preliminary experiments under the unit settings used in this study ( § A.5 ).
q 1 q_{1} is larger when Q M ( q 1 , q 2 ) > 0 QM(q_{1},q_{2})>0 , and the right quantity q 2 q_{2} is larger when Q M ( q 1 , q 2 ) < 0 QM(q_{1},q_{2})<0 .
For the example above, taking meters as the base unit gives s D ( cm ) = 10 − 2 s_{D}(\mathrm{cm})=10^{-2} and s D ( m ) = 1 s_{D}(\mathrm{m})=1 . Thus, Q M ( 110 cm , 1.2 m ) = log ( 110 ) − log ( 1.2 ) + log ( 10 − 2 ) − log ( 1 ) ≈ − 0.04 . QM(110\mathrm{cm},1.2\mathrm{m})=\log(110)-\log(1.2)+\log(10^{-2})-\log(1)\approx-0.04.
The negative margin indicates that 1.2 m 1.2\mathrm{m} is larger, and its small absolute value indicates that the comparison is close to the decision boundary.
### 2.2 Overall Experimental Setup
We consider five unit settings, each defined by a physical dimension D D and a mutually comparable unit set U D U_{D} , as shown in Tbl. 1 . For numerical values, we sample r r from the range 10 − 3 ≤ r < 10 4 10^{-3}\leq r<10^{4} across all unit settings. The first three settings contain units from a single unit system, while the last two combine metric and imperial units, allowing us to test whether the observed behavior is specific to a single unit system or generalizes across both homogeneous and heterogeneous unit comparisons.
We evaluate five base LMs (Qwen3-4B-Base, Qwen3-8B-Base, Qwen3-14B-Base ( Yang et al., 2025 ) , Olmo-3-1025-7B, and Olmo-3-1125-32B ( Olmo et al., 2026 ) ) and two instruct LMs (Qwen3-4B and Qwen3-8B). We use the default tokenizer provided with each LM, and represent all numerical values in standard decimal notation (i.e., without scientific notation).
We use multiple prompt templates that ask the LMs to output the larger quantity itself, use greedy decoding, and evaluate predictions by exact match with the correct quantity. Further details on prompt templates, unit scales, tokenization, and evaluation are provided in App. A .
## 3 Behavioral Observation: LMs Become Less Accurate Near the Boundary
In this section, we examine at the behavioral level under what conditions LMs succeed or fail at comparing quantities with measurement units.
### 3.1 Experimental Setup
To analyze LMs’ behavior systematically, we evaluate their accuracy on the quantity comparison task defined in § 2 . We create a controlled dataset by varying both the numerical values and the unit pairs. We group 5,000 comparison examples into 18 bins according to their Quantity Margin, defined in § 2 , resulting in 277 or 278 examples per bin. The bins cover negative and positive margins symmetrically, with finer intervals near the decision boundary, e.g., [ − 0.2 , 0 ) [-0.2,0) and [ 0 , 0.2 ) [0,0.2) . Using the LMs, prompt templates, and unit settings described in § 2 , we then evaluate LMs’ accuracy within each bin.
### 3.2 Results
Fig. 2 shows the result for Qwen3-4B-Base across the five unit settings. LM’s accuracy is strongly organized by the Quantity Margin. Accuracy remains high when the absolute margin is large, but decreases sharply as examples approach the comparison boundary. This indicates that comparisons are easy for LMs when the Quantity Margin is large and become gradually more difficult as the margin approaches zero. The same margin-dependent pattern appears across length and mass comparisons, as well as across homogeneous and heterogeneous unit comparisons. At the same time, heterogeneous metric-imperial comparisons tend to be less accurate than comparisons within a single unit system, with metric length comparisons generally easier than imperial length comparisons. Additional results in App. C show that the margin-dependent pattern is robust across LMs, prompt templates, and unit notations, while overall accuracy varies with prompt wording and unit surface form, such as m m vs. m e t e r meter .
## 4 Cue-Combination Hypothesis: Quantity Decisions Depend on Cue Agreement
We interpret the drop in LM accuracy near the Quantity Margin boundary shown in Fig. 2 through a cue-combination hypothesis, under which LM’s preferences can be explained by a weighted combination of multiple comparison cues. Under this view, comparison difficulty depends on how consistently the cues favor the same answer. Far from the boundary, the cues tend to support the same answer and reinforce one another, whereas near the boundary, this support becomes weaker or less consistent.
The cue-combination view raises the question of which cues actually explain its behavior. We therefore first enumerate candidate cues that could support quantity comparison, and then test their predictive power in the surrogate analysis in § 5 .
Several comparison cues are naturally available in this task. The numerical-difference and unit-scale cues are defined as NumLogDiff = log r 1 − log r 2 , \mathrm{NumLogDiff}=\log r_{1}-\log r_{2}, (3) UnitLogDiff = log s D ( u 1 ) − log s D ( u 2 ) . \mathrm{UnitLogDiff}=\log s_{D}(u_{1})-\log s_{D}(u_{2}). (4) The magnitude of each quantity on a common scale can also serve as a cue: GlobalLogX = log r 1 + log s D ( u 1 ) , \mathrm{GlobalLogX}=\log r_{1}+\log s_{D}(u_{1}), (5) GlobalLogY = log r 2 + log s D ( u 2 ) . \mathrm{GlobalLogY}=\log r_{2}+\log s_{D}(u_{2}). (6)
In addition to these comparison-related quantities, LMs may also rely on surface-level heuristic cues derived from the input representation, including numerical appearance, firstdigit ( r 1 ) > n , \mathrm{firstdigit}(r_{1})>n, (7) where n n is a threshold, unit identities, u 2 ∈ { km , kg , mile , … } , u_{2}\in\{\mathrm{km},\mathrm{kg},\mathrm{mile},\ldots\}, (8) and familiar unit-pair patterns, ( u 1 , u 2 ) = ( cm , m ) ⇒ q 2 . (u_{1},u_{2})=(\mathrm{cm},\mathrm{m})\Rightarrow q_{2}. (9)
As a concrete example of cue agreement and conflict, we group examples by NumLogDiff \mathrm{NumLogDiff} and UnitLogDiff \mathrm{UnitLogDiff} . For metric length comparisons, we bin NumLogDiff \mathrm{NumLogDiff} over [ − 6 , 6 ] [-6,6] with width 1, and use the discrete UnitLogDiff \mathrm{UnitLogDiff} values determined by unit pairs, { − 6 , − 5 , − 3 , − 2 , − 1 , 0 , 1 , 2 , 3 , 5 , 6 } \{-6,-5,-3,-2,-1,0,1,2,3,5,6\} . We generate 200 examples for each grid cell and evaluate LM accuracy in each cell. Fig. 3 shows that errors concentrate near the diagonal region where numerical-difference and unit-scale cues cancel each other out, yielding a small Quantity Margin. This pattern illustrates the cue-combination view: comparisons are easy when the relevant cues jointly support the same side, and difficult when they are weak or conflicting. The same qualitative pattern appears across the other unit settings, as shown in App. D .
## 5 Surrogate Analysis: Number and Unit Differences Best Predict LM Behavior
In this section, we test which cues best predict the LM’s actual behavior.
### 5.1 Experimental Setup
We use surrogate models to test which candidate quantities and cues explain the LM’s comparison behavior. For each comparison example, we instantiate a prompt Prompt ( q 1 , q 2 ) \mathrm{Prompt}(q_{1},q_{2}) , such as “Which is larger, q 1 q_{1} or q 2 q_{2} ? The answer is”. We then compare the LM’s average log-probability of generating the two candidate answer strings, namely the left quantity a q 1 a_{q_{1}} and the right quantity a q 2 a_{q_{2}} , as continuations of this prompt: m LM ( q 1 , q 2 ) \displaystyle m_{\mathrm{LM}}(q_{1},q_{2}) = ℓ ¯ θ ( a q 1 ∣ Prompt ( q 1 , q 2 ) ) \displaystyle=\bar{\ell}_{\theta}(a_{q_{1}}\mid\mathrm{Prompt}(q_{1},q_{2})) (10) − ℓ ¯ θ ( a q 2 ∣ Prompt ( q 1 , q 2 ) ) . \displaystyle-\bar{\ell}_{\theta}(a_{q_{2}}\mid\mathrm{Prompt}(q_{1},q_{2})). Here, ℓ ¯ θ ( a ∣ Prompt ) \bar{\ell}_{\theta}(a\mid\mathrm{Prompt}) denotes the average token log-probability of the full candidate string a a when appended to the prompt. A positive value indicates that the LM prefers the left quantity, while a negative value indicates that it prefers the right quantity.