Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics — part 2/4

Cleaned-text code points [15700, 31617). Bibliography omitted here; source.md includes it.
Tbl. 2 summarizes the representative surrogate families used in the main analysis. These families are based on a systematic feature design described in § E.1 . Starting from the four primitive variables that determine the Quantity Margin, r 1 r_{1} , s D  ( u 1 ) s_{D}(u_{1}) , r 2 r_{2} , and s D  ( u 2 ) s_{D}(u_{2}) , we construct feature families corresponding to primitive components, number/unit differences, shared-scale global quantities. We further instantiate these cues as signed, thresholded, and continuous features to capture heuristics at different levels of granularity. The representative families are chosen to compare the main candidate explanations of LM behavior: the Quantity Margin, global quantities, primitive numerical and unit components, number/unit differences, and broad heuristic cue sets. Each surrogate predicts the LM log-probability margin from one cue family or a combination of cue families. The full list of individual features is provided in Tbl. 8 , and the full list of surrogate models is provided in Tbl. 9 .

We evaluate each surrogate model using its held-out coefficient of determination, denoted by R S 2 R^{2}_{\mathrm{S}} . As a reference, we also evaluate a baseline surrogate that uses only the Quantity Margin, the variable that determines the correct comparison outcome, and denote its held-out coefficient of determination by R QM 2 R^{2}_{\mathrm{QM}} . Furthermore, for each surrogate feature set, we evaluate a surrogate model obtained by adding the Quantity Margin to that feature set, yielding R QM + S 2 R^{2}_{\mathrm{QM+S}} .

Using these scores, we evaluate each surrogate model with two metrics, Δ  R 2 \Delta R^{2} and R partial 2 R^{2}_{\mathrm{partial}} . The metric Δ  R 2 \Delta R^{2} measures the improvement in held-out predictive performance over the Quantity Margin baseline. The partial R 2 R^{2} quantifies the proportion of variance left unexplained by the Quantity Margin baseline that is additionally explained when the candidate feature set is added. The metric R partial 2 R^{2}_{\mathrm{partial}} measures the proportion of the variance left unexplained by the Quantity Margin baseline that is additionally explained by the surrogate feature set.

Δ  R 2 = R S 2 − R QM 2 \Delta R^{2}=R^{2}_{\mathrm{S}}-R^{2}_{\mathrm{QM}} (11)

R partial 2 = R QM + S 2 − R QM 2 1 − R QM 2 R^{2}_{\mathrm{partial}}=\frac{R^{2}_{\mathrm{QM+S}}-R^{2}_{\mathrm{QM}}}{1-R^{2}_{\mathrm{QM}}} (12)

We further evaluate the surrogate models on the subset of boundary examples, where | Q  M  ( q 1 , q 2 ) | ≤ 0.2 |QM(q_{1},q_{2})|\leq 0.2 . As shown in Fig. 2 , this is the region where LM accuracy drops most clearly. Under the cue-combination hypothesis, this region is also where different cues are expected to become less consistently aligned. Analyzing these boundary examples therefore allows us to test which feature sets best explain LM behavior when different cues are in conflict.

#### Implementation details.

We construct a broad surrogate-analysis dataset by sampling examples evenly across quantity-margin bins. We use 20,000 examples for training, 1,000 examples for validation, and 5,000 examples for testing. The LM outputs used as regression targets are obtained with the same LMs, unit settings, prompt templates, and decoding setup as in § 2.2 . For each feature set, we fit a ridge regression surrogate model to predict m LM  ( q 1 , q 2 ) m_{\mathrm{LM}}(q_{1},q_{2}) . All features are standardized using the training set statistics. The regularization strength is selected separately for each surrogate model from α ∈ { 0.01 , 0.1 , 1 , 10 , 100 } \alpha\in\{0.01,0.1,1,10,100\} based on validation R 2 R^{2} , and the selected surrogate model is evaluated on the held-out test set.

### 5.2 Results

Tbl. 3 shows results for the representative surrogate families summarized in Tbl. 2 , using Qwen3-4B-Base on metric length comparisons. Here, signed features use only the direction of a cue, such as whether NumLogDiff > 0 \mathrm{NumLogDiff}>0 or UnitLogDiff > 0 \mathrm{UnitLogDiff}>0 , while threshold features use coarse magnitude information, such as whether a log-scale variable exceeds a threshold n n . Other continuous feature sets use the original log-scale values directly.

For the all examples setting, the LM’s behavior is well explained by heuristic features based on NumLogDiff \mathrm{NumLogDiff} and UnitLogDiff \mathrm{UnitLogDiff} . The thresholded and signed NumLogDiff/UnitLogDiff feature sets achieve high Δ  R 2 \Delta R^{2} and partial R 2 R^{2} values among the representative models. This indicates that the LM’s log-probability margin is especially well captured by coarse cues about which side has the larger numeral and which side has the larger unit scale. By contrast, feature sets based on GlobalLogX \mathrm{GlobalLogX} , GlobalLogY \mathrm{GlobalLogY} , or the primitive input components provide only limited improvements over the Quantity Margin baseline. These results suggest that LM behavior is better explained by heuristic cues derived from numerical difference and unit-scale difference than by these alternative feature representations.

We next focus on boundary examples, where | Q  M  ( q 1 , q 2 ) | ≤ 0.2 |QM(q_{1},q_{2})|\leq 0.2 , the region where LM accuracy drops most clearly in Fig. 2 . As expected, the Quantity Margin alone provides little explanatory power for the LM’s behavior on these boundary examples. Indeed, the Quantity Margin baseline achieves only R QM 2 = 0.022 R^{2}_{\mathrm{QM}}=0.022 in this region. In contrast, our NumLogDiff \mathrm{NumLogDiff} and UnitLogDiff \mathrm{UnitLogDiff} feature set achieves even higher Δ  R 2 \Delta R^{2} and R partial 2 R^{2}_{\mathrm{partial}} than on the full evaluation set. This suggests that, in cue-conflicting cases where the LM frequently fails, its behavior remains well predicted by heuristic features based on numerical difference and unit-scale difference.

Full results across LMs, unit settings are provided in App. E , where the same overall trend is consistently observed. The results for all surrogate models are reported in Tbl. 10 .

## 6 Causal Analysis: Number and Unit Differences Steer LM Decisions

In this section, we test whether causal interventions on NumLogDiff \mathrm{NumLogDiff} and UnitLogDiff \mathrm{UnitLogDiff} explain and steer LM behavior more effectively than interventions based on alternative candidate variables.

### 6.1 Experimental Setup

We use Distributed Alignment Search (DAS) ( Geiger et al., 2024 ) to test whether the variables identified by the surrogate analysis are represented in LM activations and causally affect the LM’s output. DAS learns subspaces of the LM representation aligned with high-level variables, and evaluates whether replacing these subspaces with those from source examples changes the LM’s output according to the corresponding high-level counterfactual. We report interchange intervention accuracy (IIA), the fraction of interventions for which the LM’s intervened prediction matches the high-level counterfactual label. Full details of the DAS objective and intervention operation are provided in App. F .

Our main high-level variables are the numerical difference and unit-scale difference: Z 1 = NumLogDiff , Z 2 = UnitLogDiff . Z_{1}=\mathrm{NumLogDiff},\quad Z_{2}=\mathrm{UnitLogDiff}. (13) For an intervention example e = ( b , s 1 , s 2 ) e=(b,s_{1},s_{2}) , we jointly intervene on both variables by taking NumLogDiff \mathrm{NumLogDiff} from s 1 s_{1} and UnitLogDiff \mathrm{UnitLogDiff} from s 2 s_{2} . The high-level counterfactual label is computed as y cf int ( e ) = 𝟙 [ \displaystyle y_{\mathrm{cf}}^{\mathrm{int}}(e)=\mathbbm{1}[ NumLogDiff ⁡ ( s 1 ) \displaystyle\mathrm{NumLogDiff}(s_{1}) (14) + UnitLogDiff ( s 2 ) > 0 ] . \displaystyle+\mathrm{UnitLogDiff}(s_{2})>0].

We compare this setting against three alternative high-level variable sets. The first is a shared-scale global-quantity baseline, which tests whether LM activations are better aligned with the two quantities represented on a common scale: Z 1 = GlobalLogX , Z 2 = GlobalLogY . Z_{1}=\mathrm{GlobalLogX},\qquad Z_{2}=\mathrm{GlobalLogY}. (15) The second is an identity baseline with four variables corresponding to the primitive log-scale numerical and unit-scale components, which tests whether the intervention effect can be explained by the original numerical and unit-scale components rather than by difference variables: Z 1 \displaystyle Z_{1} = log ⁡ r 1 , \displaystyle=\log r_{1}, Z 2 \displaystyle Z_{2} = log ⁡ s D  ( u 1 ) , \displaystyle=\log s_{D}(u_{1}), (16) Z 3 \displaystyle Z_{3} = log ⁡ r 2 , \displaystyle=\log r_{2}, Z 4 \displaystyle Z_{4} = log ⁡ s D  ( u 2 ) . \displaystyle=\log s_{D}(u_{2}).

The third is a NumLogDiff & Unit-Identity baseline, which combines the numerical-difference variable with the original unit-scale identity variables. This baseline tests whether the learned subspaces associated with the unit variables in our NumLogDiff & UnitLogDiff setting are better explained by unit-scale differences than by side-specific lexical signatures.

Z 1 \displaystyle Z_{1} = NumLogDiff , \displaystyle=\mathrm{NumLogDiff}, Z 2 \displaystyle Z_{2} = log ⁡ s D  ( u 1 ) , \displaystyle=\log s_{D}(u_{1}), (17) Z 3 \displaystyle Z_{3} = log ⁡ s D  ( u 2 ) , \displaystyle=\log s_{D}(u_{2}),

For all DAS settings, we fix the total dimensionality of the non-residual intervention subspaces to 1024 1024 , allocated evenly across variables. Thus, the two-variable settings use 512 512 dimensions per variable, the three-variable setting uses 341 341 dimensions per variable (with one remaining dimension unused), while the four-variable identity baseline uses 256 256 dimensions per variable. As robustness checks, we also vary the total intervention dimensionality to 256 256 , 512 512 , and 2048 2048 , evaluate intervention accuracy using 95% confidence intervals estimated from 10,000 random bootstrap sampling of the evaluation set, and repeat DAS training with three different random seeds. Across all evaluations, we observe the same qualitative pattern, with details provided in App. F . Each intervention example consists of a base input and one source input for each high-level variable. We use 20K intervention examples for training and 1K held-out examples for evaluation. We balance output-changing and output-preserving interventions in equal proportions, so the chance-level IIA is 0.5 0.5 . For all other experimental details, we follow the LMs, prompt templates, and unit settings described in § 2.2 .

### 6.2 Results

Fig. 4 shows the DAS intervention accuracy across layers and token positions for the NumLogDiff \mathrm{NumLogDiff} / UnitLogDiff \mathrm{UnitLogDiff} setting. The strongest effects concentrate around the last token of u 2 u_{2} . This suggests that information relevant to the comparison is integrated near the end of the second quantity. We therefore focus on the last token of u 2 u_{2} for the layerwise analysis below.

Fig. 5 reports layerwise DAS results at the last token of u 2 u_{2} for Qwen3-4B-Base on metric length comparisons. The NumLogDiff \mathrm{NumLogDiff} / UnitLogDiff \mathrm{UnitLogDiff} setting achieves the highest IIA across all reported layers. This indicates that intervening on the learned subspaces for numerical difference and unit-scale difference can causally steer the LM’s comparison output in the direction predicted by the high-level counterfactual.

Both the GlobalLogX \mathrm{GlobalLogX} / GlobalLogY \mathrm{GlobalLogY} , identity, and NumLogDiff & Unit-Identity baselines achieve above-chance IIA, indicating that global quantities and primitive input components are represented to some extent. However, all three remain consistently below the NumLogDiff \mathrm{NumLogDiff} / UnitLogDiff \mathrm{UnitLogDiff} setting. These gaps suggest that LM comparison outputs are more effectively steered by interventions aligned with NumLogDiff \mathrm{NumLogDiff} / UnitLogDiff \mathrm{UnitLogDiff} variables than by interventions aligned with the tested alternatives, and that the advantage of the UnitLogDiff setting is not fully explained by side-specific unit identities.

Thus, the numerical difference and unit-scale difference identified in the surrogate analysis are not merely correlated with LM behavior; they are represented in a way that can causally affect the LM’s decisions. We observe the same qualitative pattern across other LMs, unit settings, and prompt templates. Full results are provided in App. F .

## 7 Related Work

Prior work ( Park et al., 2022 ; Spokoyny et al., 2022 ; Göpfert et al., 2022 ; Xu et al., 2024 ; Huang et al., 2024 ; Bui et al., 2025 ) has examined empirical behavior of LMs on measurement-unit tasks, including basic measurement understanding ( Park et al., 2022 ; Spokoyny et al., 2022 ) , measurement extraction from text ( Göpfert et al., 2022 ) , unit-aware chain-of-thought reasoning ( Xu et al., 2024 ) , dimension-aware quantitative reasoning ( Huang et al., 2024 ) , and generalization across measurement systems ( Bui et al., 2025 ) . Building on these behavioral and task-oriented findings, we provide, to our knowledge, the first internal-mechanism analysis of comparing quantities with measurement units.

Prior work has shown that numerical information can be read out from the hidden states of LMs and that this information sometimes causally affects their behavior. Numerical values and attributes have been analyzed through linear directions ( Zhu et al., 2025 ) , low-dimensional subspaces ( Heinzerling and Inui, 2024 ; El-Shangiti et al., 2025 ; AlQuabeh et al., 2026 ) , digit-wise structure ( Levy and Geva, 2025 ) , and helical representations ( Kantamneni and Tegmark, 2025 ) . For numerical comparison, behavioral work has found magnitude-comparison effects in LMs ( Shah et al., 2023 ) , and mechanistic studies have analyzed how LMs compute greater-than judgments ( Hanna et al., 2023 ) . More closely related, Yuchi et al. (2026) show that hidden states encode both the magnitudes of numbers written in different numerical notations, such as standard and exponential notation 5.7 × 10 2 5.7\times 10^{2} , and which number is larger. While these works focus on numerical values alone, our setting requires LMs to integrate numerals with symbolic unit scales, which introduces an additional compositional step. In addition, we not only find where the answer is encoded, but also how the numerals and units jointly contribute to the LM’s behavior.

Regarding how LMs make judgments through heuristic aggregation, Nikankin et al. (2025) argue that, for the four basic arithmetic operations, LMs solve arithmetic problems by a diverse collection of nontrivial input- and result-pattern heuristics rather than exact algorithms. In contrast, our results suggest that quantity comparison is governed more centrally by the numerical difference and the unit-scale difference.

## 8 Conclusion

We studied how language models compare quantities with measurement units, such as 110 cm and 1.2 m . Across controlled unit settings, LM accuracy decreases near the comparison boundary. Through surrogate analysis, we found that LM preferences are better explained by numerical-difference and unit-scale-difference cues than by exact shared-scale quantities. Through DAS interventions, we further showed that numerical difference and unit-scale difference are represented in activation space and can causally steer LM’s outputs. Overall, our results suggest that LM behavior is better explained by a heuristic account centered on numerical and unit-scale cues than by an account based solely on exact unit conversion. This provides a controlled case study of how numerical information and symbolic measurement units are combined in language-model representations.

## Limitations