Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics — part 4/4
Cleaned-text code points [47530, 63390). Bibliography omitted here; source.md includes it.
In § 4 , we introduced the cue-combination hypothesis, where quantity decisions are explained by the agreement and conflict among comparison cues. As a concrete example, we analyzed the interaction between numerical-difference and unit-scale-difference cues, NumLogDiff \mathrm{NumLogDiff} and UnitLogDiff \mathrm{UnitLogDiff} . In this section, we then demonstrate that the cue-agreement pattern shown in Fig. 3 is consistently observed across different unit settings.
Fig. 11 shows LM accuracy grouped by NumLogDiff \mathrm{NumLogDiff} and UnitLogDiff \mathrm{UnitLogDiff} for each unit setting. Across unit settings, accuracy is high in regions where the two cues support the same answer. In contrast, errors concentrate near the diagonal region where the two cues cancel each other out, yielding a small Quantity Margin. This shows that the cue-conflict pattern discussed in § 4 is not specific to metric length comparisons, but is shared across length, mass, and metric-imperial comparisons.
## Appendix E Surrogate Analysis: Number and Unit Differences Best Predict LM Behavior
In § 5 , we showed that LM comparison behavior is well explained by heuristic cues based on numerical difference and unit-scale difference. In this section, we first describe the full design of the surrogate features. We then provide additional surrogate results and show that the main trend is consistent across unit settings, LMs, and prompt templates. In particular, surrogate models based on signed or thresholded NumLogDiff \mathrm{NumLogDiff} and UnitLogDiff \mathrm{UnitLogDiff} consistently explain LM behavior well.
### E.1 Surrogate Feature Design
Tbl. 8 summarizes the full set of heuristic features considered in our surrogate analysis. We design these features to cover semantically distinct ways of decomposing a quantity comparison while avoiding redundant feature families.
The Quantity Margin in Eq. 2 is determined by four primitive variables: the left numeral r 1 r_{1} , the left unit scale s D ( u 1 ) s_{D}(u_{1}) , the right numeral r 2 r_{2} , and the right unit scale s D ( u 2 ) s_{D}(u_{2}) . We therefore begin with features corresponding to these primitive components. In Tbl. 8 , these include Left number log, Right number log, Left unit-scale log, and Right unit-scale log.
We then construct additional feature families by combining subsets of these four primitive variables. One possible decomposition combines three variables on one side and one variable on the other. For example, Left value in right unit together with Right number log represents a comparison between the right numeral and the left quantity converted into the right unit scale. Similarly, the right value in the left unit together with the left number log represents the corresponding comparison in the left unit scale.
Another decomposition splits the four primitive variables into two pairs. The split ( r 1 , r 2 ) (r_{1},r_{2}) and ( s D ( u 1 ) , s D ( u 2 ) ) (s_{D}(u_{1}),s_{D}(u_{2})) yields the numerical-difference and unit-scale-difference variables, NumLogDiff \mathrm{NumLogDiff} and UnitLogDiff \mathrm{UnitLogDiff} . This feature family tests whether LM behavior is explained by separately comparing numerals and unit scales. The split ( r 1 , s D ( u 1 ) ) (r_{1},s_{D}(u_{1})) and ( r 2 , s D ( u 2 ) ) (r_{2},s_{D}(u_{2})) yield GlobalLogX \mathrm{GlobalLogX} and GlobalLogY \mathrm{GlobalLogY} , corresponding to the two quantities represented on a common scale. Finally, combining all four variables gives the exact Quantity Margin itself, which we include as the Quantity Margin feature in Tbl. 8 . This organization gives a systematic set of semantically distinct feature families, ranging from primitive input components to difference variables, shared-scale quantities, and the exact comparison rule.
To capture heuristic cues at multiple levels of granularity, we instantiate these quantities in several forms. Signed features represent the coarsest cues, such as whether a quantity or difference is positive. Threshold features represent coarse magnitude information, such as whether a log-scale variable exceeds a threshold. Continuous features use the original log-scale values directly.
After all, the resulting list of individual features is shown in Tbl. 8 . The surrogate models used in the experiments are then constructed from combinations of these feature families. The full list of surrogate feature sets is provided in Tbl. 9 . Among these, Tbl. 2 selects the representative surrogate families that are most important for interpreting the results, and these are used in the main paper.
### E.2 Additional Surrogate Results
We next examine whether the main surrogate-analysis finding holds beyond the representative metric length result in Tbl. 3 . Using the R partial 2 R^{2}_{\mathrm{partial}} defined in § 5 , Fig. 12 summarizes the representative surrogate results across LMs. Fig. 13 shows the corresponding comparison across prompt templates and unit settings.
Across these settings, the same qualitative trend appears. Surrogate models based on numerical difference and unit-scale difference, especially signed or thresholded versions of NumLogDiff \mathrm{NumLogDiff} and UnitLogDiff \mathrm{UnitLogDiff} , consistently obtain high R partial 2 R^{2}_{\mathrm{partial}} . This indicates that the LM log-probability margin is well captured by coarse cues about which side has the larger numeral and which side has the larger unit scale. By contrast, feature sets based on shared-scale quantities, GlobalLogX \mathrm{GlobalLogX} and GlobalLogY \mathrm{GlobalLogY} , tend to obtain lower R partial 2 R^{2}_{\mathrm{partial}} .
Also, we therefore provide representative surrogate tables for each unit setting: Tbl. 3 for metric length, Tbl. 12 for imperial length, Tbl. 13 for metric-imperial length, Tbl. 14 for metric mass, and Tbl. 15 for metric-imperial mass. These tables show that signed and thresholded NumLogDiff \mathrm{NumLogDiff} and UnitLogDiff \mathrm{UnitLogDiff} features explain LM behavior well across unit settings.
We also provide the full detailed surrogate results for the metric length setting. While Tbl. 3 reports only the representative surrogate families from Tbl. 2 , Tbl. 10 reports results for all surrogate models defined in Tbl. 9 . The raw metric length values are shown in Tbl. 10 . Overall, these additional results support the same conclusion as the main analysis: LM quantity-comparison behavior is better explained by numerical-difference and unit-scale-difference heuristics than by exact shared-scale quantities alone.
## Appendix F Causal Analysis: Number and Unit Differences Steer LM Decisions
In § 6 , we used Distributed Alignment Search (DAS) ( Geiger et al., 2024 ) to test whether the variables identified by the surrogate analysis are represented in LM activations and causally affect the LM’s output. The main result is that interventions on subspaces aligned with NumLogDiff \mathrm{NumLogDiff} and UnitLogDiff \mathrm{UnitLogDiff} steer the LM’s comparison decisions more strongly than interventions on the GlobalLogX / GlobalLogY \mathrm{GlobalLogX}/\mathrm{GlobalLogY} baseline or the identity baseline. This suggests that numerical difference and unit-scale difference are not merely correlated with LM behavior, but are represented in a way that can causally affect the LM’s decisions. In this section, we provide the formal details of DAS and report additional robustness results across LMs, unit settings, prompt templates, and intervention dimensionalities.
### F.1 Distributed Alignment Search
We use Distributed Alignment Search (DAS) to test whether the decomposed cues identified by the surrogate analysis are represented in the LMs and causally affect their output. DAS learns an alignment between high-level causal variables and linear subspaces of a LM’s representation by optimizing an interchange intervention objective.
Let 𝐡 ( b ) ∈ ℝ d \mathbf{h}(b)\in\mathbb{R}^{d} be the hidden representation of a base input b b , and let 𝐡 ( s j ) ∈ ℝ d \mathbf{h}(s_{j})\in\mathbb{R}^{d} be the hidden representation of a source input s j s_{j} . For a set of high-level variables Z 1 , … , Z k {Z_{1}},\ldots,Z_{k} , DAS learns an orthogonal rotation matrix 𝐑 ∈ ℝ d × d \mathbf{R}\in\mathbb{R}^{d\times d} that decomposes the rotated representation space into orthogonal subspaces: 𝒴 = 𝒴 0 ⊕ 𝒴 1 ⊕ ⋯ ⊕ 𝒴 k . \mathcal{Y}=\mathcal{Y}_{0}\oplus\mathcal{Y}_{1}\oplus\cdots\oplus\mathcal{Y}_{k}. (19) Here, 𝒴 j \mathcal{Y}_{j} is aligned with the high-level variable Z j Z_{j} , and 𝒴 0 \mathcal{Y}_{0} is the residual subspace.
A distributed interchange intervention replaces the subspace corresponding to each Z j Z_{j} in the base representation with the corresponding subspace from the source representation: 𝐡 ∗ ( b ) = 𝐑 − 1 ( CLOSE \displaystyle\mathbf{h}^{*}(b)=\mathbf{R}^{-1}\bigg( Proj 𝒴 0 ( 𝐑𝐡 ( b ) ) \displaystyle\mathrm{Proj}_{\mathcal{Y}_{0}}(\mathbf{R}\mathbf{h}(b)) + ∑ j = 1 k Proj 𝒴 j ( 𝐑𝐡 ( s j ) ) ) . \displaystyle+\sum_{j=1}^{k}\mathrm{Proj}_{\mathcal{Y}_{j}}(\mathbf{R}\mathbf{h}(s_{j}))\bigg). (20) The intervened representation 𝐡 ∗ ( b ) \mathbf{h}^{*}(b) is then passed through the remaining LM’s layers to obtain the LM’s counterfactual output.
For compactness, let e = ( b , s 1 , … , s k ) e=(b,s_{1},\ldots,s_{k}) denote an intervention example. We train 𝐑 \mathbf{R} so that the LM’s counterfactual output matches the counterfactual label predicted after intervening on the corresponding high-level variables. For each intervention example e e , we construct a high-level counterfactual by taking the base input and replacing each intermediate variable Z j Z_{j} with the value computed from the corresponding source input s j s_{j} . Let y cf int ( e ) y_{\mathrm{cf}}^{\mathrm{int}}(e) denote this high-level counterfactual label, and let 𝐩 θ int ( e ) \mathbf{p}_{\theta}^{\mathrm{int}}(e) denote the LM’s output distribution after applying the distributed interchange intervention in Eq. 20 . The DAS objective minimizes the cross-entropy loss: ℒ DAS = CE ( 𝐩 θ int ( e ) , y cf int ( e ) ) . \mathcal{L}_{\mathrm{DAS}}=\mathrm{CE}\left(\mathbf{p}_{\theta}^{\mathrm{int}}(e),y_{\mathrm{cf}}^{\mathrm{int}}(e)\right). (21)
We evaluate the learned alignment using interchange intervention accuracy (IIA), the fraction of intervention examples for which the LM’s intervened prediction matches the high-level counterfactual label: IIA = 1 | 𝒟 int | ∑ e ∈ 𝒟 int 𝟙 [ y ^ θ int ( e ) = y cf int ( e ) ] , \mathrm{IIA}=\frac{1}{|\mathcal{D}_{\mathrm{int}}|}\sum_{e\in\mathcal{D}_{\mathrm{int}}}\mathbbm{1}\left[\hat{y}_{\theta}^{\mathrm{int}}(e)=y_{\mathrm{cf}}^{\mathrm{int}}(e)\right], (22) where y ^ θ int ( e ) \hat{y}_{\theta}^{\mathrm{int}}(e) is the LM’s predicted label after intervention. A high IIA indicates that intervening on the learned subspaces changes the LM’s output consistently with interventions on the corresponding high-level variables.
### F.2 Robustness Across LMs, Unit Settings, Prompts, and training random seeds
We first examine whether the main DAS result is robust across LMs. Fig. 14 shows layer-wise IIA at the last token of u 2 u_{2} for Qwen3-4B-Base, Qwen3-8B-Base, Qwen3-14B-Base, Olmo-3-1025-7B, Olmo-3-1125-32B, and an instruction-tuned LM (Qwen3-4B) on metric length comparisons. Across LMs, the NumLogDiff / UnitLogDiff \mathrm{NumLogDiff}/\mathrm{UnitLogDiff} setting achieves higher IIA than the GlobalLogX / GlobalLogY \mathrm{GlobalLogX}/\mathrm{GlobalLogY} baseline, identity baselines, and NumLogDiff \mathrm{NumLogDiff} & Unit-Identity baseline across most layers. The absolute IIA differs across LMs, but the relative ordering is stable: interventions on numerical-difference and unit-scale-difference subspaces have the strongest effect on the LM’s counterfactual output.
We next evaluate whether this pattern holds across unit settings. Fig. 15 reports the same layer-wise analysis for Qwen3-4B-Base across all unit settings in Tbl. 1 . The NumLogDiff / UnitLogDiff \mathrm{NumLogDiff}/\mathrm{UnitLogDiff} setting remains consistently above the two baselines across metric length, imperial length, metric-imperial length, metric mass, and metric-imperial mass settings. This shows that the causal effect of numerical-difference and unit-scale-difference variables is not specific to one unit system or physical dimension.
Furthermore, we test whether the result depends on prompt wording. Fig. 16 compares the larger-comparison postposed prompt P0 and the larger-comparison preposed prompt P1. In both prompt templates, the NumLogDiff / UnitLogDiff \mathrm{NumLogDiff}/\mathrm{UnitLogDiff} setting obtains higher IIA than the GlobalLogX / GlobalLogY \mathrm{GlobalLogX}/\mathrm{GlobalLogY} and identity baselines across layers.
Finally, we examine the effect of the random seed used during DAS training. Tbl. 7 reports the intervention accuracy at the last token of u 2 u_{2} for Qwen3-4B-Base on metric length comparisons across different DAS training random seeds. We evaluate three random seeds, including seed 42 used in Fig. 5 . Although the intervention accuracy varies slightly across random seeds, the superiority of the NumLogDiff / UnitLogDiff \mathrm{NumLogDiff}/\mathrm{UnitLogDiff} setting over the baselines observed in Fig. 5 is consistently maintained.
### F.3 Robustness to Intervention Dimensionality
The main DAS experiments fix the total dimensionality of the non-residual intervention subspaces to 1024 1024 , allocated evenly across variables. To test whether the result depends on this dimensionality choice, we vary the total intervention dimensionality over 256 256 , 512 512 , 1024 1024 , and 2048 2048 in the metric length setting. For each DAS setting, the total dimensionality is evenly split among the intervened high-level variables. Thus, the two-variable settings, NumLogDiff / UnitLogDiff \mathrm{NumLogDiff}/\mathrm{UnitLogDiff} and GlobalLogX / GlobalLogY \mathrm{GlobalLogX}/\mathrm{GlobalLogY} , use half of the total dimensionality for each variable, while the four-variable identity baseline uses one quarter of the total dimensionality for each primitive component. Fig. 17 shows IIA as a function of the total intervention dimensionality. Across all dimensionalities, the NumLogDiff / UnitLogDiff \mathrm{NumLogDiff}/\mathrm{UnitLogDiff} setting consistently achieves higher IIA than the GlobalLogX / GlobalLogY \mathrm{GlobalLogX}/\mathrm{GlobalLogY} and identity baselines. Moreover, the relative ordering of the three settings remains stable as the dimensionality changes. This indicates that the main causal result is not an artifact of a particular subspace size: numerical-difference and unit-scale-difference variables more directly steer the LM’s comparison output across a range of intervention dimensionalities.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .