In-Context Algebra — part 5/5

Cleaned-text code points [63435, 76891). Bibliography omitted here; source.md includes it.

Steering Away from the Identity Cluster. Injecting a false identity fact into non-identity sequences confuses the model, causing it to wrongly predict one of the two query symbols. We find that steering away from the identity cluster (subtracting 𝐯 1 \mathbf{v}_{1} instead of adding) can erase this confusion, causing the model to predict the correct non-identity answer 97.0 % 97.0\% of the time.

## Appendix F Closure and Elimination Subspaces

In this section, we provide additional details about results related to the closure and elimination subspaces described in Section 5.4 .

Sequence: ’hp=e,il=i,li=i,nc=c,bi=l,ne=e,fe=h,pp=f,ba=i,pc=h,eh=n,la=a, pp=f,pf=n,fe=h,fh=c,il=i,ef=h,hh=f,cp=h,pc=h,hh=f,cn=c,bi=l, (hc=/bi=) ’

At the left-slot of a given fact (e.g., h ​ c = hc{=} ), the “goal” of the model is to predict any variable that could be associated with the left-slot variable (e.g., h h ). This requires identifying all variables previously connected to h h in the context. This set of variables is precisely what we call the “ closure ” of the group. We find the model is very good at this task, producing near-uniform logits over all previously seen elements of the query’s group, an example of which is shown in Figure 15 .

We quantify the model’s ability to compute the closure by measuring the top- K K matching accuracy at the left-slot position. We identify the K K variables with the highest predicted logits. Top- K K matching accuracy is then computed as the proportion of these top- K K predictions that correspond to variables from the corresponding group G G that have appeared in the context so far. Perfect performance means the model assigns the K K highest logits exactly to the K K group members seen in context, regardless of their relative ordering. We also report top- 1 1 accuracy, which is whether the highest logit is one of the variables in G G . Over a batch of 2000 2000 randomly sampled algebra sequences with 200 facts, we find that our model gets 100.0 % 100.0\% top- 1 1 accuracy, and 100.0 % 100.0\% top- K K matching accuracy, indicating it has correctly learned how to compute within-group closure.

### F.1 How are Closure and Cancellation Sets Computed?

In this section, we investigate how the set difference operation introduced in Section 5.4 is implemented by the transformer model. For a given query x ​ y = xy{=} , recall that the closure set S closure S_{\text{closure}} contains all elements that have previously appeared in the context associated with x x or y y , and the cancellation set S cancel S_{\text{cancel}} is the set of answers from previously seen facts that share either x x in its left-slot or y y in its right-slot. Upon examining attention patterns and attention head outputs via the logit lens ( Nostalgebraist, 2020 ) , we find evidence that these two sets are built up from contributions across several attention heads. We find that the closure computation is reused across token positions, and also describe a few heads implicated in constructing the cancellation set in more detail below.

Closure Computation. We find that the same circuitry used to compute group closure is reused across token positions: both at the left-slot prediction, and at the predictive token (‘ = {=} ’). While we measure the model’s ability to compute closure using facts’ left-slot predictions, we also find that group closure dominates predictions at the predictive token during early phases of training – producing near-equal logits of group symbols until other strategies are learned (Figure 6 b, orange). Without additional structural information (e.g., that variables represent group elements), randomly predicting a symbol from the closure is a near-optimal strategy. But as data-specific structure is learned, the closure prediction is built upon by other mechanisms such as the elimination subspace.

##### Cancellation Set Construction.

A few attention heads at the final predictive token exhibit attention patterns that are suggestive of partial cancellation law behavior. For example, head 3.2 primarily attends to answer-slots of facts that share the same symbol in its left-slot as the query (i.e., facts of the form x ​ ? = ? x\texttt{?}=\texttt{?} , where ? can be any variable token, see Figure 16 a). We find that head 3.2 places an average of 74.4 % 74.4\% of its attention probability mass on answer-slots of facts that share the same left-slot symbol as the query (averaged over 200 prompts). After attending to these tokens, head 3.2’s attention contribution subsequently demotes the logits of each answer token (Figure 16 b). We find another attention head (layer 3, head index 4) that primarily attends to answer-slots of facts that share the same symbol in its right-slot as the query (i.e., facts of the form ? ​ y = ? \texttt{?}y=\texttt{?} , see Figure 16 c), doing so 57.1 % 57.1\% of the time. Similarly, head 3.4 demotes the logits of the answer-slot tokens it attends to (Figure 16 d). These examples show how multiple attention heads help identify a set of tokens that should be eliminated as answers, and we find that learning a low-dimensional subspace over the attention layer can cleanly capture the corresponding cancellation subspace.

### F.2 Subspace Construction

Here we describe how we construct a learnable subspace that can characterize multi-dimensional high-level variables such as the closure and cancellation sets described in Section 5.4 .

We learn a set of Householder unit-vectors { v i ∈ ℝ d , ‖ v i ‖ = 1 } \left\{v_{i}\in\mathbb{R}^{d},||v_{i}||{=}1\right\} (where d d is the model’s hidden dimension), to construct a series of Householder matrices, H i = I − 2 ​ v i ​ v i T H_{i}=I-2v_{i}v_{i}^{T} , that are composed to form an orthogonal matrix Q = H k H k − 1 ⋯ H 1 ∈ ℝ d × d Q=H_{k}H_{k-1}\cdots H_{1}\in\mathbb{R}^{d\times d} , ( Householder, 1958 ) . The first k k columns of Q Q , denoted Q k ∈ ℝ d × k Q_{k}\in\mathbb{R}^{d\times k} , form an orthonormal basis for our intervention subspace. We construct our subspace projection as W = Q k ​ Q k T W=Q_{k}Q_{k}^{T} and perform subspace-level interventions by mixing information between activations of the model on sequences s s and s ′ s^{\prime} as shown in Equation 10 : W ​ h s + ( I − W ) ​ h s ′ → h s ′ Wh_{s}+(I-W)h_{s^{\prime}}\rightarrow h_{s^{\prime}} (10) where h s h_{s} represents an activation taken from the model under sequence s s , h s ′ h_{s^{\prime}} represents an activation taken from the same location under sequence s ′ s^{\prime} , and → \rightarrow means the activation h s ′ h_{s^{\prime}} is replaced with the intervened representation W ​ h s + ( I − W ) ​ h s ′ Wh_{s}+(I-W)h_{s^{\prime}} . While we use h s h_{s} to denote “activation” here, it could be a representation taken from any location in the model.

#### F.2.1 Subspace Training Details

We train closure and elimination subspaces over different model components (attention layers, MLP layers, full layers), and sweep over different values for the subspace dimension k ∈ { 2 , 4 , 6 , 8 , 10 , 12 , 16 , 20 , 24 , 32 } k\in\{2,4,6,8,10,12,16,20,24,32\} . We measure the interchange intervention accuracy ( Wu et al., 2023 ; Prakash et al., 2025 ) for each trained subspace and report the results in Figure 17 .

We find that the closure set computation is best represented in the final attention layer and that a 16-dimensional subspace is sufficient to achieve near-perfect intervention accuracy ( 99.9 % 99.9\% ), and that it converges pretty quickly during training (Figure 18 a). Subspaces trained at most locations are unable to meaningfully influence the construction of the closure set. It can be partially controlled ( ∼ 70 % {\sim}70\% ) by a subspace trained on the input to the final attention layer (i.e., the layer 2 output is the same location as the layer 3 attention’s input), but requires more dimensions and doesn’t achieve as clean of performance as the final attention layer (Figure 17 ).

For the elimination subspace, we find that training on counterfactual sequence pairs similar to those used for the closure subspace ( d = 16 d=16 ) results in a subspace that plateaus around ∼ 75 % {\sim}75\% intervention accuracy, even after extended training (Figure 18 b, red). Analyzing the subspace intervention failure cases reveals that 99.9 % 99.9\% of these sequence pairs fit into one of two categories: either (i) at least one sequence ends in an identity fact, or (ii) the counterfactual target renders the query an identity fact under intervention. Because of this, we choose to exclude sequences that end in identity facts (and counterfactually-imposed identity facts) from the training distribution of the elimination subspace. We find that without identity sequences, the elimination subspace quickly achieves near-perfect intervention accuracy (Figure 18 b, blue).

The inability of the elimination subspace to capture the model’s predictions on identity sequences suggests that the mechanism the model uses to solve identity sequences is distinct from the closure-based cancellation mechanism under study here. The identity mechanism is explored in more detail in Section 5.3 . The fact that identity facts are easily separated from other facts under PCA might explain some of the reason why training the elimination subspace fails to converge on identity sequences, though this distinction doesn’t matter for training the closure subspace (Figure 18 a, red).

We also perform a sweep over layers and dimensions for the elimination subspace and find that the cancellation set is best represented in the final attention layer with a subspace of ∼ 16 {\sim}16 dimensions (Figure 19 ). While it is the same size as the closure subspace, we find these subspaces are distinct from each other, as they are unable to produce valid counterfactuals in the opposite setting.

#### F.2.2 Probing the Closure and Elimination Subspaces

We train probes to better characterize how the trained closure and elimination subspaces represent the closure set and cancellation set. We train individual probes to detect whether a variable is in the group closure or cancellation set using both the closure and elimination subspaces (Figure 20 ). We find the probes can accurately predict with high accuracy whether variables are in the closure set ( 91 − 96 % 91{-}96\% ) (Figure 20 , left), and with good, but slightly worse accuracy for predicting whether a variable is in the cancellation set ( 77 − 87 % 77{-}87\% ) (Figure 20 , right).

We compare the directions of the learned probes with the unembedding directions of the model to check how aligned they are with the output space. We find that the closure probes show a weak positive alignment (average cosine similarity of + 0.38 +0.38 ) with the unembedding vectors with the corresponding variable (Figure 21 , left). This matches our intuition that the closure subspace does in fact promote (increase the probability of) the variables that are present in the group closure.

In contrast, the elimination probes show a weak negative alignment (average cosine similarity of − 0.17 -0.17 ) with their corresponding unembedding vectors and near-zero similarity with off-diagonal entries (Figure 21 , right). This suggests the elimination subspace “demotes”, or lowers the probability of variables that are present in the cancellation set, again matching our intuition from Section 5.4 .

We also compute the cosine similarity between the trained probes for both subspaces and find that for each subspace the probe for one symbol is independent of those for other symbols (Figure 22 ).

## Appendix G Data Coverage Pseudocode

In this section, we provide some pseudo-code examples showing how we check the coverage of each algorithm hypothesized in Section 4 . Optimized code implementations can be found through our project website: algebra.baulab.info .

## Appendix H Use of Large Language Models

As per the ICLR 2026 author guidelines, we provide details about our use of large language models (LLMs) in the preparation of this manuscript. LLMs were primarily used as a general-purpose tool to aid and polish writing, both at the sentence level (e.g., grammar or re-wording sentences), and at the paragraph level (e.g., re-organizing sentences in a paragraph). When considering LLM suggestions, the resulting text went through many subsequent editing rounds. LLM use did not contribute in any way that we would consider equal to the level of an author.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .