The Linear Centroids Hypothesis: Features as Directions Learned by Local Experts — part 2/5

Cleaned-text code points [14520, 28112). Bibliography omitted here; source.md includes it.

intermediate activations of inputs [12, 41]. This approach is limited, as analyzing an intermediate
representation in isolation ignores the downstream mapping.
For ReLU DNs, recall that the zero-level sets of the neurons construct the DN’s geometry. Thus,
the influence of an input is determined by the sign of the intermediate activations, which in turn
determines which local expert the DN employs for that input. Using this, we can demonstrate that
representing input characteristics as linear directions of intermediate activations is only anecessary
condition for ensuring they influence the behavior of the DN.
Lemma 3.1.Let X1 andX2 be two sets of inputs with distinct characteristics. A necessary condition,
but not sufficient condition, for a ReLU DN to representX1 andX2 as distinct features, is that the
corresponding intermediate activations at some layer form linear directions.
Proof.For necessity, assume the network successfully represents X1 andX2 as distinct features.
For the representations to have consistently different influences, the network must assign different
activation patterns to the inputs ofX1 andX2. This means that, at some intermediate layer, at least one
neuron activates exclusively for the latent activations of onlyX1 orX2. Therefore, the activation level
set of this neuron must separate the latent activations of the two input sets. Because the activation
level-set of a ReLU neuron is a hyperplane, the latent activations ofX1 andX2 must lie in distinct
half-spaces defined by this hyperplane. Geometrically, being strictly separable by a hyperplane
implies that the activations separate along distinct linear directions in that intermediate feature space.
For insufficiency, assume that the intermediate activations of X1 andX2 do form distinct linear
directions (i.e., are linearly separable) at some intermediate layerl. This intermediate separation does
not guarantee that the final network output will represent them as distinct features. A subsequent
layerl+k could easily map these separated activations to the exact same output representation. For
example, if the weight matrix of a subsequent layer is composed entirely of zeros, or if a sufficiently
large negative bias is applied such that the ReLU function outputs zero for all inputs from bothX1 and
X2, the previously distinct representations collapse into a single point. Therefore, while intermediate
linear separation is required, it is not sufficient to guarantee distinct feature representation at the
output.
Lemma 3.1 formalizes a flaw in the LRH, in that it assumes features are mapped along linear sub-
spaces but entirely decouples this from the learned input-output map. Specifically, an intermediate
linear separation can be easily collapsed by subsequent layers; thus, there remains an ambiguity
as to whether identified features actually drive the network’s mapping or are merely spurious arti-
facts [51]. This mapping-blindness may explain why current interpretability techniques that study the
structure of intermediate activations, such as linear classifiers [26] and sparse autoencoders [21, 54],
exhibit brittleness when applied to downstream tasks [38, 52], and are insufficient for explaining the
functional behavior of DN sub-components [47].
3.2 Using Geometry to Identify Features
To overcome the LRH’s mapping-blindness, we propose anchoring feature discovery to the network’s
actual input-output mapping via its centroids. In particular we propose theLinear Centroids
Hypothesis (LCH):
The features of a DN are represented by linear directions in its centroid spaces.
Intuitively, the LCH states that the features of a DN can be identified by observing the collection of
inputs acted on by “aligned” local experts. Because centroids are derived directly from the network’s
Jacobians, this formulation is inherently mapping-aware.
In Appendix D, we understand what the LCH means from the perspective that views the geometry
of a DN as constructed through the collection of hyperplanes defined by each neuron of the DN
(see Appendix A). Although this is only applicable to CPA DN, in this context, linearly aligned
centroids are equivalent to a partition geometry formed by hyperplane boundaries that accumulate
along boundaries. Consequently, the LCH is supported by prior works that empirically observe that
this type of structured geometry explains many properties of DNs. Humayun et al. [24] demonstrates
that the generalization and robustness properties of DNs – including transformers, residual networks,
and convolutional neural networks – emerge as the regions of the DN’s geometry accumulate along
the decision boundaries of the input space. Similarly, the size of the linear region was linked to the
5

striped
porous
striped
flecked
0 0.5 1
50
100
Correlation
Accuracy
LRH
LCH
Figure 3:The LCH holds true for pre-trained vision models on ImageNet and mitigates the
identification of spurious features.In the first and second panels, we show that the centroids
of inputs representing distinct features of the DN separate along linear directions under principal
component analysis (PCA). We study the centroids of the fourth layer of a pre-trained ResNet50
model on ImageNet. We perform PCA on the centroids of inputs from two distinct classes in the
DTD dataset. In the third panel, we train a DN on a version of FashionMNIST that has a color
feature correlated with the corresponding classification task. We train DNs on this dataset at different
correlation levels and measure the ability of a linear probe to extract the color feature using either
the intermediate activations (LRH) or the centroids (LCH) of the DN. We consider five randomly
initialized DNs at each correlation, and report the mean and standard deviation of the probe accuracy.
toxicity property of large language models [7]. Likewise, the density of linear regions in the geometry
of DN sub-components has been connected to the reasoning capabilities of large language models
[10].
We provide evidence for the LCH with the third and fourth panels of Figure 1 and the first and second
panels of Figure 3.
In the third and fourth panels of 1, we consider a DN trained to classify whether two-dimensional
input points are inside or outside the star-shaped polygon shown in the third panel of Figure 1. In this
instance, the characteristics of the input space that the DN ought to acquire as features are the interior
and exterior of the polygon. By sampling input points from the edges of the star within the DN’s
input space and observing their centroids, we can clearly see the emergence of linear directions, see
the fourth panel of Figure 1. In Appendix E, we replicate this for other polygons and DNs that are
not CPA.
Similarly, for a ResNet50 DN [17] pre-trained on ImageNet [28], we observe in the first and second
panels of Figure 3 that the centroids for inputs exhibiting distinct texture-based features (obtained
using the DTD dataset [ 8]) separate into distinct linear directions under a principal component
analysis.
In Appendix G, we further support the LCH by showing that it validates the Platonic Representation
Hypothesis by demonstrating that increasingly larger models–of different modalities–converge to the
same representations of features as characterized by linear directions of centroids.
3.3 The Linear Centroids Hypothesis Mitigates Spurious Feature Identification
As established by Lemma 3.1, the LRH suffers from an ambiguity as to whether intermediate
activations that form linear directions actually correspond to features or are merely spurious [51]. In
contrast, the LCH is mapping-aware, as it identifies structures in the centroids that are inherently tied
to the functional behavior of the DN through the Jacobian. We demonstrate that this grounding in the
input-output map makes LCH less susceptible to spurious features than the LRH.
For this, we train a convolutional neural network to classify the FashionMNIST dataset [58]. However,
at initialization, we color the images by sampling from a discrete set of 10 colors and varying the
degree of correlation between the coloring and the dataset labels. When the coloring is fully correlated,
the color feature is not spurious; when it is random, it is spurious. We evaluate the extent to which
the color feature is linearly represented by training a linear probe to predict color from the DN’s
intermediate activations or centroids and measuring its accuracy. In the third panel of Figure 3, it
is clear that the centroids of the color feature form linear directions to the extent that the feature is
6

relevant to the task, in contrast to intermediate activations, which are linear even when the feature is
spurious. This demonstrates that LCH effectively filters out spurious correlations.
4 Interpretability Under the Linear Centroids Hypothesis
We now replicate standard LRH-based interpretability studies under the LCH, on models including
DINOv2 [40], DINOv3 [48], GPT2 [42], and Llama-3.1-8B [15]. In Appendix H, we detail the dif-
ferences in computational resources required to perform these experiments under the LCH compared
to under the LRH. The only difference arises in the extraction of the activations, for which centroids
take around 10-15% longer. In the grand scheme of running these models, this addition is negligible.
4.1 Feature Extraction with Sparse Autoencoders
Here, we compare sparse autoencoders trained on the intermediate activations and centroids from
DINOv2 and DINOv3. Because centroids encode the local mapping rather than just spatial positioning
in a latent space, we hypothesize that feature dictionaries built on LCH will be more semantically
robust and less prone to spurious correlations. Similar to Hindupur et al. [19], we extract the
intermediate activations and centroids of Imagenette [20] from these models at the last multi-layer
perceptron block to train a TopK sparse autoencoder [13].
We compare the obtained DINOv2 feature dictionaries in the following ways: Train linear probes
on the feature decompositions of the train set of Imagenette to classify its classes, and then evaluate
the accuracy of the probe on the feature decompositions of the test set of Imagenette. Record the
frequency at which the features of the sparse autoencoder fire on the test set of Imagenette.
8 16 32 64
88
90
92
94
96
K
Probe Accuracy
0 2,000 4,000 6,000 8,000
0
2
4
6
Rank
Log Activation Frequency
0 0.2 0.4 0.6 0.8 1
0
5
10
Cosine Similarity
Density
LCH
LRH
Figure 4:Feature dictionaries from sparse autoencoders trained on centroids transfer better on
downstream tasks, are more active on unseen inputs, and persist more coherently across model
sizes.In the first panel, we report the accuracy of linear probes on the Imagenette test set, trained to
classify an input’s label based on which features in the feature dictionary it activates. In the second
panel, we measure the frequency with which features fire when activations from the Imagenette test
set are passed through the sparse autoencoder. In the third panel, we measure the maximum cosine
similarity between a feature from the DINOv2 dictionary and features from the DINOv3 dictionary.
With the first panel of Figure 4, we observe that the LCH feature dictionary exhibits greater general-
ization, as compared to the LRH feature dictionary. We reproduce this result for sparse autoencoders
trained on a ten-class subset of ImageNet [28] containing dog breeds in Appendix F. In particular,
we use this setting to also demonstrate that qualitatively, the features identified using LCH are more
semantically coherent.
With the second panel of Figure 4, we show that more of the LCH feature dictionary is more active
on test samples from Imagenette than the LRH feature dictionary. This highlights how, under the
LCH, identified features are less likely to be spurious.
Next, we compare the feature dictionaries obtained from DINOv2 and DINOv3. For a given activation
type (i.e., intermediate activations or centroids), we compute the maximum cosine similarity of each
DINOv2 feature with the full DINOv3 dictionary. Since DINOv3 refines the features learned by
DINOv2, the dictionary should exhibit high cosine similarities with those features. Indeed, this is
what we observe in the third panel of Figure 4 for the feature dictionaries obtained using centroids. In
contrast, for the dictionary obtained using intermediate activations, a large proportion of the features
7

have cosine similarities of around 0.4. This bimodal distribution suggests that the features identified
by LRH do not correlate across the models and are instead spurious artifacts. In contrast, because
centroids capture the functional mapping rather than arbitrary intermediate geometries, they converge
reliably across model scales, strongly validating the Platonic Representation Hypothesis [22].
4.2 Circuit Discovery using Attribution Metrics
Because centroids explicitly summarize the local experts driving the input-output map, their direct
relationship with the computational graph offers a powerful mechanism for circuit discovery. Rather
than relying on activation magnitudes, we can directly query the mapping to introduce a novel
attribution metric using centroids [14, 35, 56], and demonstrate how it can be used to perform circuit
discovery on GPT2-Large [42].
Formally, letf be a DN andf (i,ℓ) be the same DN but with theith neuron of theℓth layer manipulated.
The attribution of neuronito the features of a collection of samplesNis quantified as
s(i,ℓ)
N := 1
|N|
X
x∈N



µf
x−µf (i,ℓ)
x



2


µf
x