The Linear Centroids Hypothesis: Features as Directions Learned by Local Experts — part 5/5

Cleaned-text code points [57166, 66916). Bibliography omitted here; source.md includes it.

DN Geometry
 Centroids
 s(i,2)
 s(i,3)
Figure 11: Here we train a DN in the same manner as the one considered in Figure 9, except we
use the GELU nonlinearity. In the first panel, we replace the nonlinearities with ReLU such that we
can use SplineCam to visualize its geometry. In the second panel, we visualize the centroids from
input samples. In the third and fourth panels, we consider the sensitivities of centroids when pruning
neurons from the second and third layers, respectively.
In the first panel of Figure 12, we can see that even in this more challenging setting, the features
learned under LCH generalize better, just like in Figure 4.
Moreover, with the right panel of Figure 12, we can see qualitatively that the features identified
using LCH are more semantically coherent. Here, we identify similar samples from the test set by
computing the Jaccard similarity of their feature activation patterns. We create a grid of images, with
the first image as a fixed sample and the remaining six as its closest neighbors by Jaccard similarity.
In both cases, the features identify the same dog breed; however, only under LCH do we consistently
see a dog with the same head position.
8 16 32 64
55
60
65
70
75
K
Probe Accuracy
LCH
LRH
LRH
LCH
Figure 12: Here we train sparse autoencoders in the same ways as Figure 4, but on a subset of
ImageNet [28] containing ten random dog breed classes, which we split into a train and test set. In
the first panel, we evaluate the sparse autoencoders by training linear probes on the feature activation
of the training set and measuring their accuracy on the test set. In the right panel, we sample from
the test set and compute the Jaccard similarity between its feature activation pattern and those of the
other samples in the test set. We visualize the sample in the left image in the grid, then identify the
six most similar samples using the other images in the grid.
G Validating the Platonic Representation Hypothesis
The Platonic Representation Hypothesis (PRH) states that as DNs become larger and more capable,
the features they form converge [22]. With Figure 13, we demonstrate that analyzing centroids in a
similar manner also supports the PRH. In Figure 13, we adopt the methodology of Huh et al. [22]
and compare the cosine similarities of activations (i.e., latent activations or centroids) computed
across layers of BLOOM language models [57] of increasing capacity to those computed across the
DINOv2 vision transformer [40]. As expected under the PRH, the cosine similarity of activations and
model capacity is positively related.
A limitation of this analysis is that it considers only the maximum cosine similarities among all
possible activations of one DN and all possible activations of another. It does not account for how the
full spectrum of features in one model aligns with those of another. This is rectified by our analysis
17

109
Number of Parameters
0.105
0.110
0.115
0.120
0.125Latent Alignment
bloomz-560mbloomz-1b1 bloomz-3b bloomz-7b1
Latents
Centroids
0.044
0.046
0.048
0.050
Centroid Alignment
Figure 13: The Platonic Representation Hypothesis is validated by observing the cosine similarities
between the centroids of DNs of different capacities. We compare the alignment of centroids from
the BLOOM language model to the centroids of the DINOv2 vision model.
shown in the third panel of Figure 4, which demonstrates that the LCH more faithfully captures this
hypothesis.
H Computational Requirements.
A valid concern with exploring the LCH is the computational burden it imposes on interpretability, as
it requires evaluating the DN’s Jacobians. Fortunately, this interrogation only requires considering
Jacobian vector products (see Proposition 2.1), which are significantly cheaper to compute in common
computational frameworks. Furthermore, the analysis of centroids often focuses on a relatively small
component of the DN. Thus, this computation would appear relatively insignificant compared to
processing the entire DN. In this section, we empirically quantify the computational burden of using
centroids rather than latents in our main experiments in Section 4.
FashionMNIST Color Correlation.This experiment took approximately 2 hours on a Quadro
RTX 8000. For each correlation value (six in total), five separate DNs were trained. Two linear
probes were then trained to classify the color feature, one using intermediate activations and the other
using centroids.
The DN comprised a three-layer convolutional feature extractor followed by a two-layer multi-layer
perceptron classifier. Intermediate activations were extracted from the input to the classifier, and
centroids were computed using the input-output Jacobian of the classifier.
The DN was trained for 10 epochs using the Adam [27] optimizer at a learning rate of 0.001. The
linear probe was trained for 3 epochs, again with the Adam optimizer at a learning rate of0.001.
DINO Feature Extraction.For this experiment, we find almost no difference in the time necessary
to extract centroids. It only takes 8.7 seconds compared to extracting intermediate representations,
which takes 7.8 seconds. After these vectors are extracted, the computational pipeline is identical
when using centroids or intermediate activations.
In the first panel of Figure 4, we train TopK sparse autoencoders [ 13] on all token positions of
DINOv2 and sparsity values 8, 16, 32, and 64. Training each sparse autoencoder for 40 epochs takes
around 1 hour on an NVIDIA TITANX. In the second panel of Figure 4, we consider the sparse
autoencoders with sparsity value32.
In the third of Figure 4, we consider similar sparse autoencoders but trained on only theclstoken
of DINOv2/v3. Training each of these sparse autoencoders takes about 2 minutes on an NVIDIA
TITANX.
18

GPT2 Circuit Discovery.Although there is no direct analog of this experiment with latent activa-
tions, we can still argue that the computational burden is relatively benign. In particular, since we
only compute centroids across the multi-layer perceptron block of the thirty-first layer, we only need
to consider the Jacobian vector product for this component. This can be done by storing the gradients
from a forward pass across this block, which, relative to performing a forward pass across the model,
is insignificant. This experiment takes less than 1 hour on an NVIDIA TITANX.
Llama-3.1-8B Probes.To implement this experiment, we use the setup of https://github.
com/saprmarks/geometry-of-truth. The only difference involves the extraction of centroids
in addition to the extraction of intermediate activations. Extracting centroids took 2329 seconds
compared to470seconds for intermediate activations.
I Robustness of Neuron Attribution with Centroids
In order for Equation (1) to prove useful as a tool for interpreting DNs, it is essential that it is robust
in its application. For example, Equation (1) ought not be sensitive to the neighborhoodN chosen.
Furthermore, since in practice we can only approximate Equation (1) by taking a finite sample of
points fromN, it is important that Equation (1) has low variance in relation to this finite sample. In
Figure 17, we test both these properties for the experiment of Figure 5. In the left plot, we observe that
attribution values have a low variance when a finite sample is used to approximate the neighborhood
N . In the right plot, we observe that the percentile of a particular neuron within the DN layer is
stable across different neighborhood sizes. This ensures that conclusions derived from Equation (1)
are robust.
J Model Licenses
Swin-B [30], ResNet [17] and ConvNeXt-L [31] are obtained through PyTorch [ 33] and Robust-
Bench [11] under an MIT license.
DINOv2 [40] is used under an Apache 2.0 license, DINOv3 [ 48] under the DINOv3-license, the
Bloom language models [57] under the BigScience RAIL license, Llama-3.1-8B under the Llama3.1
license, GPT-2 [42] is obtained through the nano-GPT6 repository under an MIT license.
6https://github.com/karpathy/nanogpt
19

Input
Sample
Local Centroid
(Input Space to Hidden Layer)
Local Centroid
(Input Space to Output)
Local Centroid
(At Random Initialization)
Figure 14: The local centroids of the pre-trained ConvNext-L DN from PyTorch.
20

Input
 Local Centroid (Input Space to Hidden Layer)
 Local Centroid (Input Space to Output)
Figure 15: The local centroids of the adversarially Swin-B transformer from Rodr´ıguez-Mu˜noz et al.
[44].
21

Input Distribution
 Functional Geometry
Input Samples
 Centroids
 s(i,3)
Input Distribution
 Functional Geometry
Input Samples
 Centroids
 s(i,2)
Figure 16: Here we perform some of the same analyses as conducted previously, but with a DN
trained on a bowtie-shaped polygon,toptwo rows, and a reuleaux-shaped polygon,bottomtwo rows.
22

Figure 17: The neuron attribution metric of Equation (1) is a robust measure for interpreting the
neurons of a DN. Here we consider the robustness of Equation (1) for the experiment in Figure 5.
More specifically, we test how the neurons’ attribution values change as we consider increasingly large
neighborhoods. The neighborhoods we consider are of the formBϵ(x), wherex is the embedding of
the last token of a prompt at the 31st of GPT2-Large. We considerϵ normalized by the norm of the
centroid ofx at this layer of the DN. To compute Equation (1) at each neuron of the layer, we sample
256 embeddings from this neighborhood. In the left plot, we observe how the average attribution
value of each neuron changes across random samplings of this neighborhood. In the right plot, we
observe how the percentile value of the 892nd changes across these random samplings. The error bars
represent one standard deviation in the observed values.
23