The Linear Centroids Hypothesis: Features as Directions Learned by Local Experts — part 3/5

Cleaned-text code points [28112, 43409). Bibliography omitted here; source.md includes it.

2
,(1)
whereµf
x andµf (i,ℓ)
x are the centroids off andf (i,ℓ) atx respectively. Quantifying the attribution
of a neuron to the local features of a sample point x can be done by taking N to beBϵ(x) =
x′∈R d :∥x−x ′∥2 <ϵ
	
.5
By patching each neuron in a layer of GPT2-Large, Clement & Joseph[9] observed that the thirty-first
layer multi-layer perceptron contains a neuron which is responsible for predicting the “an” token.
We demonstrate that similar analyses can be conducted more simply by applying Equation (1) to
a neighborhood of the last token embeddings at the input of the thirty-first layer on the prompt “I
climbed up the pear tree and picked a pear. I climbed up the apple tree and picked.” In Figure 5, the
distribution of neuron attribution values is heavily skewed, with the neuron identified by Clement &
Joseph [9], marked in black, sitting within the top 99.8th percentile of values. This demonstrates that,
rather than performing extensive ablation studies on all the neurons of GPT2-Large, querying the
network’s local mapping via equation 1 effectively isolates functional circuits and filters out irrelevant
neurons. In Appendix I, we use this example to demonstrate that equation 1 is robust as an attribution
metric.
4.3 Concept Discovery using Linear Probes
Probing is another technique that exploits linear structures in DNs to either extract features [ 26],
extract representations [37], or classify inputs [34]. Here, we explore the latter of these applications
by forming linear classifiers to discern the truthfulness of input statements to large language models.
We adopt the mass-mean probes of Marks & Tegmark [34], along with their datasets, to test the
generalization capacity of these probes. Using the likely dataset, we obtain mass-mean probes from
the twelfth layer of the Llama-3.1-8B large language model [15], either using the latent activations or
the centroids extracted from the multi-layer perceptron component. Thelikely dataset is constructed
as a classification problem of whether sample tokens are likely or unlikely under the model’s logit
distribution for non-factual textual inputs. The other datasets are formulated as classification problems
between factually truthful and untruthful statements. Therefore,likelyis identifying plausibility in
the model’s outputs rather than a concept of truthfulness. Because centroid-based mass-mean probes
generalize more effectively to these truth-identifying datasets (see Figure 6), it confirms that LCH
captures theaction(the mapping) of outputting a truthful statement, rather than just the isolated
concept. This underscores the intuition that LCH offers a mechanistic perspective by evaluating when
the local experts of a DN align.
5 Discussion
The Linear Representation Hypothesis posits that feature discovery involves finding linear subspaces
in the activation spaces of DN, but this critically decouples those features from the network’s actual
5Henceforth, we will uses (i,ℓ) to denotes (i,ℓ)
Bϵ(x) unless stated otherwise.
8

Figure 5:Neuron-attribution metrics derived
using LCH can quickly filter out irrelevant
neurons, facilitating circuit discovery.We
prompt GPT2-Large and note the normalized
attribution value (using Equation (1)) in a neigh-
borhood of the embedding at the input of the
multi-layer perceptron block at the thirty-first
layer of the last token of this prompt. The neigh-
borhood is constructed by sampling 256 points
within a radius of 0.25 of the embedding. We
normalize these values to the range [0,1] . In
black we indicate the 892nd neuron in the multi-
layer perceptron.
Figure 6:Linear probes obtained using LCH
exhibit greater generalization than those ob-
tained using LRH.We obtain mass-mean
probes on the likely dataset from the twelfth
layer of Llama-3.1-8B, either using intermedi-
ate activations or centroids extracted from the
multi-layer perceptron component. We then test
these probes using other datasets from Marks &
Tegmark [34].
input-output map. To make feature discovery mapping-aware, we introduced the Linear Centroids
Hypothesis (LCH), which posits that feature discovery amounts to finding linear directions incentroid
space. Unlike intermediate representations, DN centroids are mapping-aware, hierarchically defined,
intuitive artifacts of DNs that describe the DN’s local experts.
Because centroids are efficiently accessible via Jacobian vector products, they serve as a drop-in
replacement for activations in standard interpretability pipelines. Consequently, we can demonstrate
that LCH is operational and inherently resists the identification of spurious features.
Ultimately, LCH unifies feature dictionaries, probing, circuits, and saliency maps into a single
geometric framework, ensuring that interpretability is mechanistically grounded in how the network
actually computes.
Limitations and Future Work.The power diagram subdivision of a DN is parameterized by both
centroids and radii. This study focuses entirely on the centroids, leaving the radii—which determine
whether a centroid is contained within its own region—unexplored. Future research should investigate
these radius parameters, as integrating them could provide an even more comprehensive geometric
interpretation of the network’s local experts and their decision boundaries.
Additionally, while this work introduced thelocal centroidas a faithful, gradient-based saliency map
to highlight relevant features for single inputs, future efforts should extend this application. Testing
local centroids across a wider variety of DN architectures and broader input distributions will further
validate their explanatory power within this mapping-aware framework.
Acknowledgments
This work was supported by ONR grant N00014-23-1-2714, ONR MURI N00014-20-1-2787, DOE
grant DE-SC0020345, and DOI grant 140D0423C0076.
References
[1] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I. et al. Sanity Checks for Saliency Maps.
Advances in Neural Information Processing Systems, 31, 2018.
[2] Amodei, D., Olah, C., Steinhardt, J., Christiano, P. et al. Concrete Problems in AI Safety.
arXiv:1606.06565, 2016.
9

[3] Balagansky, N., Maksimov, I., and Gavrilov, D. Mechanistic Permutability: Match Features
across Layers. InThe Thirteenth International Conference on Learning Representations, 2025.
[4] Balestriero, R. and Baraniuk, R. Mad Max: Affine Spline Insights into Deep Learning.
arXiv:1805.06576, 2018.
[5] Balestriero, R. and Baraniuk, R. G. From Hard to Soft: Understanding Deep Network Non-
linearities via Vector Quantization and Statistical Inference. InInternational Conference on
Learning Representations, 2018.
[6] Balestriero, R., Cosentino, R., Aazhang, B., and Baraniuk, R. The Geometry of Deep Networks:
Power Diagram Subdivision. InNeural Information Processing Systems, 2019.
[7] Balestriero, R., Cosentino, R., and Shekkizhar, S. Characterizing Large Language Model
Geometry Helps Solve Toxicity Detection and Generation. InInternational Conference on
Machine Learning, 2023.
[8] Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S. et al. Describing Textures in the Wild. In
IEEE Conference on Computer Vision and Pattern Recognition, 2014.
[9] Clement, N. and Joseph, M. We Found An Neuron in GPT-2, February 2023. URLhttps://
www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-an-neuron-in-gpt-2.
[10] Cosentino, R. and Shekkizhar, S. Reasoning in Large Language Models: A Geometric Perspec-
tive.arXiv:2407.02678, 2024.
[11] Croce, F., Andriushchenko, M., Sehwag, V ., Debenedetti, E. et al. RobustBench: A Standardized
Adversarial Robustness Benchmark.arXiv:2010.09670, 2020.
[12] Elhage, N., Hume, T., Olsson, C., Schiefer, N. et al. Toy Models of Superposition.Transformer
Circuits Thread, 2022.
[13] Gao, L., la Tour, T. D., Tillman, H., Goh, G. et al. Scaling and Evaluating Sparse Autoencoders.
InThe Thirteenth International Conference on Learning Representations, 2025.
[14] Goldowsky-Dill, N., MacLeod, C., Sato, L., and Arora, A. Localizing model behavior with path
patching.arXiv:2304.05969, 2023.
[15] Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A. et al. The Llama 3 Herd of Models.
arXiv:2407.21783, 2024.
[16] Hanin, B. and Rolnick, D. Complexity of Linear Regions in Deep Networks. InProceedings of
the Thirty-sixth International Conference on Machine Learning. PMLR, 2019.
[17] He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learning for Image Recognition. In
IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[18] Hendrycks, D. and Gimpel, K. Gaussian Error Linear Units (GELUs).arXiv:1606.08415, 2023.
[19] Hindupur, S. S. R., Lubana, E. S., Fel, T., and Ba, D. E. Projecting Assumptions: The Duality
between Sparse Autoencoders and Concept Geometry. InICML Workshop on Methods and
Opportunities at Small Scale, 2025.
[20] Howard, J. and Gugger, S. Fastai: A Layered API for Deep Learning.Information, 11(2), 2020.
[21] Huben, R., Cunningham, H., Smith, L. R., Ewart, A. et al. Sparse Autoencoders Find Highly
Interpretable Features in Language Models. InThe Twelfth International Conference on
Learning Representations, 2024.
[22] Huh, M., Cheung, B., Wang, T., and Isola, P. Position: The Platonic Representation Hypothesis.
InForty-First International Conference on Machine Learning, 2024.
[23] Humayun, A. I., Balestriero, R., Balakrishnan, G., and Baraniuk, R. SplineCam: Exact
Visualization and Characterization of Deep Network Geometry and Decision Boundaries. In
IEEE Conference on Computer Vision and Pattern Recognition, 2023.
10

[24] Humayun, A. I., Balestriero, R., and Baraniuk, R. Deep Networks Always Grok and Here
Is Why. InHigh-Dimensional Learning Dynamics 2024: The Emergence of Structure and
Reasoning, 2024.
[25] Imai, H., Iri, M., and Murota, K. V oronoi Diagram in the Laguerre Geometry and Its Applica-
tions.SIAM Journal on Computing, 14(1), 1985.
[26] Kim, B., Wattenberg, M., Gilmer, J., Cai, C. et al. Interpretability beyond Feature Attribution:
Quantitative Testing with Concept Activation Vectors (TCA V). InProceedings of the Thirty-fifth
International Conference on Machine Learning, 2018.
[27] Kingma, D. P. and Ba, J. Adam: A Method for Stochastic Optimization.arXiv:1412.6980,
2017.
[28] Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet Classification with Deep Convolu-
tional Neural Networks. InAdvances in Neural Information Processing Systems, 2012.
[29] Liu, C., Dong, Y ., Xiang, W., Yang, X. et al. A Comprehensive Study on Robustness of Image
Classification Models: Benchmarking and Rethinking.International Journal of Computer
Vision, 133(2), 2024.
[30] Liu, Z., Lin, Y ., Cao, Y ., Hu, H. et al. Swin Transformer: Hierarchical Vision Transformer
Using Shifted Windows. InInternational Conference on Computer Vision, 2021.
[31] Liu, Z., Mao, H., Wu, C.-Y ., Feichtenhofer, C. et al. A ConvNet for the 2020s. InIEEE
Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, 2022.
[32] Lyche, T. and Schumaker, L. L. Local Spline Approximation Methods.Journal of Approximation
Theory, 15(4), 1975.
[33] maintainers, T. and contributors. TorchVision: PyTorch’s Computer Vision Library, 2016. URL
https://github.com/pytorch/vision.
[34] Marks, S. and Tegmark, M. The Geometry of Truth: Emergent Linear Structure in Large
Language Model Representations of True/False Datasets. InFirstConference on Language
Modeling, 2024.
[35] Meng, K., Bau, D., Andonian, A. J., and Belinkov, Y . Locating and Editing Factual Associations
in GPT. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.),Advances in Neural
Information Processing Systems, 2022.
[36] Mont´ufar, G., Pascanu, R., Cho, K., and Bengio, Y . On the Number of Linear Regions of Deep
Neural Networks. InNeural Information Processing Systems, 2014.
[37] Nanda, N., Lee, A., and Wattenberg, M. Emergent Linear Representations in World Models of
Self-Supervised Sequence Models. InProceedings of the sixth BlackboxNLP Workshop: Ana-
lyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics,
2023.
[38] Nicolson, A., Schut, L., Noble, A., and Gal, Y . Explaining Explainability: Recommendations
for Effective Use of Concept Activation Vectors.Transactions on Machine Learning Research,
2025.
[39] Olah, C., Cammarata, N., Schubert, L., Goh, G. et al. Zoom in: An Introduction to Circuits.
Distill, 2020.
[40] Oquab, M., Darcet, T., Moutakanni, T., V o, H. V . et al. DINOv2: Learning Robust Visual
Features without Supervision.Transactions on Machine Learning Research, 2024.
[41] Park, K., Choe, Y . J., and Veitch, V . The Linear Representation Hypothesis and the Geometry of
Large Language Models. InForty-first International Conference on Machine Learning, 2024.
[42] Radford, A., Wu, J., Child, R., Luan, D. et al. Language Models Are Unsupervised Multitask
Learners.OpenAI, 2019.
11

[43] Ramachandran, P., Zoph, B., and Le, Q. V . Searching for Activation Functions.
arXiv:1710.05941, 2017.
[44] Rodr´ıguez-Mu˜noz, A., Wang, T., and Torralba, A. Characterizing Model Robustness via Natural
Input Gradients. InProceedings of the European Conference on Computer Vision, 2024.
[45] Schumaker, L.Spline Functions: Basic Theory. Cambridge Mathematical Library. Cambridge
University Press, 3 edition, 2007.
[46] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R. et al. Grad-CAM: Visual Explanations
From Deep Networks via Gradient-based Localization. InIEEE International Conference on
Computer Vision, 2017.
[47] Sharkey, L., Chughtai, B., Batson, J., Lindsey, J. et al. Open Problems in Mechanistic Inter-
pretability.arXiv:2501.16496, 2025.
[48] Sim ´eoni, O., V o, H. V ., Seitzer, M., Baldassarre, F. et al. DINOv3.arXiv:2508.10104, 2025.
[49] Simonyan, K., Vedaldi, A., and Zisserman, A. Deep Inside Convolutional Networks: Visualising
Image Classification Models and Saliency Maps. InWorkshop at International Conference on
Learning Representations, 2014.
[50] Smilkov, D., Thorat, N., Kim, B., Vi´egas, F. et al. SmoothGrad: Removing Noise by Adding
Noise.arXiv:1706.03825, 2017.
[51] Smith, L. The ‘Strong’ Feature Hypothesis Could Be Wrong, August 2024. AI Alignment
Forum.
[52] Smith, L., Rajamanoharan, S., Conmy, A., McDougall, C. et al. Negative Results for Sparse
Autoencoders On Downstream Tasks and Deprioritising SAE Research, March 2025. Medium.
[53] Sundararajan, M., Taly, A., and Yan, Q. Axiomatic Attribution for Deep Networks. In
International Conference on Machine Learning. PMLR, 2017.
[54] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen et al. Towards Monosemanticity:
Decomposing Language Models With Dictionary Learning.Transformer Circuits Thread, 2023.
[55] van der Maaten, L. and Hinton, G. Visualizing Data Using T-SNE.Journal of Machine Learning
Research, 9(86), 2008.
[56] Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B. et al. Interpretability in the Wild:
A Circuit for Indirect Object Identification in GPT-2 Small. InThe Eleventh International
Conference on Learning Representations, 2023.
[57] Workshop, B., :, Scao, T. L., Fan, A. et al. BLOOM: A 176B-Parameter Open-access Multilin-
gual Language Model.arXiv:2211.05100, 2023.
[58] Xiao, H., Rasul, K., and V ollgraf, R. Fashion-MNIST: A Novel Image Dataset for Benchmarking
Machine Learning Algorithms.arXiv:1708.07747, 2017.
12