The Linear Centroids Hypothesis: Features as Directions Learned by Local Experts — part 1/5

Cleaned-text code points [0, 14520). Bibliography omitted here; source.md includes it.

The Linear Centroids Hypothesis:
Features as Directions Learned by Local Experts
Thomas Walker∗
Rice University
Ahmed Imtiaz Humayun
Google Research
Randall Balestriero
Brown University
Richard Baraniuk
Rice University
Abstract
The Linear Representation Hypothesis (LRH) identifies features of a trained deep
network (DN) as linear directions in the activation spaces, i.e., output spaces of
intermediate layers. This characterization decouples the input-output maps learned
by a DN from the organization of feature directions in its activation spaces. We in-
troduce theLinear Centroids Hypothesis (LCH), which instead identifies features
with linear directions among a DN’scentroid spaces– where any vector denotes a
centroid or summary of alocal affine expertcharacterizing the learned input-output
maps of the DN exactly (e.g., for piecewise-affine DNs) or approximately (e.g., for
smooth DNs like transformers). We show that replacing intermediate activations
with centroids yields a functional drop-in alternative for standard interpretability
tools. Empirically, this change yields sparser, more downstream-useful feature
dictionaries on DINO ViTs, suppresses spurious directions on a controlled task,
recovers interpretable circuits in GPT2-Large, and produces faithful gradient-based
saliency maps. LCH unifies dictionaries, probing, circuits, and saliency maps into
a single geometric object grounded in the network’s input-output map — making
interpretability mechanistic by construction rather than post hoc.
Input Sample
 Centroid
 Input Space
 Centroid Space
Figure 1: TheLinear Centroids Hypothesis(LCH) posits that thecentroidsof a deep network (DN)
represent features as linear directions. The centroids of a DN describe its functional mapping in a
local region of the input space – by being the row-sum of the DN’s input-output Jacobians – and thus
can be considered descriptors of the local “experts” of a DN. In the first and second panels, we show
that the centroids of a Swin-B [30] transformer operating on inputs from ImageNet [28] can be used
as saliency maps. In the third and fourth panels, we verify the LCH by training a DN to classify the
interior from the exterior of a two-dimensional star-shaped polygon (third panel) and observing its
centroids (fourth panel). Indeed, the centroids operating on inputs sampled along the edge features
(third panel) separate along linear directions (fourth panel).
∗Correspondence tothomas.walker@rice.edu.
Preprint.
arXiv:2604.11962v2  [cs.LG]  7 May 2026

1 Introduction
Understanding whatfeaturesa trained Deep Network (DN) has learned to extract from a given input
distribution is a major focus of mechanistic interpretability research today [ 2, 47].Featuresin the
context of a DN are essentially a set of signals or patterns predictive of the output that a trained
DN has learned to extract through its layer-wise operations. The current prevailing framework for
discovery and identification of such features is theLinear Representation Hypothesis(LRH) that
posits features as linear directions in a DN’s activation spaces, i.e., the output spaces of intermediate
layers of a DN [ 12, 41]. LRH has led to the development of various interpretability and feature
discovery tools [21, 26, 54], operating on the assumption that any linear direction in which samples
from an input distribution are ordered in an activation space, corresponds to a feature the network has
learned to compute.
However, LRH suffers from three critical limitations. First, it abstracts away from individual
sub-components of the DN (e.g., neurons, layers, or attention mechanisms), making it difficult to
link features to the computational graph [ 47]. Second, it is unclear whether all linear directions
of intermediate activations correspond to features, making it susceptible to identifying spurious
features [52]. Third, its disjoint application to individual latent spaces has encumbered the field with
the task of contextualizing features across sub-components [3].
To make feature discovery mapping-aware, we introduce theLinear Centroids Hypothesis (LCH).
LCH relies on the network’scentroids, which directly summarize the local mapping or “local
experts” learned by the network. Surprisingly, we observe that samples with the same features
naturally organize into linear directions within this centroid space, intuitively reframing features as
“aligned” local experts (see third and fourth panels of Figure 1). LCH is motivated by the fact that
continuous piecewise affine (CPA) DNs (e.g., ReLU networks) exactly and other smooth networks
(e.g., transformers) approximately partition their input space into a V oronoi-like tiling of polytopal
regions, on which distinct functional mappings (i.e., experts) operate [6].
Crucially, a DN’s centroids can be efficiently calculated for any differentiable DN sub-component,
since they are equal to the row-sum of its input-output Jacobian. This transitions feature identification
from understanding where in the space of intermediate representations an input lies to understanding
the action of the local expert operating on the input, providing a grounded, mechanistic perspective
on interpretability.
Our main contribution is providing a mechanistic framework for interpretability that is interchangeable
with LRH by simply replacing intermediate activations with centroids. We demonstrate that the
transition to using the LCH prevents the identification of spurious features (see third panel of Figure
3), improves the effectiveness of sparse autoencoders for constructing feature dictionaries (see Figure
4), facilitates the introduction of novel techniques for circuit discovery (see Figure 5), and provides a
faithful gradient-based saliency map in the form of thelocal centroid(see Figures 1 and 2).
Because LCH grounds feature discovery in the actual input-output map, it is significantly better at
feature discovery and inherently resists spurious correlations. Ultimately, this centroid-based lens
unifies feature dictionaries, probing, circuit discovery, and saliency maps under a single geomet-
ric object—the network’s induced input-space partition—making interpretability mechanistic by
construction rather than post hoc.2
2 Deep Networks as a Collection of Local Experts
In this section, we demonstrate how DNs can be intuitively thought of as employing local experts to
form their outputs, and thatcentroidsdescribe these experts.
2.1 Deep Networks
A DN f:R d(0)
→R d(L)
, with the convention that d(0) =d , is a composition of L functions
f (ℓ) :R d(ℓ−1)
→R d(ℓ)
. These functions, referred to as layers, typically consist of an affine
transformation followed by a nonlinearity. We will denote the mapping across multiple layers as
f (ℓ1←ℓ2) :R d(ℓ1−1)
→R d(ℓ2 )
for1≤ℓ 1 <ℓ 2≤L.
2Code to study the LCH can be found here: https://github.com/ThomasWalker1/LinearCentroidsHypothesis.
2

2.2 The Geometry of Deep Networks
For any DN, 3 the local behavior of the DN can be approximated locally with an affine transfor-
mation [32, 45]. Thus, for convenience, in the following, we consider continuous piecewise affine
(CPA) DNs (i.e., those that exclusively use CPA operations, such as affine transformations, the ReLU
activation function, or max pooling). CPA DNs construct an implicit geometry by virtue of the fact
that they partition their input space into local regions on which the input-output mapping corresponds
to a specific affine transformation [4]. We can think of these affine transformations as local “experts.”
These regions can be parametrized as apower-diagram subdivision[ 6], a hierarchical combination
ofpower diagrams[ 25] (see Definition A.1). Power diagrams are generalizations of the canonical
V oronoi tiling, which tiles a space by identifying a finite set ofmedianvectors and then assigning
each point in the input space to the closest median with respect to the Euclidean distance. In other
words, each region (or tile) is the loci of the median vectors with respect to the Euclidean distance.
Power diagrams, are similar, except that regions are the loci ofcentroidswith respect to Laguerre
distance [25]. The power diagram subdivision that parametrizes the geometry of a DN constitutes a
layer-wise intersection of power diagrams [6]. This process is described in more detail in Appendix A.
Ultimately, each region of a DN’s geometry has acentroid. It is important to note that the centroid
may not reside inside the region, indeed each region also has an associatedradiusparameter that
influences the relative position of a centroid to the points in the region it defines.
2.3 The Centroids of a Deep Network
Since each sub-component of a CPA DN (e.g., layer or sequence of layers) is also CPA, each DN
sub-component admits a geometry in its input space. For 1≤ℓ 1 <ℓ 2≤L , we denote this geometry
Ω(ℓ1←ℓ2) with regions
n
ω(ℓ1←ℓ2)
ν
o
ν∈V
. Here,V denotes the set of equivalence classes on Rd(ℓ1−1)
of regions. For notational simplicity, we will use ν(x)∈ V to denote the region occupied by
f (1←ℓ1−1)(x)∈R d(ℓ1−1)
. Consequently, for a given pointx∈R d, we can identify its corresponding
centroid at a sub-component asµ (ℓ1←ℓ2)
ν(x) ∈R d(ℓ1−1)
.
Proposition 2.1(Balestriero et al. 6).Let Jx

f (ℓ1←ℓ2)

∈R d(ℓ2 )×d(ℓ1−1)
denote the input-output
Jacobian off (ℓ1←ℓ2) atf (1←ℓ1−1)(x). Then,µ (ℓ1←ℓ2)
ν(x) =

Jx

f (ℓ1←ℓ2)

⊤
1.
Note how Proposition 2.1 enables the efficient computation of DN centroids at individual points in the
input space. Moreover, observe that forℓ2 distinct fromℓ′
2, the centroidsµ(ℓ1←ℓ2)
ν andµ(ℓ1←ℓ′
2)
ν live
in the same space Rd(ℓ1−1)
, but they are distinct since the mapsf (ℓ1←ℓ2) andf (ℓ1←ℓ′
2) are distinct.
This motivates thecentroid spaceterminology, to refer to the space of centroids corresponding to one
specific sub-component of the DN.
Importantly, Proposition 2.1 extends this geometrical perspective beyond CPA DNs, as Jacobian
vector products are readily computable for any differentiable mapping.
2.4 Centroids as Saliency Maps
Centroids describe the action and arrangement of the local experts of a DN, and reside in the input
space of the sub-component on which they are computed. Meaning, for sub-components whose input
space is that of the DN, centroids have a visceral resemblance to a saliency map [ 46, 49, 50, 53].
Indeed, with the first two panels of Figure 1, we can see that the centroid of an input sample highlights
its key features, like edges.
With Figure 2, we support this by showing that the centroid (second panel) of a standard pre-trained
ConvNext-L DN [31] at an input point (first panel) resembles some of the characteristics of the input.
In particular, we can improve thesignalby consulting a neighborhood of local experts, namely,
averaging the centroids of samples in a small neighborhood of the input point.4 Doing so yields a
saliency map that vividly highlights characteristics of the input (third panel).
3Only requiring differentiability, which is the case almost everywhere for any DN.
4This is analogous to SmoothGrad [50].
3

To reinforce the premise that these highlighted characteristics are those pertinent to the output of
the DN, we repeat the computation but for an adversarially trained ConvNext-L DN [ 29]. In the
fourth panel of Figure 2, we observe that the identified characteristics are much more salient, as
expected, because the DN exhibits significantly greater robustness. Consequently, we introduce the
local centroidof an input, namely the average centroid obtained from neighborhood samples of the
input, as a saliency method.
In Appendix B, we explore the local centroids on other inputs and DN architectures. Specifically,
since it has been shown that current gradient-based saliency methods provide essentially unchanged
explanations for both trained and untrained DNs [1], we demonstrate that local centroids of randomly
initialized DNs are uninformative. Furthermore, we show how taking the local centroids of DN
sub-components from the input space to an intermediate layer can elucidate the hierarchical features
of a DN.
Input
Sample
Centroid
(Standard Training)
Local Centroid
(Standard Training)
Local Centroid
(Adversarial Training)
Figure 2: The centroids of a DN can be used as a saliency map. Here, we compute the centroids of
a ConvNext-L DN either with the pre-trained weights from PyTorch [33] or with weights obtained
using the adversarial training methods of Liu et al. [29], at an input (first panel). In the second and
third panels, we visualize the DN’s centroid and local centroid, respectively, trained using standard
methods. In the fourth panel, we visualize the local centroid of the adversarially trained DN.
3 The Linear Centroids Hypothesis
In Section 3.1, we make explicit what we mean by thefeatures of a DNand highlight that the Linear
Representation Hypothesis (LRH) is susceptible to identifyingspuriousfeatures. In Section 3.2, we
propose the Linear Centroids Hypothesis (LCH) as a way to overcome this limitation of the LRH,
since centroids are inherently map-aware. In Section 3.3, we confirm that the LCH is not susceptible
to identifying spurious features.
3.1 Defining Features and Circuits
Inputs to DNs possess defining characteristics (e.g., the characteristic of having whiskers may be
present for input images of cats). A DN effectively groups its input space into regions corresponding
to combinations of these characteristics, which it can then extract to compute its final output.
Thefeaturesof a DN refer to the specific characteristics the DN activelyutilizesto form its output
(e.g., using the characteristic of having whiskers to classify the input as a cat). Crucially, utilization
requires a two-step process:extractinga common representation for inputs sharing the characteristic,
and ensuring this representation meaningfullyinfluencesthe network’s downstream behavior. We
refer to the influence induced by a particular feature as the corresponding circuit. We relate this
notion of DN features to prior works in Appendix C.
It is entirely possible for a DN to extract a representation for an input characteristic without it
ever influencing the final output. We refer to such characteristics asspuriousfeatures. The task of
interpretability is to identify the features and circuits of a DN that are not spurious [47].
Currently, the prevailing framework for interpretability is the Linear Representation Hypothesis
(LRH), which posits that the features of a DN can be identified with linear directions formed by the
4