The Linear Centroids Hypothesis: Features as Directions Learned by Local Experts — part 4/5

Cleaned-text code points [43409, 57166). Bibliography omitted here; source.md includes it.

A Deep Network Geometry
The regions forming the geometry of a DN are constructed by the intersection of a collection of
hyperplanes [4, 16, 36]. Each nonlinearity (neuron) in a layer of the DN determines an activation
level set within its input space (i.e., the hyperplane between being active or inactive in the case of
the ReLU nonlinearity). In a hierarchical fashion, starting from the first layer, these hyperplanes are
pulled back to the input space of the DN and intersect to form the regions [6, 23].
The power diagram subdivision parametrization of the DN geometry, instead views the geometry as a
hierarchical intersection of power diagrams [6].
Definition A.1.Given a collection of Q centroid-radius pairs{(µq,τq)}Q
q=1⊆R d×R , a power
diagram tessellates Rd intoQ disjoint tiles Ω ={ω 1,...,ω Q} such that∪Q
q=1ωq =R d, with each
tile given by
ωq =
(
x∈R d :q= arg min
q′∈{1,...,Q}

∥x−µ q′∥2
2−τq′
)
.(2)
The distance minimized in (2) is called the Laguerre distance [25], and differs from the Euclidean
distance only in the addition of the weighting provided by the radius parameter.
The power diagram subdivision induced by the DN is then constructed recursively through layer-wise
power diagrams. The first layer of the DN partitions the input space Rd(1)
as Ω(1) through a power
diagram. The second layer of the DN then partitions the projections of each tileω(1)∈R d in Rd(1)
induced byf (1) as a power diagram. These are then pulled back to Rd to yield a finer partition of
the input space Ω(1←2). This partition is an example of a power diagram subdivision. Continuing
sequentially for each layer completes the power diagram subdivisionΩof the DN [6].
Just like for a regular power diagram, we can also associate each region in a power diagram subdivision
with acentroidandradius. Despite each tile in a power diagram subdivision being defined implicitly
through a recursive combinatorial intersection of half-spaces, we will show that its centroid and
radius are defined explicitly and computable independently of other regions.
We focus on centroids and how they can be analyzed to understand thefeaturesof a DN, though the
radii are also of considerable interest. Indeed, it is the presence of the radius that means that the
centroid of a region in a power diagram need not be contained within the region itself.
B Local Centroids
Here, we provide further illustrations of local centroids computed for state-of-the-art DNs. In
particular, we demonstrate that for randomly initialized DNs, local centroids contain no information
(see Figure 14). Furthermore, we show that considering DN sub-components from the input space to
hidden layers facilitates the extraction of hierarchical saliency maps. For example, in the top row of
Figure 15, we compare local centroids computed from the input space to a hidden layer and from
the input space to the output layer of a Swin-B transformer [30]. Since the windows of the aircraft
are only apparent in the centroid of the full DN, it follows that the windows are a feature of the later
layers of the DN.
C Comparison to Prior Work
Features.Our notion of a DN feature (see Section 3.1) is analogous to the notion of a DN concept
used in Park et al. [41], which provides a rigorous theoretical characterization of the LRH. In Park
et al. [41], a DN feature is a variable that leads to a particular output when caused by a context. In
our case, the context would correspond to the input samples that possess a given characteristic. The
variable notion of Park et al. [41] would then be equivalent to our requirementrepresentation. Then
the referenced causation on the output would be equivalent to our requirement of causing aninfluence.
Circuits.Circuits were initially introduced in Olah et al. [39] to deal with the apparent poly-
semantic nature of neurons. That is, specific neurons were observed to trigger on seemingly semanti-
cally disjoint inputs, whereas ensembles of neurons demonstrated more reliable activation patterns.
13

Instead, our notion of a circuit arises naturally as the responses of the components of a DN to a
feature. Indeed, this response is likely to involve multiple neurons or components of a DN, given the
DN’s compositional construction.
D The Implication of the LCH on Hyperplane Geometry
In this section, we formalize the relationship between aligned centroids and the hyperplane geometry
of DNs described in Appendix A. We first prove that at any given layer, centroids lying on a one-
dimensional affine subspace imply that the boundaries separating their regions are strictly parallel.
Proposition D.1.At the input space of layer ℓ, if the centroids of a sequence of adjacent linear
regions lie on a one-dimensional affine subspace, then the hyperplanes forming the boundaries
between these consecutive regions are strictly parallel.
Proof. Consider a sequence of adjacent linear regions ω1,...,ω k in the input space of layer ℓ,
separated by boundary hyperplanes Π1,...,Π k−1. Let their corresponding power diagram centroids
beµ1,...,µ k. The shared boundary Πi between regioni andi+ 1 has a normal vector defined by
the difference of their centroids:ni =µ i−µi+1. Assume the centroids lie on a one-dimensional
affine subspace. Then, there exists a base point p and a unit direction vectord such that for all i,
µi =p+t idfor some scalart i. The normal vector for the boundaryΠ i is then:
ni =µ i−µi+1 = (ti−ti+1)d.
Because every normal vector ni is a scalar multiple of the exact same direction d, all boundary
hyperplanesΠ 1,...,Π k−1 share the same normal direction and are therefore strictly parallel.
While Proposition D.1 establishes parallel boundaries in the intermediate representation space of layer
ℓ, the network’s overall behavior is determined by how these boundaries map back to the original
input spaceRd.
The parallel hyperplanes Π1,...,Π k−1 at layerℓ form a tightly packed, linear decision band. The
boundary in the original input space corresponding to Πi is its pre-image under this mapping
x∈R d :f (1←ℓ−1)(x)∈Π i
	
. Because f (1←ℓ−1) is a composition of affine transformations and
non-linearities (like ReLU), it fundamentally acts by folding, stretching, and compressing the input
space.
Low Robustness
 High Robustness
Figure 7: Taken from Humayun et al. [23], this is an illustration of the geometry of a fully connected
ReLU DN as it trains on MNIST. Three training samples are fixed to construct a two-dimensional
slice of the DN’s input domain, and the linear regions that intersect this slice are then visualized. On
theleftis the geometry of the DN when it has generalized but exhibits low levels of robustness. On
therightis the geometry of the DN when it exhibits the additional property of robustness.
When we “pull back” the parallel hyperplanesΠi through this mapping, a single straight hyperplane in
layerℓ pulls back to a continuous, piecewise linear surface in the input space. Each time the pre-image
crosses an activation boundary from an earlier layer, the surface bends, allowing it to approximate
macroscopic curves. Because the hyperplanes Π1,...,Π k−1 are parallel and sequentially packed in
14

layerℓ, their pre-images map to a dense bundle of piecewise linear surfaces in the input space. In
regions where the mappingf (1←ℓ−1) highly compresses the input manifold, this bundle of boundaries
packs densely together. By accumulating these intricately folded, tightly packed linear regions, the
network effectively constructs smooth, arbitrarily curved decision boundaries to separate complex
features in the input data.
Consequently, the linear alignment of centroids at an intermediate layer is the geometric mechanism
by which a DN coordinates a massive accumulation of linear regions to trace out curved, semantically
meaningful feature boundaries in the original input space. An example of this is shown in Figure 7.
E Analyzing Deep Networks Trained to Classify Interiors of Polygons
Here we continue the analysis of the DN training to classify the interior of a star-shaped polygon, as
depicted in the third and fourth panels of Figure 1.
First, we can observe in Figure 8 that the linearity of the centroids of Figure 1 emerges gradually
through training. At initialization, the centroids have a similar arrangement to the input samples
because the DN is randomly initialized. However, as training progresses, we observe that the centroids
slowly migrate and align themselves. In particular, we can see the alignment of the centroids manifest
before they reach their final positions.
Epoch0
 Epoch18
 Epoch27
 Epoch147
 Epoch512
Figure 8: Throughout the training of the DN of the bottom row Figure 1, we tracked the DN centroids
of the input samples highlighted in the bottom left panel Figure 1.
Second, as expected from the discussion of Appendix D, in the second panel of Figure 9 we see that
the trained DN geometry exhibits linear regions with roughly parallel boundaries along the polygon’s
edges. Moreover, in the third and fourth panels of Figure 9, we observe the activation level-sets of the
nonlinearities of the second and third hidden layers of this DN. It is clear that the full identification of
the interior and exterior of the polygon is achieved in the third layer, as the nonlinearities delineate
the polygon’s edges. The second-layer nonlinearities capture the coarser features relating to the three
exterior sectors created by the star shape. This is evidenced by each nonlinearity forming along one
of the polygon’s outer edges.
Third, in addition to centroids possessing a linear structure under the LCH, it is evident from Figure 1
that this structure is semantically coherent. Suggesting that applications of these topological point-
cloud analyses may be particularly fruitful under the LCH. To aid the study of centroids as point clouds,
we soften DNs using CPA nonlinearities (e.g., ReLU) by replacing them with smooth approximations
(e.g., GELU). The GELU nonlinearity [18] belongs to the swish family of nonlinearities [43], which
are theoretically known to provide an appropriate softening of a ReLU DN’s geometry [5]. In the
first panel of Figure 10, we show that this smooth approximation does not impact the structure of the
centroids. In the second panel of Figure 10, we consider a t-SNE embedding [55] of these softened
centroids at the second hidden layer of the DN. We observe clustering of centroids by the sector
of the input domain in which the input samples were located. This corroborates our prior analysis
using the second hidden layer’s geometry. Moreover, it demonstrates how the LCH can be applied
hierarchically. In Figure 1, centroids were computed across the entire DN to understand the finest
features of the DN, whereas in Figure 9, centroids were computed across a single layer to identify
coarser features. Similar analyses under the LRH are challenging, as there is no notion of latent
activations across different scales of sub-components [3].
Fourth, we can analyze the neurons of the DN using Equation (1). In the third panel of Figure 10, we
see that the neurons of the third hidden layer have an encompassing effect on the entire boundary of
the polygon,
15

Input Polygon
 DN Geometry
 2nd Hidden Layer
 3rd Hidden Layer
Figure 9: The activation level-sets of the nonlinearities of a DN have a structure that is related to
features of a DN. Here we train a fully connected ReLU DN with a two-dimensional input space and
a one-dimensional output space to classify the interior and exterior of the star-shaped polygon shown
in the first panel. In the second panel, we visualize the geometry of the trained DN. The color of the
linear regions represents the Frobenius norm of the affine transformation parameterAω operating on
that linear region. We then separate the hyperplanes corresponding to the nonlinearities of the second
layer (third panel) and those of the third layer (fourth panel).
GELU Centroids
 2nd Layer Centroids t-SNE
 s(i,3)
Figure 10: In the first panel, we show that replacing the ReLU nonlinearities with GELU nonlinearities
for the DN of Figure 9 does not change the structure of the centroids significantly. In the second
panel, we show, using t-SNE analysis of the second-layer centroids of the DN in Figure 9, that the
second layer encodes features corresponding to the three external sectors of the star-shaped polygon.
In the third panel, we show that the third layer of the DN of Figure 9 captures the inter-exterior feature
by demonstrating that neurons in this layer are sensitive (as per Equation (1)). More specifically, for
a grid of points in the input space, we apply Equation (1) to a small neighborhood around each point
and visualize the resulting value as a heatmap.
In addition to the star-shaped polygon considered in the main text, in Figure 16 we corroborate the
observed patterns when the input distribution is a bowtie-shaped and reuleaux-shaped polygon.
Moreover, we can show that starting from a non CPA DN, we can repeat the same investigations and
derive similar conclusions. For example, as showing in Figure 11, when replacing the nonlinearity
back to a ReLU we can observe its geometry using SplineCam [23] and see that nonlinearities still
align along the boundary of the polygon, when we observe the centroids of input samples we see the
same linear structures, and the influence of pruning neurons on the centroids is still effective as a
neuron attribution metric.
F Qualitative Analysis of Feature Dictionaries
Here, we provide a qualitative analysis of the feature dictionaries constructed in Section 4.1. To
address the concern that Imagenette may provide a relatively easy setting, we train sparse autoencoders
on a subset of ImageNet sampled from ten random dog breeds.
16