Revisiting the Platonic Representation Hypothesis: An Aristotelian View — part 6/8
Cleaned-text code points [78332, 94047). Bibliography omitted here; source.md includes it.
Thus, at any fixed sample size n n , increasing the width drives the estimated CKA toward 1 1 regardless of the true alignment ρ 2 \rho^{2} , inflating it by up to 1 − ρ 2 1-\rho^{2} . This width-driven gap is distinct from the 𝒪 ( 1 / n ) \mathcal{O}(1/n) sample-size effect that vanishes as n n grows, and it is exactly what permutation calibration removes. Section E.5 verifies this on real pretrained networks with nonzero true similarity.
##### CCA-based scores.
CCA , SVCCA , and PWCCA fall outside the energy analysis above: their canonical correlations are computed from the whitened cross-covariance, which normalizes away the cross-covariance energy of Proposition 4.1 . They therefore do not follow the 𝒪 ( d / n ) \mathcal{O}(d/n) baseline. Their null instead reflects the rank deficiency of the whitening near and beyond d ≈ n d\approx n : mean CCA peaks at d ≈ n d\approx n and decays as 𝒪 ( n / d ) \mathcal{O}(n/d) for d > n d>n , PWCCA saturates toward its ceiling once d ≳ n d\gtrsim n , and SVCCA is largely width-insensitive because it first projects onto a fixed low-dimensional subspace ( Section E.7 ). Permutation calibration removes this inflation without closed-form solutions, unlike the energy-based metrics.
##### Why we use permutation rather than closed forms.
Closed-form bulk edges are ensemble- and normalization-specific and are brittle to the preprocessing used in practice ( e.g. , centering, whitening, kernelization). Moreover, finite- n n corrections can be non-negligible. We therefore estimate the relevant right-tail behavior nonparametrically via permutation. This yields a conservative, implementation-faithful estimate of chance fluctuations without relying on fragile analytical formulas.
### C.5 The depth confounder
Here we formalize why selection-based summaries ( e.g. , maximum similarity over layer pairs) inflate with the size of the search space using Extreme Value Theory (EVT) .
Let 𝒮 = { S ℓ , ℓ ′ : 1 ≤ ℓ ≤ L A , 1 ≤ ℓ ′ ≤ L B } \mathcal{S}=\{S_{\ell,\ell^{\prime}}:1\leq\ell\leq L_{A},\,1\leq\ell^{\prime}\leq L_{B}\} denote the collection of null similarity fluctuations under H 0 H_{0} , and let M = L A L B M=L_{A}L_{B} .
###### Assumption C.6 (Uniform sub-Gaussian right tails and integrability) .
There exist μ ∈ ℝ \mu\in\mathbb{R} and σ > 0 \sigma>0 such that for all ( ℓ , ℓ ′ ) (\ell,\ell^{\prime}) and all t ≥ 0 t\geq 0 , ℙ ( S ℓ , ℓ ′ − μ ≥ t ) ≤ exp ( − t 2 2 σ 2 ) . \mathbb{P}(S_{\ell,\ell^{\prime}}-\mu\geq t)\leq\exp\!\left(-\frac{t^{2}}{2\sigma^{2}}\right). (49) Moreover, each S ℓ , ℓ ′ S_{\ell,\ell^{\prime}} is integrable: 𝔼 | S ℓ , ℓ ′ | < ∞ \mathbb{E}|S_{\ell,\ell^{\prime}}|<\infty for all ( ℓ , ℓ ′ ) (\ell,\ell^{\prime}) .
###### Proposition C.7 (Maximal inequality, no independence required) .
Under Assumption C.6 and for M ≥ 2 M\geq 2 , 𝔼 [ max ℓ , ℓ ′ S ℓ , ℓ ′ ] ≤ μ + C σ log M , \mathbb{E}\Big[\max_{\ell,\ell^{\prime}}S_{\ell,\ell^{\prime}}\Big]\;\leq\;\mu+C\,\sigma\sqrt{\log M}, (50) where C > 0 C>0 is a constant (e.g., one can take C = 3 C=3 ).
###### Proof.
Let Z := max ℓ , ℓ ′ S ℓ , ℓ ′ − μ Z:=\max_{\ell,\ell^{\prime}}S_{\ell,\ell^{\prime}}-\mu . Since M < ∞ M<\infty and 𝔼 | S ℓ , ℓ ′ | < ∞ \mathbb{E}|S_{\ell,\ell^{\prime}}|<\infty for all ( ℓ , ℓ ′ ) (\ell,\ell^{\prime}) , we have 𝔼 | Z | ≤ 𝔼 [ max ℓ , ℓ ′ | S ℓ , ℓ ′ | ] + | μ | ≤ ∑ ℓ , ℓ ′ 𝔼 | S ℓ , ℓ ′ | + | μ | < ∞ , \mathbb{E}|Z|\leq\mathbb{E}\Big[\max_{\ell,\ell^{\prime}}|S_{\ell,\ell^{\prime}}|\Big]+|\mu|\leq\sum_{\ell,\ell^{\prime}}\mathbb{E}|S_{\ell,\ell^{\prime}}|+|\mu|<\infty, (51) so Z Z is integrable, and the tail-integration formula applies. By the union bound and Assumption C.6 , ℙ ( Z ≥ t ) ≤ M exp ( − t 2 2 σ 2 ) for all t ≥ 0 . \mathbb{P}(Z\geq t)\leq M\exp\!\left(-\frac{t^{2}}{2\sigma^{2}}\right)\qquad\text{for all }t\geq 0. (52) Using the tail-integration formula for an integrable real-valued random variable Z Z , 𝔼 [ Z ] = ∫ 0 ∞ ℙ ( Z ≥ t ) 𝑑 t − ∫ 0 ∞ ℙ ( Z ≤ − t ) 𝑑 t ≤ ∫ 0 ∞ ℙ ( Z ≥ t ) 𝑑 t , \mathbb{E}[Z]=\int_{0}^{\infty}\mathbb{P}(Z\geq t)\,dt-\int_{0}^{\infty}\mathbb{P}(Z\leq-t)\,dt\leq\int_{0}^{\infty}\mathbb{P}(Z\geq t)\,dt, (53) and the bound ℙ ( Z ≥ t ) ≤ 1 \mathbb{P}(Z\geq t)\leq 1 , we obtain 𝔼 [ Z ] ≤ ∫ 0 ∞ min { 1 , M exp ( − t 2 2 σ 2 ) } 𝑑 t . \mathbb{E}[Z]\;\leq\;\int_{0}^{\infty}\min\!\left\{1,\;M\exp\!\left(-\frac{t^{2}}{2\sigma^{2}}\right)\right\}\,dt. (54) Let t 0 = σ 2 log M t_{0}=\sigma\sqrt{2\log M} . This value of t 0 t_{0} is the solution of M exp ( − t 0 2 / 2 σ 2 ) = 1 M\exp\!\left(-t_{0}^{2}/2\sigma^{2}\right)=1 , i.e. , the crossover where the bound min { 1 , ⋅ } \min\{1,\cdot\} switches. Splitting the integral at t 0 t_{0} yields 𝔼 [ Z ] ≤ t 0 + M ∫ t 0 ∞ exp ( − t 2 2 σ 2 ) 𝑑 t . \mathbb{E}[Z]\;\leq\;t_{0}\;+\;M\int_{t_{0}}^{\infty}\exp\!\left(-\frac{t^{2}}{2\sigma^{2}}\right)\,dt. (55) Applying the standard Gaussian tail bound ∫ t 0 ∞ e − t 2 / ( 2 σ 2 ) d t ≤ ( σ 2 / t 0 ) e − t 0 2 / ( 2 σ 2 ) \int_{t_{0}}^{\infty}e^{-t^{2}/(2\sigma^{2})}dt\leq(\sigma^{2}/t_{0})e^{-t_{0}^{2}/(2\sigma^{2})} gives 𝔼 [ Z ] ≤ σ 2 log M + σ 2 log M . \mathbb{E}[Z]\;\leq\;\sigma\sqrt{2\log M}\;+\;\frac{\sigma}{\sqrt{2\log M}}. (56) For M ≥ 2 M\geq 2 , the right-hand side is at most 3 σ log M 3\sigma\sqrt{\log M} , proving the claim with C = 3 C=3 . ∎
##### Remark.
When the S ℓ , ℓ ′ S_{\ell,\ell^{\prime}} are i.i.d. (or weakly dependent), classical Extreme Value Theory yields sharper asymptotics. For example, if S ℓ , ℓ ′ ∼ 𝒩 ( μ 0 , σ 0 2 ) S_{\ell,\ell^{\prime}}\sim\mathcal{N}(\mu_{0},\sigma_{0}^{2}) i.i.d., the centered maximum converges to a Gumbel distribution and 𝔼 [ T max ] ≈ μ 0 + σ 0 ( 2 ln M − ln ln M + ln 4 π 2 2 ln M ) , \mathbb{E}[T_{\max}]\approx\mu_{0}+\sigma_{0}\left(\sqrt{2\ln M}-\frac{\ln\ln M+\ln 4\pi}{2\sqrt{2\ln M}}\right), (57) as stated in standard references ( Cramér, 1999 ; Embrechts et al., 2013 ) . Real layer-wise similarities are dependent, so the approximation above should be treated as heuristic; Proposition C.7 provides a dependence-robust upper bound.
### C.6 Null Baselines for Neighborhood Metrics
The preceding analysis focused on the cross-covariance-energy metrics ( CKA and the RV coefficient), whose null baselines scale with d / n d/n . Neighborhood-based metrics such as mutual k k -NN follow a fundamentally different regime, which we now characterize.
###### Definition C.8 (Mutual k k -NN overlap) .
For representations 𝐗 ∈ ℝ n × d x , 𝐘 ∈ ℝ n × d y \mathbf{X}\in\mathbb{R}^{n\times d_{x}},\mathbf{Y}\in\mathbb{R}^{n\times d_{y}} and neighborhood size k < n k<n , let N 𝐗 ( i ) ⊆ { 1 , … , n } ∖ { i } N_{\mathbf{X}}(i)\subseteq\{1,\ldots,n\}\setminus\{i\} denote the indices of the k k nearest neighbors of sample i i in 𝐗 \mathbf{X} ( e.g. , Euclidean or cosine), and similarly for N 𝐘 ( i ) N_{\mathbf{Y}}(i) . The mutual k k -NN overlap is mKNN ( 𝐗 , 𝐘 ) = 1 n ∑ i = 1 n | N 𝐗 ( i ) ∩ N 𝐘 ( i ) | k . \mathrm{mKNN}(\mathbf{X},\mathbf{Y})=\frac{1}{n}\sum_{i=1}^{n}\frac{|N_{\mathbf{X}}(i)\cap N_{\mathbf{Y}}(i)|}{k}. (58)
###### Proposition C.9 (Uniformity of k k -NN index sets under i.i.d. sampling) .
Fix an anchor index i ∈ { 1 , … , n } i\in\{1,\dots,n\} . Let 𝐱 1 , … , 𝐱 n ∈ ℝ d \mathbf{x}_{1},\dots,\mathbf{x}_{n}\in\mathbb{R}^{d} be i.i.d. and define the k k -NN set N 𝐗 ( i ) ⊆ { 1 , … , n } ∖ { i } N_{\mathbf{X}}(i)\subseteq\{1,\dots,n\}\setminus\{i\} using a fixed distance dist ( ⋅ , ⋅ ) \mathrm{dist}(\cdot,\cdot) . Assume either (i) { dist ( 𝐱 i , 𝐱 j ) } j ≠ i \{\mathrm{dist}(\mathbf{x}_{i},\mathbf{x}_{j})\}_{j\neq i} are almost surely distinct, or (ii) ties are broken by selecting a uniformly random k k -subset among the set of minimizers. Then N 𝐗 ( i ) N_{\mathbf{X}}(i) is uniformly distributed over the ( n − 1 k ) \binom{n-1}{k} k k -subsets of { 1 , … , n } ∖ { i } \{1,\dots,n\}\setminus\{i\} .
###### Proof.
Let ℐ := { 1 , … , n } ∖ { i } \mathcal{I}:=\{1,\dots,n\}\setminus\{i\} be the candidate-neighbor index set. For any permutation π \pi of ℐ \mathcal{I} , i.i.d. sampling implies ( 𝐱 j ) j ∈ ℐ = d ( 𝐱 π ( j ) ) j ∈ ℐ . (\mathbf{x}_{j})_{j\in\mathcal{I}}\stackrel{{\scriptstyle d}}{{=}}(\mathbf{x}_{\pi(j)})_{j\in\mathcal{I}}. The k k -NN selection rule depends on the candidate points only through their distances to 𝐱 i \mathbf{x}_{i} , so permuting the candidate indices permutes the resulting neighbor set. Under either the no-ties assumption or the stated uniform tie-break rule, for any two k k -subsets S , S ′ ⊆ ℐ S,S^{\prime}\subseteq\mathcal{I} there exists a permutation π \pi with π ( S ) = S ′ \pi(S)=S^{\prime} and hence ℙ ( N 𝐗 ( i ) = S ) = ℙ ( N 𝐗 ( i ) = S ′ ) . \mathbb{P}\!\big(N_{\mathbf{X}}(i)=S\big)=\mathbb{P}\!\big(N_{\mathbf{X}}(i)=S^{\prime}\big). Since the events { N 𝐗 ( i ) = S } \{N_{\mathbf{X}}(i)=S\} over all | S | = k |S|=k partition the sample space, each has probability ( n − 1 k ) − 1 \binom{n-1}{k}^{-1} . ∎
###### Theorem C.10 (Null baseline for mutual k k -NN) .
Let 𝐗 , 𝐘 ∈ ℝ n × d \mathbf{X},\mathbf{Y}\in\mathbb{R}^{n\times d} have i.i.d. rows, with 𝐗 \mathbf{X} independent of 𝐘 \mathbf{Y} . Define N 𝐗 ( i ) N_{\mathbf{X}}(i) and N 𝐘 ( i ) N_{\mathbf{Y}}(i) as in Definition C.8 , using either almost sure absence of distance ties or uniform random tie-breaking. Then 𝔼 H 0 [ mKNN ( 𝐗 , 𝐘 ) ] = k n − 1 . \mathbb{E}_{H_{0}}\!\bigg[\mathrm{mKNN}(\mathbf{X},\mathbf{Y})\bigg]\;=\;\frac{k}{n-1}.
###### Proof.
Fix an anchor i i . By Proposition C.9 , N 𝐗 ( i ) N_{\mathbf{X}}(i) and N 𝐘 ( i ) N_{\mathbf{Y}}(i) are each uniform random k k -subsets of the ( n − 1 ) (n-1) -element set { 1 , … , n } ∖ { i } \{1,\dots,n\}\setminus\{i\} . Moreover, since 𝐗 \mathbf{X} and 𝐘 \mathbf{Y} are independent and N 𝐗 ( i ) N_{\mathbf{X}}(i) (resp. N 𝐘 ( i ) N_{\mathbf{Y}}(i) ) is a measurable function of 𝐗 \mathbf{X} (resp. 𝐘 \mathbf{Y} ), the sets N 𝐗 ( i ) N_{\mathbf{X}}(i) and N 𝐘 ( i ) N_{\mathbf{Y}}(i) are independent.
Therefore | N 𝐗 ( i ) ∩ N 𝐘 ( i ) | |N_{\mathbf{X}}(i)\cap N_{\mathbf{Y}}(i)| has a hypergeometric distribution with population size n − 1 n-1 , number of “successes” k k , and draws k k , giving 𝔼 H 0 [ | N 𝐗 ( i ) ∩ N 𝐘 ( i ) | ] = k 2 n − 1 . \mathbb{E}_{H_{0}}\!\bigg[|N_{\mathbf{X}}(i)\cap N_{\mathbf{Y}}(i)|\bigg]=\frac{k^{2}}{n-1}. Substituting into the definition of mKNN \mathrm{mKNN} , 𝔼 H 0 [ mKNN ( 𝐗 , 𝐘 ) ] = 1 n ∑ i = 1 n 𝔼 H 0 [ | N 𝐗 ( i ) ∩ N 𝐘 ( i ) | k ] = 1 n ∑ i = 1 n k n − 1 = k n − 1 . \mathbb{E}_{H_{0}}\bigg[\mathrm{mKNN}(\mathbf{X},\mathbf{Y})\bigg]=\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}_{H_{0}}\!\left[\frac{|N_{\mathbf{X}}(i)\cap N_{\mathbf{Y}}(i)|}{k}\right]=\frac{1}{n}\sum_{i=1}^{n}\frac{k}{n-1}=\frac{k}{n-1}. ∎
###### Proposition C.11 (Per-anchor variance and generic bounds for mKNN \mathrm{mKNN} under the null) .
Under the assumptions of Theorem C.10 , for each anchor i i the intersection size H i := | N 𝐗 ( i ) ∩ N 𝐘 ( i ) | H_{i}:=|N_{\mathbf{X}}(i)\cap N_{\mathbf{Y}}(i)| is hypergeometric with mean k 2 / ( n − 1 ) k^{2}/(n-1) and variance Var [ H i ] = k 2 ( n − 1 − k ) 2 ( n − 1 ) 2 ( n − 2 ) . \mathrm{Var}[H_{i}]\;=\;\frac{k^{2}(n-1-k)^{2}}{(n-1)^{2}(n-2)}. Moreover, since mKNN ( 𝐗 , 𝐘 ) ∈ [ 0 , 1 ] \mathrm{mKNN}(\mathbf{X},\mathbf{Y})\in[0,1] deterministically, we have the fully general bound Var [ mKNN ( 𝐗 , 𝐘 ) ] ≤ 1 4 . \mathrm{Var}[\mathrm{mKNN}(\mathbf{X},\mathbf{Y})]\leq\frac{1}{4}. If one additionally assumes that the per-anchor terms { | N 𝐗 ( i ) ∩ N 𝐘 ( i ) | / k } i = 1 n \{|N_{\mathbf{X}}(i)\cap N_{\mathbf{Y}}(i)|/k\}_{i=1}^{n} are independent (this is a modeling assumption, not a consequence of H 0 H_{0} ), then Var [ mKNN ( 𝐗 , 𝐘 ) ] = 𝒪 ( 1 / n ) \mathrm{Var}[\mathrm{mKNN}(\mathbf{X},\mathbf{Y})]=\mathcal{O}(1/n) .
###### Proof.
The hypergeometric variance formula gives Var [ H i ] = k ⋅ k n − 1 ( 1 − k n − 1 ) ⋅ ( n − 1 ) − k ( n − 1 ) − 1 = k 2 ( n − 1 − k ) 2 ( n − 1 ) 2 ( n − 2 ) . \mathrm{Var}[H_{i}]=k\cdot\frac{k}{n-1}\left(1-\frac{k}{n-1}\right)\cdot\frac{(n-1)-k}{(n-1)-1}=\frac{k^{2}(n-1-k)^{2}}{(n-1)^{2}(n-2)}. The bound Var [ mKNN ] ≤ 1 / 4 \mathrm{Var}[\mathrm{mKNN}]\leq 1/4 follows from mKNN ∈ [ 0 , 1 ] \mathrm{mKNN}\in[0,1] . Under the stated additional independence assumption across anchors, Var [ mKNN ( 𝐗 , 𝐘 ) ] = 1 n Var ( H 1 k ) = 1 n k 2 Var [ H 1 ] , \mathrm{Var}\bigg[\mathrm{mKNN}(\mathbf{X},\mathbf{Y})\bigg]=\frac{1}{n}\,\mathrm{Var}\!\left(\frac{H_{1}}{k}\right)=\frac{1}{nk^{2}}\,\mathrm{Var}[H_{1}], which is 𝒪 ( 1 / n ) \mathcal{O}(1/n) for fixed k k . ∎
## Appendix D Implementation
A key advantage of null calibration is its simplicity: the framework can be applied to any similarity metric with minimal code changes. This section provides pseudocode for the two main calibration procedures described in the paper.
##### Scalar null calibration.
Algorithm 1 shows the complete procedure for calibrating a single similarity comparison. The only requirement is a function similarity(X,Y) that computes the raw metric. The algorithm returns both a permutation p p -value and a calibrated score with a principled zero point.
##### Aggregation-aware calibration for layer-wise comparisons.
When comparing models with multiple layers and reporting a summary statistic ( e.g. , maximum similarity across layer pairs), the aggregation step must also be calibrated. Algorithm 2 shows how to extend scalar calibration to this setting. The key insight is that the same sample permutation must be applied consistently across all layers.
##### Computational cost.
Let C sim C_{\mathrm{sim}} denote the cost of a single similarity evaluation on n n samples. Scalar calibration ( Algorithm 1 ) performs K + 1 K+1 evaluations at cost 𝒪 ( ( K + 1 ) C sim ) \mathcal{O}\!\left((K{+}1)\,C_{\mathrm{sim}}\right) , and aggregation-aware calibration ( Algorithm 2 ) over an L A × L B L_{A}\times L_{B} layer grid costs 𝒪 ( ( K + 1 ) L A L B C sim ) \mathcal{O}\!\left((K{+}1)\,L_{A}L_{B}\,C_{\mathrm{sim}}\right) . The K K null evaluations are independent and run in parallel. The marginal cost per permutation is typically far below C sim C_{\mathrm{sim}} , because permuting the rows of 𝐘 \mathbf{Y} leaves each model’s own representations unchanged. Metrics that are functionals of the n × n n\times n Gram, dissimilarity, or neighbor structures ( CKA , the RV coefficient, RSA , CKNNA) and the neighborhood metrics ( mKNN , cycle- k k NN) build these structures once in 𝒪 ( n 2 d ) \mathcal{O}(n^{2}d) and reduce each null draw to a relabeling, at 𝒪 ( n 2 ) \mathcal{O}(n^{2}) or 𝒪 ( n k ) \mathcal{O}(nk) and independent of the representation width d d . The subspace and shape metrics ( CCA , SVCCA , PWCCA , Procrustes) are the exception: each null draw recomputes an eigendecomposition or SVD of a d d -dependent operator, at up to 𝒪 ( n d 2 + d 3 ) \mathcal{O}(nd^{2}+d^{3}) per permutation, which we batch across permutations on the GPU. On a single NVIDIA GTX 1080 Ti with K = 200 K=200 , this overhead is 20 20 ms per layer pair for linear CKA and 52 52 ms for mKNN , so a full layer-wise comparison between two models completes within a few seconds. We use K ∈ { 200 , … , 500 } K\in\{200,\dots,500\} permutations throughout, which we find sufficient for stable thresholds and p p -values ( Section E.6 ).
## Appendix E Additional Experimental Results
This appendix provides additional analyses that support the main text claims.
### E.1 Phase diagrams across different noise distributions