Proof of a perfect platonic representation hypothesis β€” part 2/4

Cleaned-text code points [15361, 27680). Bibliography omitted here; source.md includes it.

Note a key property of the entropy term: S = βˆ‘ i 𝔼 ​ β€– βˆ‡ W i β„“ β€– 2 = βˆ‘ i 𝔼 ⁑ [ β€– βˆ‡ h i + 1 β„“ β€– 2 ​ β€– h i β€– 2 ] . S=\sum_{i}\mathbb{E}\|\nabla_{W_{i}}\ell\|^{2}=\sum_{i}\mathbb{E}[{\|\nabla_{h^{i+1}}\ell\|^{2}}{\|h^{i}\|^{2}}]. (14) where β€– βˆ‡ h i β„“ β€– 2 {\|\nabla_{h^{i}}\ell\|^{2}} is the representation gradient, a global quantity, β€– h i β€– 2 \|h^{i}\|^{2} is the representation, a local quantity. The entropy thus introduces a coupling between the global gradient and the local representation , encouraging both terms to be as simple as possible. 11 11 11 In the sense of having a small norm. However, these two terms cannot be simultaneously minimized, and the tradeoff effects between the representational simplicity and gradient simplicity will give rise to the PRH.

### Main result.

What we aim to prove in this note is the following theorem.

###### Theorem 1 .

(Perfect Platonic Representation Hypothesis Ziyin et al. (2025) ) We train f A f_{A} on π’Ÿ Z A \mathcal{D}_{Z_{A}} and f B f_{B} on π’Ÿ Z B \mathcal{D}_{Z_{B}} . Let the width of A A and B B be no smaller than the rank of V βˆ— V^{*} . Let both networks be at any global minimum of Eq. ( 12 ). Then, for any invertible M A O , M A I , Z A , M B O , M B I , Z B M^{O}_{A},M^{I}_{A},Z_{A},M^{O}_{B},M^{I}_{B},Z_{B} , for any possible pair of i i and j j , 1. h A i h^{i}_{A} and h A j h^{j}_{A} are perfectly aligned;

2. h B i h^{i}_{B} and h B j h^{j}_{B} are perfectly aligned;

3. h A i h^{i}_{A} and h B j h^{j}_{B} are perfectly aligned.

Part (3) is what one would usually call a PRH, and essentially implies parts (1) and (2). To summarize the setting and its meaning, this model is different from the standard deep linear network with the three additional arbitrary matrices M O , M I , Z M^{O},\ M^{I},\ Z such that β€’ the deep linear network W D β‹― W 1 W_{D}\cdots W_{1} models a block of network module under consideration;

β€’ the matrices M I M^{I} and M O M^{O} model the part of the architecture that comes after and before this specific module, and are different for different networks; 12 12 12 An important remaining question is to what extent these matrices can be made nonlinear. In some sense, the matrix M I M^{I} can be regarded as a linearization of a nonlinear module around some cluster mean x β€² x^{\prime} , and so the theory states that to first order in the variance of x x , the model learns a universal representation. The matrix M O M^{O} is more easily generalizable to nonlinear cases because a lot of empirical works have shown that the last layers of a well-trained neural network essentially behave like linear layers Masarczyk et al. (2024) , and universal representations tend to emerge in the later layers.

β€’ the matrix Z Z models different views/modalities of the same underlying data.

Taking these interpretations, the theorem thus states that any EDLN module in the same or two different networks will learn the same representation (different up to a rotation and scaling), even if the architectures of the models are different and even if they are trained on different forms of input data. Also, note that the two EDLNs do not have to have the same width or depth, and do not have to have the same initializations.

## 3 Most global minima are not platonic

It should first be emphasized that Theorem 1 is rather extraordinary. Even for the simple model of EDLN, most of the global minima of L L are not Platonic. Consider the empirical loss: L ( ΞΈ ) = 𝔼 | | M O W D β‹― W 1 M I x βˆ’ y | | 2 , L(\theta)=\mathbb{E}||M^{O}W_{D}\cdots W_{1}M^{I}x-y||^{2}, (15) There are a lot of symmetries (transformations of ΞΈ \theta that leave the loss function value unchanged) in the loss function. For example, if ( W D βˆ— , … , W i + 1 βˆ— , W i βˆ— , … , W 1 βˆ— ) (W_{D}^{*},...,W_{i+1}^{*},W_{i}^{*},...,W_{1}^{*}) is a global minimum, then, ( W D βˆ— , … , W i + 1 βˆ— ​ T , T βˆ’ 1 ​ W i βˆ— , … , W 1 βˆ— ) (W_{D}^{*},...,W_{i+1}^{*}T,T^{-1}W_{i}^{*},...,W_{1}^{*}) (16) is also a global minimum for any invertible matrix T T . Therefore, if ΞΈ A βˆ— = ( W D βˆ— , … , W i + 1 βˆ— , W i βˆ— , … , W 1 βˆ— ) \theta^{*}_{A}=(W_{D}^{*},...,W_{i+1}^{*},W_{i}^{*},...,W_{1}^{*}) has a representation that is perfectly aligned to the representations of f B f_{B} , one has h A ⊀ ​ ( x 1 A ) ​ h A ​ ( x 2 A ) = h B ⊀ ​ ( x 1 B ) ​ h B ​ ( x 2 B ) . h_{A}^{\top}(x_{1}^{A})h_{A}(x_{2}^{A})=h_{B}^{\top}(x_{1}^{B})h_{B}(x_{2}^{B}). (17) After the transformation by T T on net A A , the l.h.s. becomes h A ⊀ ​ ( x 1 ) ​ ( T βˆ’ 1 ) ⊀ ​ T βˆ’ 1 ​ h A ​ ( x 2 ) , h_{A}^{\top}(x_{1})(T^{-1})^{\top}T^{-1}h_{A}(x_{2}), (18) whereas the r.h.s. remains unchanged. For a general T T , these two layers can no longer be perfectly aligned. Therefore, there are infinitely many minimizers of L L that are not Platonic (in fact, almost all solutions are non-Platonic). This means that it is highly nontrivial for SGD training to only learn the Platonic solution, even for the simple EDLN model.

## 4 Proof

We present the proof of Theorem 1 in a rather pedagogical manner. Also, because the derivation is the same for net A A and net B B , we focus on net A A . The most important equations in the derivation are placed in a b ​ o ​ x .

First of all, it is obvious (or, a good exercise to show) that any global minimum of 𝔼 x ∼ π’Ÿ Z A ​ [ β„“ ] \mathbb{E}_{x\sim\mathcal{D}_{Z_{A}}}[\ell] satisfies W D ​ … ​ W 1 = ( M O ) βˆ’ 1 ​ V βˆ— ​ ( M I ) βˆ’ 1 ​ Z A βˆ’ 1 W_{D}...W_{1}=(M^{O})^{-1}V^{*}(M^{I})^{-1}Z_{A}^{-1} , and this is achievable as long as the model’s width is larger than rank ⁑ ( V βˆ— ) {\rm rank}(V^{*}) . So, it suffices to identify the solutions ΞΈ \theta that minimize 𝔼 ⁑ [ β€– βˆ‡ β„“ β€– 2 ] \mathbb{E}[\|\nabla\ell\|^{2}] (19) subject to the constraint W D ​ … ​ W 1 = ( M O ) βˆ’ 1 ​ V βˆ— ​ ( M I ) βˆ’ 1 ​ Z A βˆ’ 1 . W_{D}...W_{1}=(M^{O})^{-1}V^{*}(M^{I})^{-1}Z_{A}^{-1}. (20)

Thus, let us consider a solution that satisfies W D ​ … ​ W 1 = ( M O ) βˆ’ 1 ​ V βˆ— ​ ( M I ) βˆ’ 1 ​ Z A βˆ’ 1 W_{D}...W_{1}=(M^{O})^{-1}V^{*}(M^{I})^{-1}Z_{A}^{-1} . Now, we perform the following transformation of this solution such that it does not change L L but reduces 𝔼 ⁑ [ β€– βˆ‡ β„“ β€– 2 ] \mathbb{E}[\|\nabla\ell\|^{2}] . Fix a layer i i , and let W i β†’ e Ξ» ​ T ​ W i , W_{i}\to e^{\lambda T}W_{i}, (21) W i + 1 β†’ W i + 1 ​ e βˆ’ Ξ» ​ T , W_{i+1}\to W_{i+1}e^{-\lambda T}, (22) where T T is an arbitrary symmetric matrix. By standard results of Lie groups, e βˆ’ Ξ» ​ T e^{-\lambda T} is full-rank, and its inverse is e Ξ» ​ T e^{\lambda T} . Thus, L L does not change after this transformation. How does the entropy term change?

Let W i Ξ» = e Ξ» ​ T ​ W i W_{i}^{\lambda}=e^{\lambda T}W_{i} , and W i + 1 Ξ» = W i + 1 ​ e βˆ’ Ξ» ​ T W_{i+1}^{\lambda}=W_{i+1}e^{-\lambda T} . Then, by parameter symmetry, for any Ξ» \lambda , β„“ ⁑ ( W i Ξ» , W i + 1 Ξ» ) = β„“ ⁑ ( W i , W i + 1 ) . \ell(W_{i}^{\lambda},W_{i+1}^{\lambda})=\ell(W_{i},W_{i+1}). (23) Taking derivative with respect to W i W_{i} and W i + 1 W_{i+1} and applying chain rule, we obtain: e Ξ» ​ T ​ βˆ‡ W i Ξ» β„“ ​ ( W i Ξ» , W i + 1 Ξ» ) = βˆ‡ W i β„“ ​ ( W i , W i + 1 ) . \boxed{e^{\lambda T}\nabla_{W_{i}^{\lambda}}\ell(W_{i}^{\lambda},W_{i+1}^{\lambda})=\nabla_{W_{i}}\ell(W_{i},W_{i+1}).} (24) βˆ‡ W i + 1 Ξ» β„“ ​ ( W i Ξ» , W i + 1 Ξ» ) ​ e βˆ’ Ξ» ​ T = βˆ‡ W i + 1 β„“ ​ ( W i , W i + 1 ) . \boxed{\nabla_{W_{i+1}^{\lambda}}\ell(W_{i}^{\lambda},W_{i+1}^{\lambda})e^{-\lambda T}=\nabla_{W_{i+1}}\ell(W_{i},W_{i+1}).} (25) Thus, the gradient is not invariant to the symmetry transformation. We will see in the following derivation that it is this equivariance of the gradient with respect to the symmetry transformation that leads to the Platonic representation.

Applying these equations, one can show that this transformation only affects the part of S S that is due to βˆ‡ W i β„“ \nabla_{W_{i}}\ell and βˆ‡ W i + 1 β„“ \nabla_{W_{i+1}}\ell . Thus, S = Tr ⁑ [ 𝔼 ⁑ [ βˆ‡ W i β„“ ​ βˆ‡ W i ⊀ β„“ ] ] + Tr ⁑ [ βˆ‡ W i + 1 β„“ ​ βˆ‡ W i + 1 ⊀ β„“ ] + other terms. S={\rm Tr}[\mathbb{E}[\nabla_{W_{i}}\ell\nabla_{W_{i}}^{\top}\ell]]+{\rm Tr}[\nabla_{W_{i+1}}\ell\nabla_{W_{i+1}}^{\top}\ell]+\text{ other terms.} (26) This transformation changes the two terms to Tr ⁑ [ e βˆ’ 2 ​ Ξ» ​ T ​ 𝔼 ​ [ βˆ‡ W i β„“ ​ βˆ‡ W i ⊀ β„“ ] ] + Tr ⁑ [ e 2 ​ Ξ» ​ T ​ 𝔼 ​ [ βˆ‡ W i + 1 β„“ ​ βˆ‡ W i + 1 ⊀ β„“ ] ] . {\rm Tr}[e^{-2\lambda T}\mathbb{E}\left[\nabla_{W_{i}}\ell\nabla_{W_{i}}^{\top}\ell]\right]+{\rm Tr}[e^{2\lambda T}\mathbb{E}\left[\nabla_{W_{i+1}}\ell\nabla_{W_{i+1}}^{\top}\ell]\right]. (27) Observe (!!!) that this equation reaches its minimum at a unique Ξ» = Ξ» βˆ— \lambda=\lambda^{*} (which can be checked by taking the derivative of Ξ» \lambda and that its derivative is a monotonic function that passes through zero or extrapolates towards zero). This means that there exists a unique Ξ» \lambda for which the entropy is minimized. This is a general result for any Lie-group symmetries (e.g., see Theorem 4.3 of Ref. Ziyin et al. (2024b) and Theorem 4 of Ref. Ziyin et al. (2025) ). Formally, the Ξ» \lambda that minimizes S S can be found by taking the derivative of S S with respect to Ξ» \lambda and set to zero: d d ​ Ξ» ​ S \displaystyle\frac{d}{d\lambda}S ∝ Tr ⁑ [ βˆ’ T ​ e βˆ’ 2 ​ Ξ» ​ T ​ 𝔼 ​ [ βˆ‡ W i β„“ ​ βˆ‡ W i ⊀ β„“ ] ] + Tr ⁑ [ T ​ e 2 ​ Ξ» ​ T ​ 𝔼 ​ [ βˆ‡ W i + 1 β„“ ​ βˆ‡ W i + 1 ⊀ β„“ ] ] \displaystyle\propto{\rm Tr}[-Te^{-2\lambda T}\mathbb{E}\left[\nabla_{W_{i}}\ell\nabla_{W_{i}}^{\top}\ell]\right]+{\rm Tr}[Te^{2\lambda T}\mathbb{E}\left[\nabla_{W_{i+1}}\ell\nabla_{W_{i+1}}^{\top}\ell]\right] (28) = Tr ⁑ [ βˆ’ T ​ 𝔼 ​ [ βˆ‡ W i Ξ» β„“ ​ βˆ‡ W i Ξ» ⊀ β„“ ] ] + Tr ⁑ [ T ​ 𝔼 ​ [ βˆ‡ W i + 1 Ξ» β„“ ​ βˆ‡ W i + 1 Ξ» ⊀ β„“ ] ] \displaystyle={\rm Tr}[-T\mathbb{E}\left[\nabla_{W_{i}^{\lambda}}\ell\nabla_{W_{i}^{\lambda}}^{\top}\ell]\right]+{\rm Tr}[T\mathbb{E}\left[\nabla_{W_{i+1}^{\lambda}}\ell\nabla_{W_{i+1}^{\lambda}}^{\top}\ell]\right] (29) = 0 . \displaystyle=0. (30) Equivalently, at any global minimum (or local minimum), the following equation must be satisfied for any symmetric T T : Tr ⁑ [ T ​ 𝔼 ​ [ βˆ‡ W i Ξ» β„“ ​ βˆ‡ W i Ξ» ⊀ β„“ ] ] = Tr ⁑ [ T ​ 𝔼 ​ [ βˆ‡ W i + 1 Ξ» β„“ ​ βˆ‡ W i + 1 Ξ» ⊀ β„“ ] ] . \boxed{{\rm Tr}[T\mathbb{E}\left[\nabla_{W_{i}^{\lambda}}\ell\nabla_{W_{i}^{\lambda}}^{\top}\ell]\right]={\rm Tr}[T\mathbb{E}\left[\nabla_{W_{i+1}^{\lambda}}\ell\nabla_{W_{i+1}^{\lambda}}^{\top}\ell]\right]}. (31) Now, one can consider two types of T T . The first type is a diagonal T T such that for a fixed index j j , T k ​ l = { 1 if k = l = j ; 0 otherwise. T_{kl}=\begin{cases}1&\text{if $k=l=j$;}\\ 0&\text{otherwise.}\end{cases} (32) Note that this transformation is identical to the standard β€˜β€˜rescaling symmetry.” Using its definition, one immediately arrives at a β€˜β€˜gradient” balance condition: 13 13 13 Note the following interesting fact: for an EDLN or a ReLU network, gradient regularization leads to gradient balance, whereas weight decay leads to a norm balance, and these two balancing conditions must trade off with each other in reality. 𝔼 [ βˆ₯ βˆ‡ W i j : β„“ βˆ₯ 2 ] = 𝔼 [ βˆ₯ βˆ‡ W i + 1 : j β„“ βˆ₯ 2 ] {\mathbb{E}\left[\|\nabla_{W_{i}^{j:}}\ell\|^{2}\right]}={\mathbb{E}\left[\|\nabla_{W_{i+1}^{:j}}\ell\|^{2}\right]} (33) where W i j : W_{i}^{j:} is the j j -th row of W i W_{i} , and W i + 1 : j W_{i+1}^{:j} is the j j -th column of W i + 1 W_{i+1} , and this condition holds for every i i and j j . 14 14 14 To see why this is the unique minimizer of S S , note that for this symmetry, the two terms in Eq. ( 27 ) simplifies to: Tr [ e βˆ’ 2 ​ Ξ» ​ T 𝔼 [ βˆ‡ W i β„“ βˆ‡ W i ⊀ β„“ ] ] = e βˆ’ 2 ​ Ξ» 𝔼 [ βˆ₯ βˆ‡ W i j : β„“ βˆ₯ 2 ] + constants in Ξ» , {\rm Tr}[e^{-2\lambda T}\mathbb{E}\left[\nabla_{W_{i}}\ell\nabla_{W_{i}}^{\top}\ell]\right]=e^{-2\lambda}\mathbb{E}\left[\|\nabla_{W_{i}^{j:}}\ell\|^{2}\right]+\text{ constants in $\lambda$}, (34) Tr [ e βˆ’ 2 ​ Ξ» ​ T 𝔼 [ βˆ‡ W i + 1 β„“ βˆ‡ W i + 1 ⊀ β„“ ] ] = e 2 ​ Ξ» 𝔼 [ βˆ₯ βˆ‡ W i + 1 : j β„“ βˆ₯ 2 ] constants in Ξ» , {\rm Tr}[e^{-2\lambda T}\mathbb{E}\left[\nabla_{W_{i+1}}\ell\nabla_{W_{i+1}}^{\top}\ell]\right]=e^{2\lambda}\mathbb{E}\left[\|\nabla_{W_{i+1}^{:j}}\ell\|^{2}\right]\text{ constants in $\lambda$}, (35) where W i j : W_{i}^{j:} is the j j -th row of W i W_{i} , and W i + 1 : j W_{i+1}^{:j} is the j j -th column of W i + 1 W_{i+1} . The sum of these two terms is minimized uniquely at: e 4 ​ Ξ» = 𝔼 [ βˆ₯ βˆ‡ W i j : β„“ βˆ₯ 2 ] 𝔼 [ βˆ₯ βˆ‡ W i + 1 : j β„“ βˆ₯ 2 ] . e^{4\lambda}=\frac{\mathbb{E}\left[\|\nabla_{W_{i}^{j:}}\ell\|^{2}\right]}{\mathbb{E}\left[\|\nabla_{W_{i+1}^{:j}}\ell\|^{2}\right]}. (36)