Proof of a perfect platonic representation hypothesis — part 3/4

Cleaned-text code points [27680, 42259). Bibliography omitted here; source.md includes it.

The second type of T T we choose is a symmetric matrix with exactly two off-diagonal nonvanishing terms. For two fixed but arbitrary indices j ≠ j ′ j\neq j^{\prime} , T j ​ j ′ = T j ​ j ′ = 1 , T_{jj^{\prime}}=T_{jj^{\prime}}=1, (37) and all the other elements are zero. Plug this definition into Eq. ( 31 ), we obtain 𝔼 [ ∇ W i j : ⊤ ℓ ∇ W i j ′ : ℓ ] = 𝔼 [ ∇ W i + 1 : j ⊤ ℓ ∇ W i + 1 : j ′ ℓ ] . {\mathbb{E}\left[\nabla_{W_{i}^{j:}}^{\top}\ell\nabla_{W_{i}^{j^{\prime}:}}\ell\right]}={\mathbb{E}\left[\nabla_{W_{i+1}^{:j}}^{\top}\ell\nabla_{W_{i+1}^{:j^{\prime}}}\ell\right]}. (38) This result, together with Eq. ( 41 ), can be written as a concise matrix form: 𝔼 [ ∇ W i j : ℓ ∇ W i j ′ : ⊤ ℓ ] = 𝔼 [ ∇ W i + 1 : j ℓ ∇ W i + 1 : j ′ ⊤ ℓ ] . \mathbb{E}\left[\nabla_{W_{i}^{j:}}\ell\nabla_{W_{i}^{j^{\prime}:}}^{\top}\ell\right]=\mathbb{E}\left[\nabla_{W_{i+1}^{:j}}\ell\nabla_{W_{i+1}^{:j^{\prime}}}^{\top}\ell\right]. (39) Now, define K = W i + 1 ​ W i K=W_{i+1}W_{i} , and using the chain rule, W i ∇ K ⊤ ℓ = ∇ W i j : ℓ , W_{i}\nabla_{K}^{\top}\ell=\nabla_{W_{i}^{j:}}\ell, (40) we can obtain a suggestive equation that any (local or) global minimizer of Eq. ( 12 ) must satisfy: 15 15 15 Note that this equation is very general and independent of the actual forms of the loss function or the architecture. It can be proved to hold whenever the loss function has the matrix rescaling symmetry: ℓ ⁡ ( W , U ) = ℓ ⁡ ( W ​ U ) \ell(W,U)=\ell(WU) . This equation, for example, holds for the query and key weight matrices within any self-attention layer in a transformer. Therefore, this equation is really only a consequence of parameter symmetry and entropic forces. For a general theory of how the entropy term changes in the presence of Lie-group parameter symmetries, see Ref. Ziyin et al. (2025) . W i ​ 𝔼 ​ [ ∇ K ⊤ ℓ ​ ∇ K ℓ ] ​ W i ⊤ = W i + 1 ​ 𝔼 ​ [ ∇ K ℓ ​ ∇ K ⊤ ℓ ] ​ W i + 1 ⊤ , \boxed{W_{i}\mathbb{E}[\nabla_{K}^{\top}\ell\nabla_{K}\ell]W_{i}^{\top}=W_{i+1}\mathbb{E}[\nabla_{K}\ell\nabla_{K}^{\top}\ell]W_{i+1}^{\top},} (41) where K = W i + 1 ​ W i K=W_{i+1}W_{i} . Simply using the chain rule leads to ∇ K ⊤ ℓ = ∇ h i + 1 ℓ ​ h i − 1 ⊤ , \nabla_{K}^{\top}\ell=\nabla_{h_{i+1}}\ell h_{i-1}^{\top}, (42) and so 𝔼 ⁡ [ ∇ K ⊤ ℓ ​ ∇ K ℓ ] = 𝔼 ⁡ [ ‖ ∇ h i + 1 ℓ ‖ 2 ​ h i − 1 ​ h i − 1 ⊤ ] . \mathbb{E}[\nabla_{K}^{\top}\ell\nabla_{K}\ell]=\mathbb{E}[\|\nabla_{h_{i+1}}\ell\|^{2}h_{i-1}h_{i-1}^{\top}]. (43) The quantity ∇ h i + 1 ℓ = W i + 2 ⊤ ​ … ​ W D ⊤ ​ ( M O ) ⊤ ​ ( f A ​ ( x ) − y ) = W i + 2 ⊤ ​ … ​ W D ⊤ ​ ( M O ) ⊤ ​ ϵ , \nabla_{h_{i+1}}\ell=W_{i+2}^{\top}...W_{D}^{\top}(M^{O})^{\top}(f_{A}(x)-y)=W_{i+2}^{\top}...W_{D}^{\top}(M^{O})^{\top}\epsilon, (44) where we have used the relation f A ​ ( x ) − y = ϵ f_{A}(x)-y=\epsilon at the global minimum ( 20 ), and so this term is independent of h i − 1 h_{i-1} because of the independence between ϵ \epsilon and x x . Thus, 𝔼 ⁡ [ ∇ K ⊤ ℓ ​ ∇ K ℓ ] = 𝔼 ⁡ [ ‖ ∇ h i + 1 ℓ ‖ 2 ] ​ 𝔼 ​ [ h i − 1 ​ h i − 1 ⊤ ] . \mathbb{E}[\nabla_{K}^{\top}\ell\nabla_{K}\ell]=\mathbb{E}[\|\nabla_{h_{i+1}}\ell\|^{2}]\mathbb{E}[h_{i-1}h_{i-1}^{\top}]. (45) A similar argument leads to 𝔼 ⁡ [ ∇ K ℓ ​ ∇ K ⊤ ℓ ] = 𝔼 ⁡ [ ‖ h i − 1 ‖ 2 ] ​ 𝔼 ​ [ ∇ h i + 1 ℓ ​ ∇ h i + 1 ⊤ ℓ ] . \mathbb{E}[\nabla_{K}\ell\nabla_{K}^{\top}\ell]=\mathbb{E}[\|h_{i-1}\|^{2}]\mathbb{E}[\nabla_{h_{i+1}}\ell\nabla_{h_{i+1}}^{\top}\ell]. (46) Together, this means that Eq. ( 41 ) can be written as 16 16 16 Note that this equation can be written as 𝔼 ⁡ [ h i ​ h i ⊤ ] ∝ 𝔼 ⁡ [ ∇ h i ℓ ​ ∇ h i ⊤ ℓ ] \mathbb{E}[h_{i}h_{i}^{\top}]\propto\mathbb{E}[\nabla_{h_{i}}\ell\nabla_{h_{i}}^{\top}\ell] and so this step also proves the representation-gradient alignment hypothesis in Ref. Ziyin et al. (2024a) for the EDLN model. a h ​ W i ​ 𝔼 ​ [ h i − 1 ​ h i − 1 ⊤ ] ​ W i ⊤ = a g ​ W i + 1 ⊤ ​ 𝔼 ​ [ ∇ h i + 1 ℓ ​ ∇ h i + 1 ℓ ⊤ ] ​ W i + 1 . a_{h}W_{i}\mathbb{E}[h_{i-1}h_{i-1}^{\top}]W_{i}^{\top}=a_{g}W_{i+1}^{\top}\mathbb{E}[\nabla_{h_{i+1}}\ell\nabla_{h_{i+1}}\ell^{\top}]W_{i+1}. (47) where a h = 1 / 𝔼 [ ∥ [ h i − 1 ∥ 2 ] a_{h}=1/\mathbb{E}[\|[h_{i-1}\|^{2}] and a g = 1 / 𝔼 ⁡ [ ‖ ∇ h i + 1 ℓ ‖ 2 ] a_{g}=1/\mathbb{E}[\|\nabla_{h_{i+1}}\ell\|^{2}] .

We can define M ¯ I := W i − 1 ​ … ​ W 1 ​ M I ​ Z A \bar{M}^{I}:=W_{i-1}...W_{1}M^{I}Z_{A} and M ¯ O := M O ​ W D ​ W i + 2 \bar{M}^{O}:=M^{O}W_{D}W_{i+2} so that h i − 1 = M ¯ I ​ x , h_{i-1}=\bar{M}^{I}x, (48) ∇ h i + 1 ℓ = ( M ¯ O ) ⊤ ​ ϵ . \nabla_{h_{i+1}}\ell=(\bar{M}^{O})^{\top}\epsilon. (49) Thus, Eq. ( 41 ) can be written as a h ​ W i ​ M ¯ I ​ Σ x ​ ( M ¯ I ) ⊤ ​ W i ⊤ = a g ​ W i + 1 ⊤ ​ ( M ¯ O ) ⊤ ​ Σ ϵ ​ M ¯ O ​ W i + 1 . a_{h}W_{i}\bar{M}^{I}\Sigma_{x}(\bar{M}^{I})^{\top}W_{i}^{\top}=a_{g}W_{i+1}^{\top}(\bar{M}^{O})^{\top}\Sigma_{\epsilon}\bar{M}^{O}W_{i+1}. (50)

Together with Eq. ( 20 ), we have obtained two equations to solve for two unknown matrices W i W_{i} and W i + 1 W_{i+1} . 17 17 17 Here, assuming that both matrices are square matrices with dimension d 2 d^{2} , there are 2 ​ d 2 2d^{2} many degrees of freedom to be determined. The first equation has d 2 d^{2} constraints, and the second equation has d ⁡ ( d + 1 ) / 2 d(d+1)/2 constraints. Together, this implies that there are d ⁡ ( d − 1 ) / 2 d(d-1)/2 remaining degrees of freedom, which (as it turns out) corresponds to the rotation degrees of freedom between W i + 1 W_{i+1} and W i W_{i} . However, these degrees of freedom do not affect the universal alignment at all and so are benign for our purpose. This set of equations can be simplified greatly if we define W ¯ i = W i ​ M ¯ I ​ Σ x \bar{W}_{i}={W}_{i}\bar{M}^{I}\sqrt{\Sigma_{x}} (51) W ¯ i + 1 = Σ ϵ ​ ( M ¯ O ) ⊤ ​ W i + 1 . \bar{W}_{i+1}=\sqrt{\Sigma_{\epsilon}}(\bar{M}^{O})^{\top}{W}_{i+1}. (52) From which we obtain two very simple equations to solve: W ¯ i + 1 ​ W ¯ i = Σ ϵ ​ V ∗ ​ Σ x := V ¯ , \boxed{\bar{W}_{i+1}\bar{W}_{i}=\sqrt{\Sigma_{\epsilon}}V^{*}\sqrt{\Sigma_{x}}:=\bar{V},} (53) a h ​ W ¯ i ​ W ¯ i ⊤ = a g ​ W ¯ i + 1 ⊤ ​ W ¯ i + 1 . \boxed{a_{h}\bar{W}_{i}\bar{W}_{i}^{\top}=a_{g}\bar{W}_{i+1}^{\top}\bar{W}_{i+1}.} (54) The solution to this set of equations can be written using the SVD of V ¯ = E l ​ Σ ​ E r \bar{V}=E_{l}\Sigma E_{r} , where E l E_{l} and E r E_{r} are the singular vectors and Σ \Sigma is a positive diagonal matrix of all the singular values. The solutions that satisfy this set of equations can be exhaustively written as W ¯ i = R ​ S ​ E r , \bar{W}_{i}=R\sqrt{S}E_{r}, (55) W ¯ i = E l ​ S ​ R ⊤ , \bar{W}_{i}=E_{l}\sqrt{S}R^{\top}, (56) where R R is an arbitrary orthogonal matrix, which does not affect the universality of the solution.

For our purpose, it is sufficient to note that the solutions of W ¯ i \bar{W}_{i} and W ¯ i + 1 \bar{W}_{i+1} are completely independent of M I , M O M^{I},M^{O} and Z A Z_{A} , and once we determine W ¯ i \bar{W}_{i} , we can determine W i W_{i} .

Now, what is the representation h i h_{i} ? h i ​ ( Z A ​ x ) = W i ​ M ¯ I ​ x = W ¯ i ​ Σ x + ​ x , h_{i}(Z_{A}x)=W_{i}\bar{M}^{I}x=\bar{W}_{i}\sqrt{\Sigma_{x}}^{+}x, (57) where the + superscript denotes the pseudoinverse. But recall that W ¯ i \bar{W}_{i} is completely independent of M I , M O M^{I},M^{O} and Z A Z_{A} (up to a scalar constant a h / a g a_{h}/a_{g} ), and so h i h_{i} must be independent of these matrices (!!!). This equation is also independent of the layer index i i , and so we have proved the first result: every layer of f A f_{A} must be aligned with every possible layer of f A f_{A} .

The same argument applies to net B B , whose layers all obey the same equation in Eq. ( 53 ) and ( 54 ) (which are independent of the network index A A and B B ). This proves the second and third parts of the theorem: every layer of A A is aligned to every layer of B B . Thus, we have proved Theorem 1 .

###### Remark .

A key tool in the proof is the Lie-group symmetries in a deep linear network. While the proof we presented here does not rely on the learning dynamics, it may be possible to do an alternative proof by establishing the convergence of SGD to Platonic solutions. One possibility is to leverage the stochastic differential equation formalism of Ref. Ziyin et al. (2024b) to prove the convergence.

It is also worthwhile to comment on this derivation from the perspective of conventional numerical analysis. In numerical analysis, it is well known that the numerical integrators tend to solve an alternative “modified” problem that is perturbatively away from the original. In our theory, the entropic loss in Eq. ( 12 ) is exactly the modified problem being solved by the SGD algorithm (and η \eta is the perturbative parameter), and Eq. ( 12 ) can be seen as the modified energy functional of the continuous gradient flow when integrated with Euler discretization. It is an established wisdom in numerical analysis that it is these backward errors (the entropy term) that dominate the long-term behavior of the integrator, especially when there are symmetries in the dynamical variables Hairer et al. (2006) . Thus, it is no surprise that these integrators select very special solutions from a large degenerate manifold of solutions.

The network we studied is a deep linear model, which raises the question of whether linearity is necessary for PRH. There are two possible ways to interpret this result: (1) perfect PRH is achievable even in nonlinear models, and there is a hidden mechanism that is shared between deep linear and nonlinear models that leads to this effect (one such mechanism could be overparametrization); (2) perfect PRH is only achievable in deep linear networks, and so the closeness to perfect alignment can be seen as a metric of how close the representation is to a linearized representation. This might be a particularly relevant hypothesis because linearized structures are found to emerge quite often in later layers of neural networks. That being said, whether this is the case or not is not answerable in our current framework, and is an interesting open problem.

## 5 What breaks the perfect plato?

This theory suggests an interesting new perspective for understanding and studying the PRH. Conventionally, one assumes that having no alignment is the default and asks why and when one gets a positive alignment. However, through the perfect PRH, one can think of perfect alignment as the default scenario, and ask the question of what breaks one away from this perfection, which could be much easier to study. 18 18 18 Just like how deviations from the ideal gas are much easier to study than deviations from a, say, strongly interacting fluid with a complicated equation of state, deviations from a simple “idealized” learning system are far more tractable than a fully realistic, messy system. In this section, we take this new perspective and tentatively answer the question of “when is the perfect PRH broken?”

### Weight Decay Breaks PRH.

One thing that breaks the perfect alignment between two layers is the use of weight decay. This can be understood easily (if we are hand-wavy). For simplicity, consider training two models while setting both of their M O M^{O} and M I M^{I} to be identity, and we have two different Z A ≠ Z B Z_{A}\neq Z_{B} applied to transform the input data x x . If we train with a small but positive weight decay and with gradient flow (so that the entropic term η ​ S \eta S is exactly zero), we are really minimizing the following objective: min θ : L ⁡ ( θ ) = Tr ⁡ [ Σ ϵ ] ∥ θ ∥ 2 . \min_{\theta:L(\theta)={\rm Tr}[\Sigma_{\epsilon}]}{\|\theta\|^{2}}. (58) For illustration and for conciseness, let us also focus on the case where Z A Z_{A} and Z B Z_{B} commute with V ∗ V^{*} and with each other, and let us also assume that they are all positive semidefinite matrices so that we can write things like Z A 1 / 3 Z_{A}^{1/3} , but the conclusion still holds without these assumptions. They are really there to make the equations easy to read. Now, the solution of Eq. ( 58 ) for the EDLN model is (and up to rotations that do not matter) W D = … = W 1 = ( V ∗ ​ Z A − 1 ) 1 / D . W_{D}=...=W_{1}=(V^{*}Z_{A}^{-1})^{1/D}. (59) Essentially, this is because the use of weight decay causes different layers to have a balanced norm. Now, consider the first-layer representation, for example, h i = W i ​ … ​ W 1 ​ Z A ​ x = ( V ∗ ​ Z A − 1 ) i / D ​ Z A ​ x = ( V ∗ ) i / D ​ Z A D − i D ​ x . h^{i}=W_{i}...W_{1}Z_{A}x=(V^{*}Z_{A}^{-1})^{i/D}Z_{A}x=(V^{*})^{i/D}Z_{A}^{\frac{D-i}{D}}x. (60) Therefore, as long as i ≠ D i\neq D , the representation always depends on Z A , Z_{A}, which is an arbitrary transformation. Thus, the latent representations of network A A depend on Z A Z_{A} , and those of B B depend on Z B Z_{B} for all the latent layers. This directly implies that these layers cannot be perfectly aligned. A more technical proof of this can be found in Ref. Ziyin et al. (2025) .

### Gradient Flow Breaks PRH.

Now, assume that we are really training with the gradient flow (GF) algorithm on L L (instead of training in the zero learning rate limit of GD or SGD): θ ˙ = − ∇ θ L ​ ( θ ) . \dot{\theta}=-\nabla_{\theta}L(\theta). (61) This training procedure is thus not regularized by gradient or by weight decay. Now, because of the ‘‘double rotation” symmetries of the EDLN loss function, a large set of conserved quantities 19 19 19 Namely, functions of θ \theta , g ⁡ ( θ ) g(\theta) that do not change during training: d d ​ t ​ f ​ ( θ ) = 0 \frac{d}{dt}f(\theta)=0 . exist: for all i i , d d ​ t ​ ( W i + 1 ⊤ ​ W i + 1 − W i ​ W i ⊤ ) = 0 , \frac{d}{dt}(W_{i+1}^{\top}W_{i+1}-W_{i}W_{i}^{\top})=0, (62) which implies that W i + 1 ⊤ ​ ( t ) ​ W i + 1 ​ ( t ) = W i ​ ( t ) ​ W i ⊤ ​ ( t ) + A 0 , W_{i+1}^{\top}(t)W_{i+1}(t)=W_{i}(t)W_{i}^{\top}(t)+A_{0}, (63) where A 0 A_{0} is the difference between these two matrices at initialization and can be arbitrary. This means that the spectra of W i W_{i} and W i + 1 W_{i+1} are dependent on their initializations. This result is rather well-known and has been used in a lot of prior works to analyze the learning dynamics of neural networks Du et al. (2018) .

The existence of these conserved quantities essentially implies that GF converges to solutions that are strongly initialization-dependent. Thus, if networks A A and B B have different initializations, they will not share perfectly aligned representations. In contrast, when the training proceeds with SGD, these quantities are no longer conserved, and it is the breaking of these conservation laws due to entropy that leads to the PRH .

### Label Transformations Breaks PRH.