Proof of a perfect platonic representation hypothesis — part 4/4

Cleaned-text code points [42259, 55429). Bibliography omitted here; source.md includes it.

As the theorem implies, transformations of the input data do not break the PRH, and so one naturally wonders if the learned representations are also invariant to full-rank transformations of the label y y . It turns out that, surprisingly, transformations of the label indeed break the PRH. To see this, let us consider the simple case of a two-layer linear network. We assume that all six matrices M O M^{O} , M I M^{I} , Z Z are identities, and the network is trained on different views of the same label: 𝒟 A = { ( x i , Φ A ​ y i ) } i , \mathcal{D}_{A}=\{(x_{i},\Phi_{A}y_{i})\}_{i}, (64) 𝒟 B = { ( x i , Φ B ​ y i ) } i , \mathcal{D}_{B}=\{(x_{i},\Phi_{B}y_{i})\}_{i}, (65) where Φ A \Phi_{A} and Φ B \Phi_{B} are invertible symmetric matrices. With this data, the global minimum for net A A transforms to : V ∗ → Φ A , V^{*}\to\Phi_{A}, (66) and the noise spectrum becomes to Σ ϵ ′ = Φ A ​ Σ ϵ ′ ​ Φ A . \Sigma_{\epsilon}^{\prime}=\Phi_{A}\Sigma_{\epsilon}^{\prime}\Phi_{A}. (67) We can apply Eq. ( 50 ) to the case D = 2 D=2 to obtain that at any global minimum, W 2 ⊤ ​ Σ ′ ¯ ϵ ​ W 2 = W 1 ​ Σ ¯ x ​ W 1 ⊤ , W_{2}^{\top}\bar{\Sigma^{\prime}}_{\epsilon}W_{2}=W_{1}\bar{\Sigma}_{x}W_{1}^{\top}, (68) where Σ ¯ ϵ = Σ ϵ / Tr ⁡ [ Σ ϵ ] \bar{\Sigma}_{\epsilon}={\Sigma}_{\epsilon}/{\rm Tr}[{\Sigma}_{\epsilon}] and Σ ¯ x = Σ x / Tr ⁡ [ Σ x ] \bar{\Sigma}_{x}={\Sigma}_{x}/{\rm Tr}[{\Sigma}_{x}] are the normalized second moment matrices of data. Defining new parameters W ¯ 1 = W 1 ​ Σ x \bar{W}_{1}={W}_{1}\sqrt{\Sigma_{x}} and W ¯ 2 = Σ ϵ ​ Φ A ​ W 2 \bar{W}_{2}=\sqrt{\Sigma_{\epsilon}}\Phi_{A}{W}_{2} . We obtain two equations to solve: W ¯ 2 ⊤ ​ W ¯ 2 = W ¯ 1 ​ W ¯ 1 ⊤ , \bar{W}_{2}^{\top}\bar{W}_{2}=\bar{W}_{1}\bar{W}_{1}^{\top}, (69) W ¯ 2 ​ W ¯ 1 = Σ ϵ ​ Φ A 2 ​ V ∗ . \bar{W}_{2}\bar{W}_{1}=\sqrt{\Sigma_{\epsilon}}\Phi_{A}^{2}V^{*}. (70) The solution is not independent of the transformation Φ A \Phi_{A} . In sharp contrast, the set of equations ( 53 ) and ( 54 ) we had in the proof is transformation-independent. This implies that Φ A \Phi_{A} directly influences the weight representation W 1 ​ x W_{1}x and that the representation is no longer Platonic after the Φ A \Phi_{A} transform. This is an interesting theoretical prediction that has not yet been empirically confirmed.

### Convergence to Saddle or Local Minima Breaks the PRH.

Apparently, if an EDLN converges to a saddle instead of a global minimum, it cannot be perfectly aligned to another EDLN that converges to the global minimum. For EDLN, the saddle points are the low-rank approximations V ∗ V^{*} Fukumizu and Amari (2000) . One can show that these saddle points all have non-negative alignment to each other and to the global minimum, but it is no longer the case that the alignment will be perfect.

### Data Discrepancy / Heterogeneity Breaks the PRH.

For example, suppose x i A = x i + z i A x_{i}^{A}=x_{i}+z_{i}^{A} for some additional and unique feature z i A z_{i}^{A} while x i B = x i + z i B x_{i}^{B}=x_{i}+z_{i}^{B} for some additional and unique feature z i B z_{i}^{B} . Then, it is obvious that at the global minimum, as long as the labels do not depend only on x i x_{i} , the learned representation will not be universal. This agrees with the findings in Ref. Tjandrasuwita et al. (2025) , for example.

### Edge of Stability (EOS) breaks PRH.

It turns out that the same derivation we have shown here can be used to prove the progressive sharpening phenomenon Cohen et al. (2021) , and, surprisingly, within the EDLN model, the force (due to entropy) that gives rise to the progressive sharpening is exactly the force that gives rise to the Platonic representations. As the models learn universal representations, they tend to move towards a gradually sharper loss landscape, eventually reaching a fixed point (namely, the unique minimizers of the Lie-group symmetries discussed below Eq. ( 27 )). This is because the solution a model learns is independent of data (in the case of PRH), whereas the sharpness of a solution is dependent on the data (say, on the Z A Z_{A} matrix). Therefore, for a very ill-conditioned Z A Z_{A} , the Platonic solution can be arbitrarily sharp -- thus, converging towards the Platonic solution leads to progressive sharpening. 20 20 20 More precisely, depending on the data distribution, the same force can also lead to progressive flattening. Therefore, an empirical prediction of the theory is that PRH always happens together with progressive sharpening and/or progressive flattening. Also, it is worthwhile to clarify the phenomenology of EOS. The EOS really is a combination of two independent phenomena: (1) progressive sharpening, where a neural network moves to sharper and sharper places during training, and (2) proper EOS, where the sharpness stays at a critical value and stops increasing further. Our theory shows that progressive sharpening is due to entropy, while the proper EOS is not due to leading-order entropic effects.

However, SGD training can only take the model so far as the “edge of stability” because higher-order effects due to discretized time steps tend to suppress sharpness. When the Platonic solution is very sharp, SGD training cannot reach it due to the need to stay at the edge of stability. There is a lot more depth to this part, which is no longer directly related to the proof of the PRH, and we refer the interested readers to Ref. Ziyin et al. (2025) for solving the EDLN model for progressive sharpening and its intriguing relationship with the PRH.

## 6 Discussion

In this note, we have presented an exactly solvable model that exhibits the perfect PRH. This model suggests that it is the discretization error of training and gradient noise that determines the representation learning, consistent with the observation made in Ref. Ziyin et al. (2024a) . We now discuss various implications of the theory in more detail.

### Cause of the Platonic Representation Hypothesis.

From a pure machine learning perspective, Ref. Huh et al. (2024) conjectured three possible mechanisms that give rise to the PRH: (a) increasing capacity, (b) simplicity bias of training, or (c) multitasking training. Interestingly, the mechanism studied here does not really match these previously conjectured mechanisms. Our mechanism has nothing to do with multitasking. Moreover, our result holds for any (embedded) deep linear network, all having the same capacity and the same level of simplicity, because all solutions parametrize the same input-output map. Here, the cause of the universal representation is the parameter symmetry alone: in the degenerate manifold of solutions, the training algorithm prefers a particular and universal one. Our example highlights how symmetry has been an overlooked fundamental mechanism in deep learning.

### Imperfect Platonic Representation Hypothesis.

In this work, we proved the most strict and idealized version of the PRH, where all possible layers become mutually and perfectly aligned . Experimentally, this has indeed been confirmed to happen for deep linear networks in Ref. Ziyin et al. (2025) . We have also studied how the solution will break away from this perfect PRH. The perfect PRH is a very strong condition. The fact that it can be proved makes it possible to prove weaker forms of it. In reality, the PRH only holds imperfectly, where the data are weakly but significantly aligned. For example, if measured with CKA, the perfect PRH is satisfied when the CKA alignment score is exactly one, whereas, for real models, this value has been measured to be somewhere around 0.2 0.2 Huh et al. (2024) . Thus, extending our results to more realistic settings of nonlinear models and weaker forms of PRH is a crucial future theoretical step.

### Emergent Invariances in Learning Algorithms.

Our theory implies that optimization-unrelated (i.e., due to the implicit regularization effect of the entropic force) learning dynamics may give rise to models that share universal features with each other. What we have focused on is SGD, and a remaining question is to what extent this is true for other learning algorithms. Also, we note that the learned representation is not fully universal in the sense that the learning representation is invariant to variations of the input data (modeled by the Z Z matrices) but is not invariant to the label, as discussed in the previous section. This raises the interesting open question of whether it is possible to have algorithms that learn representations that are also invariant to the transformations of the label. Such a global minimum certainly exists for deep linear networks. For example, if we let the first layer be W 1 = Σ x − 1 W_{1}=\Sigma_{x}^{-1} , then the representation of the first layer will be both invariant to the input data from and to the label; the question is whether it is possible to find learning algorithms that achieve this. This also implies a hierarchy of universality: at the very top, there are learning algorithms whose solutions are almost invariant to every variation of the data (maybe in addition to other things), and below it, there are algorithms whose learning representations are either invariant to input or output, and at the bottom of the hierarchy are those algorithms that are invariant to neither (such as gradient flow).

### Universal Phenomena and Unified Theories.

A rather striking point we made in the previous section is that two seemingly unrelated universal phenomena in deep learning, (1) PRH and (2) progressive sharpening, can have the same hidden cause and can be explained by a unified theory of entropic forces. This raises the hope of being able to find unified explanations of more interesting and intriguing phenomena in deep learning. In particular, this raises the interesting question of whether or not other seemingly unrelated phenomena in deep learning can be found to share a hidden root. Identifying these connections will be a big step forward towards building the theory of deep learning.

### Feature Learning and Infinitesimal Learning Rate.

In the context of the NTK and feature learning, a common result states that for an infinitesimal learning rate (and under certain scaling), SGD converges to the solution of kernel regression. Let θ \theta denote the solution found by SGD at time t t and with learning rate η \eta ; this result states that lim t → ∞ lim η → 0 + θ ⁡ ( t , η ) = θ kernal regression ∗ . \lim_{t\to\infty}\lim_{\eta\to 0^{+}}\theta(t,\eta)=\theta^{*}_{\text{kernal regression}}. (71) This argument is often made assuming that the order of taking these two limits does not matter. However, a key aspect of the proof of our theorem is that these two limits are not equal: θ kernal regression ∗ = lim t → ∞ lim η → 0 + θ ⁡ ( t , η ) ≠ lim η → 0 + lim t → ∞ θ ⁡ ( t , η ) = ? \theta^{*}_{\text{kernal regression}}=\lim_{t\to\infty}\lim_{\eta\to 0^{+}}\theta(t,\eta)\neq\lim_{\eta\to 0^{+}}\lim_{t\to\infty}\theta(t,\eta)=\ ? (72) and cannot be exchanged. The l.h.s. completely ignores the entropic effect, whereas the r.h.s. takes that into account. This could hint at a fundamental problem with understanding AI training with continuous time formalisms, such as in the NTK Jacot et al. (2018) and feature learning Yang and Hu (2020) limit. Arguably, the second limit is much closer to reality because of, for example, the ubiquity of the edge of stability phenomenon, which cannot be approximated by the simple gradient flow.

### More Technical Implications.

While our derivation ended with the solution of Eq. ( 53 ) and ( 54 ), these solutions can be solved in much more depth to give an explicit solution of the global minimum of the EDLN model. This solution can be shown to have many other interesting properties. For example, one can show that all the intermediate layers must become rotation matrices and have the same norm as each other. This explains why all the intermediate layers are aligned with each other even within the same model – it is because the intermediate layers do not do much meaningful computation. At the same time, the first and last layers will be special and do most of the interesting computations. In particular, the first layer will convert the representation to a universal form such that it does not depend on the covariance of x x , in effect whitening the data. The first and last layers will also have different norms and spectrums from all the intermediate layers. This could imply that there are interesting dynamics that only happen at the “surface” of neural networks, resembling the surface modes and edge modes common in physical systems. Thus, one might be interested in developing a theory of surface physics for neural networks. We refer the readers to Refs. Ziyin et al. (2024b) and Ziyin et al. (2025) for more discussion of these technical properties.

## Acknowledgement

The authors thank Yizhou Xu, Tomaso Poggio, Brian Cheung, and Phillip Isola for discussion during the writing of this note. ILC acknowledges support in part from the Institute for Artificial Intelligence and Fundamental Interactions (IAIFI) through NSF Grant No. PHY-2019786.