Proof of a perfect platonic representation hypothesis — part 1/4

Cleaned-text code points [0, 15361). Bibliography omitted here; source.md includes it.

# Proof of a perfect platonic representation hypothesis

###### Abstract

In this note, we elaborate on and explain in detail the proof given by Ziyin et al. (2025) of the “perfect” Platonic Representation Hypothesis (PRH) for the embedded deep linear network model (EDLN). We show that if trained with the stochastic gradient descent (SGD), two EDLNs with different widths and depths and trained on different data will become Perfectly Platonic, meaning that every possible pair of layers will learn the same representation up to a rotation. Because most of the global minima of the loss function are not Platonic, that SGD only finds the perfectly Platonic solution is rather extraordinary. The proof also suggests at least six ways the PRH can be broken. We also show that in the EDLN model, the emergence of the Platonic representations is due to the same reason as the emergence of progressive sharpening. This implies that these two seemingly unrelated phenomena in deep learning can, surprisingly, have a common cause. Overall, the theory and proof highlight the importance of understanding emergent “entropic forces” due to the irreversibility of SGD training and their role in representation learning. The goal of this note is to be instructive while avoiding jargon and lengthy technical details.

## 1 Introduction

A recent line of research in AI has uncovered the emergence of universally aligned and structurally similar representations across different models – a phenomenon referred to as the Platonic Representation Hypothesis Huh et al. (2024) . 1 1 1 We will use the adjectives “universal” and “Platonic” interchangeably. For example, a Platonic solution is also said to be a universal solution. This hypothesis states that large models trained on larger and larger datasets will learn representations that are very close (if not identical) to each other, even if these models have different architectures and may even see different (but often related or paired) data. 2 2 2 In fact, Ref. Tjandrasuwita et al. (2025) showed that if the two models are trained on completely unrelated data, there is no alignment. For example, a language model taking in captions of an image has been found to have a similar representation to the representation of this image in a vision model. Interestingly and perhaps importantly, this similarity of learned representations occurs not only between different AI models but also between AI and biological brains Yamins et al. (2014) , which suggests that understanding this phenomenon may also offer clues for understanding the biological brains and advancing neuroscience.

So far, the alignment between different models is rather weak empirically, and it is unclear whether the PRH is coincidental or reflects something truly profound. This raises three fundamental questions: (1) Is the phenomenology of mutual alignment a reflection of a truly fundamental and universal phenomenon? (2) If so, can we find an idealized scenario where pure and perfect alignment can be observed? (3) Can we have a theory that establishes this fact? We offer positive answers to all three questions. We will identify and solve a nontrivial mathematical model that can exhibit what we call a perfect Platonic Representation Hypothesis. Within this model, the precise cause of this phenomenon can be exactly identified and is due to none of the previously conjectured mechanisms – and the actual cause is a universal mechanism of nature and physics: entropy maximization drives out-of-equilibrium systems towards universal states.

The model we will solve is called an embedded deep linear network (EDLN), which can be thought of as a deep linear network that is embedded in a larger network. We will find its exact solution (the exact form of global minimum) for what we call an ‘‘entropic loss 3 3 3 Not to be confused with the “cross-entropy loss.” ” due to its very close analogy to nonequilibrium physics and thermodynamics. The entropic loss is a first-order approximation of the discrete-time nature and stochasticity of the SGD learning dynamics when modeling it using a continuous-time dynamics Barrett and Dherin (2020) ; Smith et al. (2021) . 4 4 4 This loss is also known as the “modified” loss. As we will discuss below, from a numerical-analysis perspective, the effect is related to the crude numerical integrators (such as SGD) not respecting the symmetries of the dynamics. This entropic loss captures the implicit regularization effects of SGD training, and using this loss allows us to focus only on the geometrical properties of the loss function and avoid the difficult problem of actually solving the SGD dynamics. We will see that even with these simplifications, the solution is still very difficult to reach. In summary, the conclusion of this theory is that in this particular model we solve, the PRH is entirely due to the following mechanism, which we state in three equivalent terms for the ease of understanding for three different communities: • implicit regularization (deep learning theory); 5 5 5 Because the result is due to interesting properties of the learning dynamics and cannot be understood through looking at the training objective alone. However, the mechanism relevant to PRH is the discretization and gradient-noise effect, not the continuous-time limit of SGD, which is the more common scenario in the study of implicit regularizations.

• the growth of backward error of the numerical integrator (numerical analysis);

• the microscopic irreversibility of the learning dynamics (physics). 6 6 6 For this reason, we will use the words “entropic” interchangeably with the “implicit-regularization.”

Other interesting conclusions and broader implications are discussed in Section 6 .

## 2 Problem Setting

### Notation.

For a vector v v , v ⊤ v^{\top} denotes its transpose, v ⊤ ​ v v^{\top}v its inner product, and v ​ v ⊤ vv^{\top} its outer product. θ \theta always denotes the entirety of trainable parameters, and ∇ v \nabla_{v} denotes the gradient with respect to v v . When used without a subscript, ∇ := ∇ θ \nabla:=\nabla_{\theta} is a shorthand for the gradient for all parameters θ \theta . Vectors without transposes are all viewed as column vectors; for example, ∇ v f ​ ( v ) \nabla_{v}f(v) is a column vector and ∇ v ⊤ ​ f ​ ( v ) \nabla_{v}^{\top}f(v) is a row vector. For a matrix W ∈ ℝ d 1 × d 2 W\in\mathbb{R}^{d_{1}\times d_{2}} , the gradient of a scalar function f f with respect to W W is a matrix of the same size: ∇ W ⊤ ​ f ​ ( W ) ∈ ℝ d 1 × d 2 \nabla_{W}^{\top}f(W)\in\mathbb{R}^{d_{1}\times d_{2}} . Other notations are introduced in the context.

### Platonic Representation Hypothesis.

We first define the idealized form of PRH. Let x x denote the input data. Consider two potentially different neural networks A A and B B . Let h A = h A ​ ( x ) h_{A}=h_{A}(x) denote a hidden layer of network A A and h B = h B ​ ( x ) h_{B}=h_{B}(x) that of network B B . It is possible for the input to the two models to take different forms; it is thus a good idea to consider two paired data points ( x i A , x i B ) (x_{i}^{A},x_{i}^{B}) , where x i A x_{i}^{A} is the data point seen by A A and x i B x_{i}^{B} is the data point seen by B B . One example of such a dataset is a multimodal dataset with image and caption pairs, and net A A can be a CNN taking images x i A x_{i}^{A} , and B B can be a language model taking language input x i B x_{i}^{B} . For the purpose of modeling, our theory will consider the case where x i A = Z ​ x i B x_{i}^{A}=Zx_{i}^{B} for some invertible matrix Z Z .

###### Definition 1 .

Two layers h A h_{A} and h B h_{B} are said to be (perfectly) aligned if there exists a constant c 0 c_{0} such that for any x 1 A , x 1 B , x 2 A , x 2 B x_{1}^{A},\ x_{1}^{B},\ x_{2}^{A},\ x_{2}^{B} , c 0 ​ h A ​ ( x 1 A ) ⊤ ​ h A ​ ( x 2 A ) = h B ​ ( x 1 B ) ⊤ ​ h B ​ ( x 2 B ) . c_{0}h_{A}(x_{1}^{A})^{\top}h_{A}(x_{2}^{A})=h_{B}(x_{1}^{B})^{\top}h_{B}(x_{2}^{B}). (1)

The meaning of this definition is that when satisfied, the two models really encode the distance relationships between two different data points in the same way. 7 7 7 If we think of human knowledge as the relationship between different objects, then a perfect alignment implies that the two models really have the same “knowledge” of the world. This is an idealization of the PRH. 8 8 8 This definition of alignment corresponds to the CKA metric used in Ref. Huh et al. (2024) . Any distance between the two sides can be called the “degree of disalignment.” When two networks reach a perfect alignment between every possible pair of hidden layers, we say that these two networks satisfy the “Perfect” PRH. A model parameter θ \theta that satisfies this property is called a “Platonic” or “universal” solution.

A point worth raising is that this usage of the word and definition seems to imply that “being Platonic” is a property of a single model, rather than a property of a pair of models. This criticism is valid because it is possible to imagine four models, A, B, C, and D, such that (1) A and B learn aligned representations, (2) C and D learn aligned representations, yet (3) A/B and C/D learned nonaligned solutions. This can indeed be a problem in general. However, as our proof will show, for the specific model we consider, running SGD makes all networks mutually perfectly aligned. Therefore, the pairwise alignment property can also be regarded as an individual property. It thus makes sense to say that a solution for A is “Platonic” without referring to the reference network B. For now, we will simply assume that this is the case and say words like “Platonic” solutions.

### Model.

The model we study is what we call an embedded deep linear network, whose output for input x x is given by f M ( x ) = M O W D ⋯ W 1 M I x , f_{M}(x)=M^{O}W_{D}\cdots W_{1}M^{I}x, (2) where W i W_{i} is the trainable weight matrix of the i i -th layer, and M O M^{O} and M I M^{I} are arbitrary invertible matrices and are frozen during training. The M M matrices roughly model the layers coming before or after the deep linear network (thus the name “embedded”). The width of this model is defined as the smallest row dimension of all the weight matrices. We will require the width to be larger than or equal to the rank of the target mapping.

We will denote the latent representation of layer i i with a superscript: h i = W i ​ … ​ W 1 ​ M I ​ x . h^{i}=W_{i}...W_{1}M^{I}x. (3) Because we will consider training two different networks, M O M^{O} , and M I M^{I} will be different for the two networks and be written with a subscript A A or B B .

That this model reaches a Platonic solution is by no means trivial. We will explain in Section 3 that almost all of the global minimizers of the loss function (to be defined in the next paragraph) are non-Platonic. That SGD only finds the Platonic ones is rather extraordinary. Also, while this model is a linear function of the input, its learning dynamics and loss landscape are regarded as good models of those of a nonlinear network. Thus, while it may not be a good model to capture the generalization capability or expressivity of neural networks, it captures the learning dynamics of neural networks rather well, as many prior works on deep linear networks have suggested Saxe et al. (2013) .

### Data and loss function.

Let θ = ( W D , … , W 1 ) \theta=(W_{D},...,W_{1}) denote the set of all trainable parameters. The per-sample loss ℓ \ell of a D − D- layer EDLN is: ℓ ( θ , x ) = | | M O W D ⋯ W 1 M I x − y | | 2 , \ell(\theta,x)=||M^{O}W_{D}\cdots W_{1}M^{I}x-y||^{2}, (4) where y = V ∗ ​ x + ϵ y=V^{*}x+\epsilon for an i.i.d. zero-mean label noise ϵ \epsilon with a full-rank covariance matrix Σ ϵ \Sigma_{\epsilon} . 9 9 9 One might wonder if perfect PRH can be reached for a more complicated data distribution where y = y ⁡ ( x ) y=y(x) is a generic function of x x . It turns out that PRH can no longer be achieved for generic labels. A key step in this proof requires the noise (or, prediction residual y ^ − y \hat{y}-y ) to be independent of x x . If the prediction residual is x x -dependent, the global minimum will not be Platonic – this is yet another way to break the PRH. Also, we stress that the labels having a nonvanishing and full-rank noise is quite crucial for the proof of the PRH. This is consistent with a key insight of this theory: it is noise in the gradient that determines representation learning. Similarly, we denote the second moment of x x as Σ x = 𝔼 ⁡ [ x ​ x ⊤ ] . \Sigma_{x}=\mathbb{E}[xx^{\top}]\,. (5) ℓ \ell is called a “per-sample” loss because it is the loss for a single data point and the empirical loss we would like to minimize is its average over the training set: L ⁡ ( θ ) = 𝔼 ⁡ [ ℓ ⁡ ( θ , x ) ] L(\theta)=\mathbb{E}[\ell(\theta,x)] . In the following, we consistently use 𝔼 \mathbb{E} to denote the average over the training set (or over the distribution of data points if the training is online).

Now, we will train two networks (with potentially different depths and widths) on different “views” of the same data, which can be seen as a mathematical abstraction of the multimodal experimental setting of Ref. Huh et al. (2024) . The network A A is trained on 𝒟 Z A \mathcal{D}_{Z_{A}} : 𝒟 Z A = { ( Z A ​ x i , y i ) } i \mathcal{D}_{Z_{A}}=\{(Z_{A}x_{i},y_{i})\}_{i} (6) and network B B is trained on 𝒟 Z B \mathcal{D}_{Z_{B}} , 𝒟 Z B = { ( Z B ​ x i , y i ) } i , \mathcal{D}_{Z_{B}}=\{(Z_{B}x_{i},y_{i})\}_{i}, (7) where Z A Z_{A} and Z B Z_{B} are arbitrary invertible matrices. Using the notation of Eq. ( 1 ), this means that x i A = Z A ​ x i , x_{i}^{A}=Z_{A}x_{i}, (8) x i B = Z B ​ x i . x_{i}^{B}=Z_{B}x_{i}. (9) Note that with the three matrices Z A , M O , M I Z_{A},M^{O},M^{I} , the global minimum of L L satisfies: W D ​ … ​ W 1 = ( M O ) − 1 ​ V ∗ ​ ( M I ) − 1 ​ Z A − 1 , W_{D}...W_{1}=(M^{O})^{-1}V^{*}(M^{I})^{-1}Z_{A}^{-1}, (10) and at the global minimum θ ∗ \theta^{*} , the loss function value is determined by the noise ϵ \epsilon : L ⁡ ( θ ∗ ) = Tr ⁡ [ Σ ϵ ] . L(\theta^{*})={\rm Tr}[\Sigma_{\epsilon}]. (11)

### Entropic loss due to SGD training.

Recent works showed that training with SGD at a finite learning rate η \eta and batch size is equivalent to minimizing the following objective Barrett and Dherin (2020) ; Smith et al. (2021) ; Ziyin et al. (2025) : 𝔼 ⁡ [ ℓ ⁡ ( θ , x ) ] + η ​ 𝔼 ​ ‖ ∇ ℓ ‖ 2 , \mathbb{E}[\ell(\theta,x)]+\eta{\mathbb{E}\|\nabla\ell\|^{2}}, (12) which states that SGD training implicitly introduces an additional regularization term equal to the expected norm of the gradient. Because this can be seen as an analogue of the entropic force 10 10 10 An entropic force is a force in physics that cannot be explained by looking at the energy function alone. , we will refer to 𝔼 ⁡ [ ‖ ∇ θ ℓ ‖ 2 ] := S \mathbb{E}[\|\nabla_{\theta}\ell\|^{2}]:=S as the “entropy.” We will consider the minimizers of the η → 0 + \eta\to 0^{+} limit of the loss function, which is equivalent to finding the solution of min θ : L ⁡ ( θ ) = Tr ⁡ [ Σ ϵ ] 𝔼 ∥ ∇ ℓ ∥ 2 , \min_{\theta:L(\theta)={\rm Tr}[\Sigma_{\epsilon}]}{\mathbb{E}\|\nabla\ell\|^{2}}, (13) meaning that we are finding the minimal gradient norm solution θ \theta with the constraint that θ \theta is a global minimum of L ⁡ ( θ ) L(\theta) .