The Platonic Representation Hypothesis β€” part 3/6

Cleaned-text code points [30339, 45878). Bibliography omitted here; source.md includes it.

This has been previously termed as the Contravariance principle by Cao & Yamins (2024) , which states that the set of solutions to an easy goal is large, while the set of solutions to a challenging goal is comparatively smaller. Moreover, we argue that this narrower solution set also generalizes better. As data scales, models that optimize the empirical risk 𝔼 x ∼ ​ [ β„’ ​ ( f , x ) ] \mathbb{E}_{x\sim{\scriptsize\hbox to25.19pt{\vbox to8.86pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lx@inpgf@ignorespaces\lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3FFB3} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 9.49 C 0 11.02 1.24 12.26 2.77 12.26 L 32.09 12.26 C 33.62 12.26 34.86 11.02 34.86 9.49 L 34.86 2.77 C 34.86 1.24 33.62 0 32.09 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3FFB3} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 9.49 C 0 11.02 1.24 12.26 2.77 12.26 L 32.09 12.26 C 33.62 12.26 34.86 11.02 34.86 9.49 L 34.86 2.77 C 34.86 1.24 33.62 0 32.09 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{}}{{}}{{}}{{}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{2.0pt}{2.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 2.77 2.77)} \pgfsys@hbox{58}\lxSVG@closescope }\lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}[{{{\mathcal{L}}}}(f,x)] also improve on the population risk 𝔼 x ∼ ​ [ β„’ ​ ( f , x ) ] \mathbb{E}_{x\sim{\scriptsize\hbox to21.77pt{\vbox to10.22pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lx@inpgf@ignorespaces\lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3FFB3} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 11.38 C 0 12.91 1.24 14.14 2.77 14.14 L 27.36 14.14 C 28.89 14.14 30.13 12.91 30.13 11.38 L 30.13 2.77 C 30.13 1.24 28.89 0 27.36 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3FFB3} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 11.38 C 0 12.91 1.24 14.14 2.77 14.14 L 27.36 14.14 C 28.89 14.14 30.13 12.91 30.13 11.38 L 30.13 2.77 C 30.13 1.24 28.89 0 27.36 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{}}{{}}{{}}{{}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{2.0pt}{3.3611pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 2.77 4.65)} \pgfsys@hbox{58}\lxSVG@closescope }\lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}[{{{\mathcal{L}}}}(f,x)] , and become better at capturing statistical structures of the true data generating process ( 𝗋𝖾𝖺𝗅𝗂𝗍𝗒 \mathsf{reality} ).

Recent work has demonstrated a power law relationship between data scale and model performance ( Hestness et al., 2017 ) . This implies that with enough data ( e.g. , consisting of the entire internet and all offline scientific measurements) one ought to converge to a very small solution set with irreducible error – the inherent epistemic uncertainty of the world. As more models are trained on internet-scale data, the set of solutions that satisfies all data constraints must become relatively small.

In addition to data-scaling, many modern representation learning objectives ​ ( f , x ) \hbox to10.25pt{\vbox to10.83pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lx@inpgf@ignorespaces\lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3FFB3} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 12.22 C 0 13.75 1.24 14.99 2.77 14.99 L 11.42 14.99 C 12.94 14.99 14.18 13.75 14.18 12.22 L 14.18 2.77 C 14.18 1.24 12.94 0 11.42 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3FFB3} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 12.22 C 0 13.75 1.24 14.99 2.77 14.99 L 11.42 14.99 C 12.94 14.99 14.18 13.75 14.18 12.22 L 14.18 2.77 C 14.18 1.24 12.94 0 11.42 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{}}{{}}{{}}{{}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{2.0pt}{2.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 2.77 2.77)} \pgfsys@hbox{58}\lxSVG@closescope }\lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}(f,x) directly optimize for multi-task solving. Contrastive learning finds a distance structure over data samples that optimizes many classification tasks ( Arora et al., 2019b ; Wang & Isola, 2020 ; Tian et al., 2020b ) . Masked Autoencoders ( He et al., 2021 ) optimize randomly sampled reconstruction tasks. In fact, autoregressive language modeling can also be seen as optimizing a diverse set of tasks ( Radford et al., 2019 ) . Such multi-task objectives may be more effective than single-task ones ( e.g. , ImageNet classification) due to the fact that they impose more task constraints on the representation, leading to a smaller and higher-quality solution space ( Chen et al., 2020 ; He et al., 2020 ; Radford et al., 2017 ; Radford et al., 2019 ) .

### 3.2 Convergence via

Suppose there is a globally optimal representation for standard learning objectives. Then, under sufficient data, scaling a model ( i.e. , using larger function classes ), as well as , should be more effective at finding better approximations to this optimum, as illustrated in Figure 5 . With the same training objective, larger models, even of different architectures, will thus tend to converge toward this optimum. When different training objectives share similar minimizers, larger models are better at finding these minimizers, and will train to similar solutions over the training tasks. We summarize this hypothesis as follows:

### 3.3 Convergence via

Arriving at the same mapping on the training data does not prohibit the models from developing distinct internal representations. It is not unreasonable to posit that the representations used to detect a dog in a 1M parameter model could be quite different than that used by a 1B parameter model. What would stop a billion-parameter (and counting) model from learning an overly complicated and distinct representation? One key factor might be simplicity bias:

Such simplicity bias could be coming from explicit regularization commonly used in deep learning ( e.g. , weight decay and dropout). However, even in the absence of external influences, deep networks naturally adhere to Occam’s razor, that fit the data ( Solomonoff, 1964 ; Gunasekar et al., 2018 ; Arora et al., 2019a ; Valle-Perez et al., 2019 ; Huh et al., 2023 ; Dingle et al., 2018 ; Goldblum et al., 2023 ) . Figure 7 visualizes how simplicity bias can drive convergence.

## 4 What representation are we converging to?

By now, we hope to have convinced the reader that task and data pressures, combined with increasing model capacity, can lead to convergence. We next turn our attention to what exactly is the endpoint of all this convergence.

Our central hypothesis, stated in Figure 1 , is that the representation we are converging toward is a statistical model of the underlying reality that generates our observations. Consistent with the multitask scaling hypothesis, such a representation would naturally be useful toward many tasks (or at least toward any task grounded in reality). Additionally, this representation might be relatively simple, assuming that scientists are correct in suggesting that the fundamental laws of nature are indeed simple functions ( Gell-Mann, 1995 ) , in line with the simplicity bias hypothesis.

But what exactly do we mean by β€œa statistical model of the underlying reality.” In this section, we formalize one definition with concrete mathematical statements. Importantly , this section should be read as just one concrete candidate for the form of the platonic representation; other candidates could be arrived at from other modeling assumptions.

### 4.1 An idealized world

We consider a world that works as follows, consistent with the cartoon in Figure 1 . The world consists of a sequence of T T discrete events, denoted as 𝐙 β‰œ [ z 1 , … , z T ] \mathbf{Z}\triangleq[z_{1},\ldots,z_{T}] , sampled from some unknown distribution β„™ ⁑ ( 𝐙 ) \mathbb{P}(\mathbf{Z}) . Each event can be observed in various ways. An observation is a bijective, deterministic function obs : 𝒡 β†’ β‹… \texttt{obs}:\mathcal{Z}\rightarrow\cdot{}\, that maps events to an arbitrary measurement space, such as pixels, sounds, mass, force, torque, words, etc. Later, in Section 6 , we discuss limitations and potential extensions to continuous and unbounded worlds, and stochastic observations, that could yield a model that better reflects real learning scenarios.

One can think of an event as corresponding to the state of the world at some point in time 3 3 3 Here we only analyze temporal sequences, but note that the same could be done with respect to events laid out in space instead. , but it is also fine to simply consider an event as any variable that indexes observations, with no further physical meaning 4 4 4 This latter interpretation may be more consistent with Plato’s intent. Scholars have argued that his allegory of the cave rejects any notion of a true world state ( Nettleship, 1897 ) . Instead, we could say that the joint distribution of observation indices is itself the platonic reality. .

In this idealized world, knowing β„™ ⁑ ( 𝐙 ) \mathbb{P}(\mathbf{Z}) would be useful for many kinds of predictions; this would constitute a world model over the events that cause our observations ( Werbos, 1987 ; Ha & Schmidhuber, 2018 ; Richens & Everitt, 2024 ) . We will next show that a particular representation of β„™ ⁑ ( 𝐙 ) \mathbb{P}(\mathbf{Z}) is recovered by certain contrastive learners.

### 4.2 A family of contrastive learners converge to a representation of β„™ ⁑ ( 𝐙 ) \mathbb{P}(\mathbf{Z})

Consider a contrastive learner that models observations that cooccur together. For simplicity, we ground our discussion with the following definition of the cooccurrence probability , P π–Όπ—ˆπ—ˆπ—‹ {P}_{\mathsf{coor}} , of two observations x a x_{a} and x b x_{b} both occurring within some window T π—π—‚π—‡π–½π—ˆπ— T_{\mathsf{window}} : P π–Όπ—ˆπ—ˆπ—‹ ( x a , x b ) ∝ βˆ‘ ( t , t β€² ) : | t βˆ’ t β€² | ≀ T π—π—‚π—‡π–½π—ˆπ— β„™ ( X t = x a , X t β€² = x b ) . \displaystyle{P}_{\mathsf{coor}}(x_{a},x_{b})\hskip 7.22743pt\propto\hskip 0.0pt\sum_{(t,t^{\prime})\colon\left\lvert t-t^{\prime}\right\rvert\leq T_{\mathsf{window}}}\hskip-14.45377pt\mathbb{P}(X_{t}=x_{a},X_{t^{\prime}}=x_{b}). Analogously, we can define P π–Όπ—ˆπ—ˆπ—‹ {P}_{\mathsf{coor}} for 𝐙 \mathbf{Z} and other observation modalities. Note that P π–Όπ—ˆπ—ˆπ—‹ {P}_{\mathsf{coor}} is symmetric.

Consider positive pairs as two observations nearby in time (sampled from P π–Όπ—ˆπ—ˆπ—‹ {P}_{\mathsf{coor}} ) and negative pairs as observations drawn from any point in time (sampled independently from the marginal). Our contrastive learner tries to classify if a pair is positive or negative by learning a representation f X : X β†’ ℝ d f_{X}\colon X\rightarrow\mathbb{R}^{d} such that the dot-product kernel approximates the log odds ratio up to some offset: ⟨ f X ​ ( x a ) , f X ​ ( x b ) ⟩ \displaystyle\langle f_{X}(x_{a}),f_{X}(x_{b})\rangle β‰ˆ log ⁑ β„™ ⁑ ( pos | x a , x b ) β„™ ⁑ ( neg | x a , x b ) + c ~ X ​ ( x a ) \displaystyle\approx\log\frac{\mathbb{P}(\texttt{pos}\mathrel{|}x_{a},x_{b})}{\mathbb{P}(\texttt{neg}\mathrel{|}x_{a},x_{b})}+\tilde{c}_{X}(x_{a}) (3) = log ⁑ P π–Όπ—ˆπ—ˆπ—‹ ​ ( x a | x b ) P π–Όπ—ˆπ—ˆπ—‹ ​ ( x a ) + c X ​ ( x a ) \displaystyle=\log\frac{{P}_{\mathsf{coor}}(x_{a}\mathrel{|}x_{b})}{{P}_{\mathsf{coor}}(x_{a})}+c_{X}(x_{a}) (4) = K 𝖯𝖬𝖨 ​ ( x a , x b ) + c X ​ ( x a ) , \displaystyle=K_{\mathsf{PMI}}(x_{a},x_{b})+c_{X}(x_{a}), (5) where K 𝖯𝖬𝖨 K_{\mathsf{PMI}} is the pointwise mutual information (PMI) kernel, and c X ​ ( x a ) c_{X}(x_{a}) is constant in x b x_{b} . We note that this is a common setting for self-supervised contrastive learners with NCE objectives ( Gutmann & HyvΓ€rinen, 2010 ; Oord et al., 2018 ) , including SimCLR ( Chen et al., 2020 ) and SimCSE ( Gao et al., 2021 ) . (See Oord et al. (2018) and Section F.1 for detailed derivations.)

Under mild conditions that the world is smooth enough (see Section F.2 ), a choice of f X f_{X} can exactly represent K 𝖯𝖬𝖨 K_{\mathsf{PMI}} : ⟨ f X ​ ( x a ) , f X ​ ( x b ) ⟩ \displaystyle\langle f_{X}(x_{a}),f_{X}(x_{b})\rangle = K 𝖯𝖬𝖨 ​ ( x a , x b ) + c X , \displaystyle=K_{\mathsf{PMI}}(x_{a},x_{b})+c_{X}, (6) where we observed that c X ​ ( x a ) c_{X}(x_{a}) from Equation 5 must be a constant since both sides are symmetric.

Therefore, the contrastive learners we consider are minimized by a representation f X f_{X} whose kernel is K 𝖯𝖬𝖨 K_{\mathsf{PMI}} (up to a constant offset). With sufficient data and optimization, we will observe convergence to this point.