The Platonic Representation Hypothesis — part 2/6

Cleaned-text code points [15916, 30339). Bibliography omitted here; source.md includes it.

We set out to address these claims in a broader scope to determine whether models are indeed learning an increasingly modality-agnostic representation of the world. We sampled a variety of models trained either solely on vision or solely on language, and compared their representations as they became larger and more competent over many tasks.

In Figure 3 , we assess alignment between a suite of language models and vision models. So far we have only defined alignment for two kernels defined over the same input space. To measure cross-modal alignment, we use paired datasets to bridge the two modalities. For vision and text, we use the Wikipedia captions dataset { ( x i , y i ) } i \{(x_{i},y_{i})\}_{i} ( Srinivasan et al., 2021 ) , composed of images from Wikipedia ( x i x_{i} ) and their corresponding captions ( y i y_{i} ). We then measure alignment between a language model f text f_{\texttt{text}} and a vision model f img f_{\texttt{img}} as the alignment of the two following kernels: K img ​ ( i , j ) = ⟨ f img ​ ( x i ) , f img ​ ( x j ) ⟩ \displaystyle K_{\texttt{img}}(i,j)=\langle f_{\texttt{img}}(x_{i}),f_{\texttt{img}}(x_{j})\rangle (1) K text ​ ( i , j ) = ⟨ f text ​ ( y i ) , f text ​ ( y j ) ⟩ . \displaystyle K_{\texttt{text}}(i,j)=\langle f_{\texttt{text}}(y_{i}),f_{\texttt{text}}(y_{j})\rangle. (2) Using this analysis, we find that the better an LLM is at language modeling, the more it tends to aligns with vision models, as shown in Figure 3 . The converse effect also holds: the better a vision models is, the more it tends to align with LLMs. See Section C.2 for more details.

### 2.4 Models are increasingly aligning to brains

Neural networks also show substantial alignment with biological representations in the brain ( Yamins et al., 2014 ) . This commonality may be due to similarities in the task and data constraints both systems are confronted with. Even though the mediums may differ – silicon transistors versus biological neurons – the fundamental problem faced by brains and machines is the same: efficiently extracting and understanding the underlying structure in images, text, sounds, etc. ( Barlow et al., 1961 ; Olshausen & Field, 1997 ) . Sorscher et al. (2022) developed a theoretical framework for how the efficient extraction of novel concepts occurs for both the human visual system and deep networks. The tasks that the human visual system has been honed to perform through evolution – like segmentation, detection, and whole-image classification – are also the ones that we train our neural nets to perform. Yamins et al. (2014) went as far as to title their work in the spirit that performance over such tasks implies brain alignment. Antonello & Huth (2024) posited that it is less the particular task and more the generality of the representations that explain their alignment with biological representations. Further, Conwell et al. (2022) showed that training data plays a large role in alignment. Psychophysical studies have also shown agreement between how humans perceive visual similarity and how models do, even when the models are trained on tasks, such as self-supervised prediction, that are seemingly unrelated to mimicking human perception ( Zhang et al., 2018 ) .

### 2.5 Does alignment predict downstream performance?

If models are converging towards a more accurate representation of reality, we expect that alignment should correspond to improved performance on downstream tasks. Figure 4 supports this hypothesis by demonstrating improved performance on commonsense reasoning (Hellaswag; Zellers et al. (2019) ) and mathematical problem solving (GSM8K; Cobbe et al. (2021) ) as alignment improves.

## 3 Why are representations converging?

Modern machine learning models are generally trained to minimize the empirical risk with possible implicit and/or explicit regularization: f ∗ ﹇ trained model = f ∈ ﹈ function class ​ 𝔼 x ∼ ​ [ ﹇ training objective ​ ( f , x ) ] + ﹈ regularization ​ ( f ) {\color[rgb]{0.5,0.5,0.5}\overbracket{\color[rgb]{0,0,0}f^{*}}^{\mathclap{\textsf{trained model}}}}{}=\hbox to36.25pt{\vbox to12.62pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lx@inpgf@ignorespaces\lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3B3FF} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 14.7 C 0 16.23 1.24 17.47 2.77 17.47 L 47.39 17.47 C 48.92 17.47 50.16 16.23 50.16 14.7 L 50.16 2.77 C 50.16 1.24 48.92 0 47.39 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3B3FF} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 14.7 C 0 16.23 1.24 17.47 2.77 17.47 L 47.39 17.47 C 48.92 17.47 50.16 16.23 50.16 14.7 L 50.16 2.77 C 50.16 1.24 48.92 0 47.39 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{}}{{}}{{}}{{}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{2.0pt}{3.94444pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 2.77 5.46)} \pgfsys@hbox{58}\lxSVG@closescope }\lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}_{f\in{\color[rgb]{0.5,0.5,0.5}\underbracket{\scriptsize\hbox to9.16pt{\vbox to8.78pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#808080} \lxSVG@begingroup@{fill=#808080} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lx@inpgf@ignorespaces\lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3B3FF} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 9.39 C 0 10.91 1.24 12.15 2.77 12.15 L 9.91 12.15 C 11.44 12.15 12.67 10.91 12.67 9.39 L 12.67 2.77 C 12.67 1.24 11.44 0 9.91 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3B3FF} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 9.39 C 0 10.91 1.24 12.15 2.77 12.15 L 9.91 12.15 C 11.44 12.15 12.67 10.91 12.67 9.39 L 12.67 2.77 C 12.67 1.24 11.44 0 9.91 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{}}{{}}{{}}{{}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{2.0pt}{2.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 2.77 2.77)} \pgfsys@hbox{58}\lxSVG@closescope }\lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}_{\mathclap{\scriptstyle\textsf{function class}}}}}\mathbb{E}_{x\sim{\scriptsize\hbox to25.19pt{\vbox to8.86pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lx@inpgf@ignorespaces\lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3FFB3} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 9.49 C 0 11.02 1.24 12.26 2.77 12.26 L 32.09 12.26 C 33.62 12.26 34.86 11.02 34.86 9.49 L 34.86 2.77 C 34.86 1.24 33.62 0 32.09 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3FFB3} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 9.49 C 0 11.02 1.24 12.26 2.77 12.26 L 32.09 12.26 C 33.62 12.26 34.86 11.02 34.86 9.49 L 34.86 2.77 C 34.86 1.24 33.62 0 32.09 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{}}{{}}{{}}{{}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{2.0pt}{2.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 2.77 2.77)} \pgfsys@hbox{58}\lxSVG@closescope }\lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}[{\color[rgb]{0.5,0.5,0.5}\overbracket{\hbox to10.25pt{\vbox to10.83pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#808080} \lxSVG@begingroup@{fill=#808080} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lx@inpgf@ignorespaces\lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3FFB3} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 12.22 C 0 13.75 1.24 14.99 2.77 14.99 L 11.42 14.99 C 12.94 14.99 14.18 13.75 14.18 12.22 L 14.18 2.77 C 14.18 1.24 12.94 0 11.42 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#B3FFB3} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 12.22 C 0 13.75 1.24 14.99 2.77 14.99 L 11.42 14.99 C 12.94 14.99 14.18 13.75 14.18 12.22 L 14.18 2.77 C 14.18 1.24 12.94 0 11.42 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{}}{{}}{{}}{{}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{2.0pt}{2.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 2.77 2.77)} \pgfsys@hbox{58}\lxSVG@closescope }\lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}^{\mathclap{\textsf{training objective}}}}(f,x)]+{\color[rgb]{0.5,0.5,0.5}\underbracket{\hbox to11.36pt{\vbox to10.83pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#808080} \lxSVG@begingroup@{fill=#808080} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lx@inpgf@ignorespaces\lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#FFB3B3} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 12.22 C 0 13.75 1.24 14.99 2.77 14.99 L 12.95 14.99 C 14.48 14.99 15.72 13.75 15.72 12.22 L 15.72 2.77 C 15.72 1.24 14.48 0 12.95 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} {}{}{}{}{}{}{}{}\lxSVG@begingroup@{fill=#FFB3B3} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\lxSVG@fill\lxSVG@drawpath@unclipped{M 0 2.77 L 0 12.22 C 0 13.75 1.24 14.99 2.77 14.99 L 12.95 14.99 C 14.48 14.99 15.72 13.75 15.72 12.22 L 15.72 2.77 C 15.72 1.24 14.48 0 12.95 0 L 2.77 0 C 1.24 0 0 1.24 0 2.77 Z}{stroke:none} \lx@inpgf@ignorespaces\lxSVG@closescope \lxSVG@begingroup@{_scopebegin=1} \lxSVG@fill@opacity{1.0}\lxSVG@begingroup@{fill-opacity=1.0} {{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{\lx@inpgf@ignorespaces}}{{}}{{}}{{}}{{}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{2.0pt}{2.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 2.77 2.77)} \pgfsys@hbox{58}\lxSVG@closescope }\lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}_{\mathclap{\textsf{regularization}}}}(f) In the following sections, we lay out how each colored component in this optimization process potentially plays a role in facilitating representational convergence.

### 3.1 Convergence via

Each training datapoint and objective (task) places an additional constraint on the model. As data and tasks scale, the volume of representations that satisfy these constraints must proportionately grow smaller, as visualized in Figure 6 and stated below: