The Platonic Representation Hypothesis — part 1/6

Cleaned-text code points [0, 15916). Bibliography omitted here; source.md includes it.

# The Platonic Representation Hypothesis

###### Abstract

We argue that representations in AI models, particularly deep networks, are converging. First, we survey many examples of convergence in the literature: over time and across multiple domains, the ways by which different neural networks represent data are becoming more aligned. Next, we demonstrate convergence across data modalities: as vision models and language models get larger, they measure distance between datapoints in a more and more alike way. We hypothesize that this convergence is driving toward a shared statistical model of reality, akin to Plato’s concept of an ideal reality. We term such a representation the platonic representation and discuss several possible selective pressures toward it. Finally, we discuss the implications of these trends, their limitations, and counterexamples to our analysis.

###### Keywords:

## 1 Introduction

AI systems are rapidly evolving into highly multifunctional entities. For example, whereas in the past we had special-purpose solutions for different language processing tasks ( e.g. , sentiment analysis, parsing, dialogue), modern large language models (LLMs) are competent at all these tasks using a single set of weights ( Srivastava et al., 2022 ) . Unified systems are also being built across data modalities: instead of using a different architecture for processing images versus text, recent models, such as GPT4-V ( OpenAI, 2023 ) , Gemini ( Google, 2023 ) , and LLaVA ( Liu et al., 2023 ) , handle both modalities with a combined architecture. More and more systems are built off of general-purpose pretrained backbones, sometimes called foundation models ( Bommasani et al., 2021 ) , that support a large range of tasks, including robotics ( Driess et al., 2023 ; Brohan et al., 2023 ) , bioinformatics ( Ma et al., 2024 ) , and healthcare ( Steinberg et al., 2021 ) . In short, AI systems are becoming increasingly homogeneous in both their architectures and their capabilities.

This paper explores one aspect of this trend: representational convergence. We argue that there is a growing similarity in how datapoints are represented in different neural network models. This similarity spans across different model architectures, training objectives, and even data modalities.

What has led to this convergence? Will it continue? And ultimately, where does it end?

Our central hypothesis, stated above in Figure 1 , is that there is indeed an endpoint to this convergence and a principle that drives it: different models are all trying to arrive at a representation of reality , meaning a representation of the joint distribution over events in the world that generate the data we observe. Figure 1 conveys this hypothesis: there exists a real world (labeled Z Z ), which we measure with various sensors, such as the camera shown to the left ( X X ). Other projections of these measurements, such as the textual description shown, can be produced from the first set of measurements or mediated by some other set of measurements, e.g. , touch or other camera views (dotted arrow from X X to Y Y ) 1 1 1 Touch could convey the shapes in this example but not the colors. This is an important limitation to our hypothesis that we discuss at several points in the paper: different sensors and views might capture different information, which may limit their potential to converge to identical representations. . Representation learning algorithms find vector embeddings that statistically model the various measurements and projections. The resulting vector embeddings are all derived from the underlying reality in Z Z and thereby become aligned. As models are trained on more data and for more tasks, they require representations that capture more and more information about Z Z , and hence alignment toward Z Z increases toward a convergent point as a function of scale.

We call this converged hypothetical representation the “platonic representation” in reference to Plato’s Allegory of the Cave ( Plato, c. 375 BC ) , and his idea of an ideal reality that underlies our sensations. The training data for our algorithms are shadows on the cave wall, yet, we hypothesize, models are recovering ever better representations of the actual world outside the cave. This idea is not unique to Plato; our hypothesis is also related to the notion of “convergent realism” ( Newton-Smith, 1981 ; Putnam, 1982 ; Doppelt, 2007 ; Hardin & Rosenberg, 1982 ) in the philosophy of science ( i.e. , that science is converging on truth), and to many arguments that have been put forth in the representation learning literature ( e.g. , Tian et al. (2020a) ; Zimmermann et al. (2021) ; Richens & Everitt (2024) ; Cao & Yamins (2024) ).

Also closely related to our hypothesis is the “Anna Karenina scenario” described by Bansal et al. (2021) , referring to the possibility that all well-performing neural nets represent the world in the same way. We discuss the evidence they give for this possibility in Section 2 2 2 2 Borrowed from Tolstoy (1877) , similar analogies have been made in other domains, such as the “Anna Karenina principle” popularized by Diamond (1998) to explain animal domestication. . The platonic representation hypothesis refers to the situation where we are in an Anna Karenina scenario and the “happy representation” that is converged upon is one that reflects a statistical model of the underlying reality. We discuss the potential nature of this statistical model in more detail in Section 4 .

## 2 Representations are converging

#### Preliminaries

We restrict our attention to representations that are vector embeddings . We characterize such a representation by the similarity structure it induces, referred to as its kernel. Kernels are commonly used to assess representations ( Kornblith et al., 2019 ; Klabunde et al., 2023 ) ; this can be justified by the fact that they capture the relative structures among data samples, which are also the learning signal for many machine learning algorithms ( Aronszajn, 1950 ; Smola & Schölkopf, 1998 ) . Following prior literature, we define representational alignment as a measure of the similarity of the similarity structures induced by two representations, i.e. , a similarity metric over kernels. We give the mathematical definition of these concepts below: • A representation is a function f : 𝒳 → ℝ n f\colon\mathcal{X}\rightarrow\mathbb{R}^{n} that assigns a feature vector to each input in some data domain 𝒳 \mathcal{X} .

• A kernel , K : 𝒳 × 𝒳 → ℝ K\colon\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R} , characterizes how a representation measures distance/similarity between datapoints. K ⁡ ( x i , x j ) = ⟨ f ⁡ ( x i ) , f ⁡ ( x j ) ⟩ K(x_{i},x_{j})=\langle f(x_{i}),f(x_{j})\rangle , where ⟨ ⋅ , ⋅ ⟩ \langle{{}\cdot{}},{{}\cdot{}}\rangle denotes inner product, x i , x j ∈ 𝒳 x_{i},x_{j}\in\mathcal{X} and K ∈ 𝒦 K\in\mathcal{K} .

• A kernel-alignment metric , m : 𝒦 × 𝒦 → ℝ m\colon\mathcal{K}\times\mathcal{K}\rightarrow\mathbb{R} , measures the similarity between two kernels, i.e. , how similar is the distance measure induced by one representation to the distance measure induced by another. Examples include Centered Kernel Distance (CKA) ( Kornblith et al., 2019 ) , SVCCA ( Raghu et al., 2017 ) , and nearest-neighbor metrics ( Klabunde et al., 2023 ) .

In our experiments, we use a mutual nearest-neighbor metric that measures the mean intersection of the k k -nearest neighbor sets induced by two kernels, K 1 K_{1} and K 2 K_{2} , normalized by k k . This metric is a variant of those proposed in Park et al. (2024) , Klabunde et al. (2023) and Oron et al. (2017) . See Appendix A for the exact definition and Appendix B for comparisons with alternative alignment metrics.

Next, we explore several ways in which representations are converging. First, we argue that different neural networks are converging to aligned representations. Then, we show that this continues to hold across modalities, where image embeddings in vision models align with text embeddings in language models.

### 2.1 Different models, with different architectures and objectives, can have aligned representations

One indication of representational convergence is the rising number of systems built on top of pre-trained foundation models. These models are becoming standard backbones across a growing spectrum of tasks. Their versatility across numerous applications implies a level of universality in the way they represent data.

While this trend implies convergence toward a relatively small set of foundation models, it does not imply that different foundation models will arrive at the same representation. Yet that is what has been observed by several recent papers.

Lenc & Vedaldi (2015) conducted one such study, in which they measured representational similarity through a technique called model stitching . Given two models, f f and g g , each composed of multiple layers ( f = f 1 ∘ ⋯ ∘ f n f=f_{1}\circ\cdots\circ f_{n} , g = g 1 ∘ ⋯ ∘ g m g=g_{1}\circ\cdots\circ g_{m} ), an intermediate representation from f f is integrated into g g via a learned affine stitching layer h h , resulting in a new stitched model F = f 1 ∘ ⋯ ∘ f k ∘ h ∘ g k + 1 ∘ ⋯ ∘ g m F=f_{1}\circ\cdots\circ f_{k}\circ h\circ g_{k+1}\circ\cdots\circ g_{m} . If F F has good performance, it indicates that f f and g g have compatible representations at layer k k , up to the transform h h .

In their study, Lenc & Vedaldi (2015) made two notable findings: (1) A vision model trained on ImageNet ( Russakovsky et al., 2015 ) can be aligned with a model trained on Places-365 ( Zhou et al., 2017 ) while maintaining good performance; (2) The early layers of these convolutional networks are more interchangeable than later layers. The first finding illustrates a level of data independence where distinct image datasets lead to similar representations. The second finding agrees with extensive research that oriented Gabor-like filters are common in both artificial and biological vision systems. This suggests a convergence to a similar initial layer of representation across various neural network architectures ( Olshausen & Field, 1996 ; Krizhevsky et al., 2017 ) . Bansal et al. (2021) expanded on the idea of model stitching, showing that models trained using self-supervised objectives align closely with their supervised counterparts.

Moschella et al. (2022) further demonstrated the feasibility of “zero-shot” model stitching without learning a stitching layer. Despite the fact that different text models were trained on different modalities, they found that the models often embed data in remarkably similar ways. In particular, they considered the kernel K K defined by learned representations and showed that K K serves as a bridge between models, allowing an encoder trained in one language, like English, to work effectively with a decoder in another, like French.

Dravid et al. (2023) extended this idea to individual neurons, and found “Rosetta Neurons” that are activated by the same pattern across a range of vision models. Such neurons form a common dictionary independently discovered by all models.

### 2.2 Alignment increases with scale and performance

Kornblith et al. (2019) and Roeder et al. (2021) observed model alignment not only exists but also increases with model scale and dataset size. On CIFAR-10 classification, Krizhevsky et al. (2009) found that larger models exhibit greater alignment with each other compared to smaller ones. Theoretically, Balestriero & Baraniuk (2018) showed that models with similar outputs ( e.g. , as a result of having high performance) also have similar internal activations. With the continuing trend of models scaling up, this suggests model alignment will increase over time – we might expect that the next generation of bigger, better models will be even more aligned with each other.

We expand upon this observation by evaluating the transfer performance of 78 78 vision models. These models were trained with varying architectures, training objectives, and datasets (detailed in Section C.1 ). In Figure 2 (left), we bin these models based on their average transfer performance on the VTAB dataset ( Zhai et al., 2019 ) , and then measure the average kernel alignment of the models within each bin. The results indicate that models with high transfer performance form a tightly clustered set of representations, while models with weak performance have more variable representations. We further visualize this structure with UMAP ( McInnes et al., 2018 ) over models representation in Figure 2 (right). This suggests that models that are competent all represent data in a similar way. Echoing Bansal et al. (2021) and Tolstoy (1877) , we might say: all strong models are alike, each weak model is weak in its own way.

The discussion so far indicates that various models are aligning toward a unified representation. But does the convergence extend to model weights? While models with different architectures might not have compatible weight spaces, there exists ample evidence that models with the same architecture will often converge to the same basin of weights ( Nagarajan & Kolter, 2019 ; Garipov et al., 2018 ; Lubana et al., 2023 ) . This holds even for models with different initializations, up to permutations over weight space ( Ainsworth et al., 2022 ) . Because of this, it is possible to merge separately trained models of the same architecture, and achieve some of the capabilities of all models in the mixture ( Stoica et al., 2023 ; Jordan et al., 2022 ; Wortsman et al., 2022 ) .

### 2.3 Representations are converging across modalities

Do models trained on different data modalities also converge? Several works indicate that the answer is yes .

Merullo et al. (2022) extended model stitching to the cross-modal setting, finding that a single linear projection is sufficient to stitch a vision model to an LLM and achieve good performance on visual question answering and image captioning. Koh et al. (2023) showed that linear stitching can also work in the opposite direction, aligning text inputs to visual outputs. In fact, many recent language-vision models stitch pre-trained language and vision models together. For example, LLaVA ( Liu et al., 2023 ) demonstrated state-of-the-art results by projecting visual features into a language model with a 2-layer MLP.

Other works show further kinds of evidence of cross-modal synergy. OpenAI (2023) found that jointly training a language model with a vision model improves performance on language tasks, compared to training the language model on its own. Sorscher et al. (2022) show a setting in which word embeddings of visual concept names can be isometrically mapped to image embeddings for those same concepts. In work concurrent to ours, Maniparambil et al. (2024) show well-trained vision encoders on large datasets exhibit high semantic similarity with language encoders regardless of the training paradigm (supervised, self-supervised, or language-supervised). Sharma et al. (2024) probed the visual knowledge of LLMs trained only on language data, by converting images into code that an LLM can process. They found that LLMs have rich knowledge of visual structures, to the extent that decent visual representations can be trained on images generated solely by querying an LLM to produce code and rendering the response. In visual generation, LLMs show abilities to augment captions with visual structures ( e.g. , bounding boxes) and improve generation quality ( Betker et al., 2023 ; Lian et al., 2023a ; Lian et al., 2023b ; Wu et al., 2023 ) . Over other modalities, Ngo & Kim (2024) showed auditory models are also roughly aligned with LLMs up to a linear transformation, and Ng et al. (2023) demonstrated the effectiveness of using pre-trained LLMs for facial motion prediction.