The Platonic Representation Hypothesis — part 6/6
Cleaned-text code points [77152, 87178). Bibliography omitted here; source.md includes it.
system: You are given a full-text description of an image. You should summarize it into about <num_words> words, being sure to include as much salient visual information as possible given the <num_words> word constraint, especially information from the start of the original description. The new description should apply for the original image. Respond with only the summary, in one line.
user: <original_caption>
We measure the alignment with this generated caption to test our hypothesis that denser captations would result in higher alignment scores. In Figure 9 , we find that the alignment score also improves as caption length increases.
## Appendix F Analysis of Contrastive Learners
### F.1 Contrastive objectives learn pointwise mutual information
There are two widely used forms of contrastive objectives. We now discuss each form in detail and show how they both are minimized by the pointwise mutual information (PMI) as stated in Equation 5 . To simplify notation, we consider learning the bivariate model g ( x a , x b ) ∈ ℝ g(x_{a},x_{b})\in\mathbb{R} . In Section 4 , such g g is optimized within the family of { g = ⟨ f X , f X ⟩ : f X ∈ ℱ X } \{g=\langle f_{X},f_{X}\rangle\colon f_{X}\in\mathcal{F}_{X}\} .
Recall that our positive pairs are sampled from ( x , x + ) ∼ P 𝖼𝗈𝗈𝗋 (x,x_{+})\sim{P}_{\mathsf{coor}} , and that the negative pairs are sampled independently from its marginals which we denote as ( x , x − ) ∼ i.i.d. P (x,x_{-})\overset{\text{i.i.d.}}{\sim}P where P ( x ) = ∑ x + P 𝖼𝗈𝗈𝗋 ( x , x + ) P(x)=\sum_{x_{+}}{P}_{\mathsf{coor}}(x,x_{+}) . 1. The binary NCE loss ( Gutmann & Hyvärinen, 2010 ) is defined with a certain prior over sampling positive vs. negative pairs. Let p 𝗉𝗈𝗌 p_{\mathsf{pos}} be the probability of sampling a positive pair. Then the loss is given by ℒ 𝖻𝗂𝗇𝖺𝗋𝗒 - 𝖭𝖢𝖤 ( g ) ≜ p 𝗉𝗈𝗌 ⋅ 𝔼 ( x , x + ) ∼ P 𝖼𝗈𝗈𝗋 [ − log σ ( g ( x , x + ) ) ] + ( 1 − p 𝗉𝗈𝗌 ) ⋅ 𝔼 ( x , x − ) ∼ i.i.d. P [ − log σ ( − g ( x , x − ) ) ] . \mathcal{L}_{\mathsf{binary\mbox{-}NCE}}(g)\triangleq p_{\mathsf{pos}}\cdot\mathbb{E}_{(x,x_{+})\sim{P}_{\mathsf{coor}}}\left[-\log\sigma(g(x,x_{+}))\right]+(1-p_{\mathsf{pos}})\cdot\mathbb{E}_{(x,x_{-})\overset{\text{i.i.d.}}{\sim}P}\left[-\log\sigma(-g(x,x_{-}))\right]. (20)
The Bayes optimal solution is given by g ( x a , x b ) \displaystyle g(x_{a},x_{b}) = log P ( pos | x a , x b ) 1 − P ( pos | x a , x b ) \displaystyle=\log\frac{P(\texttt{pos}\mathrel{|}x_{a},x_{b})}{1-P(\texttt{pos}\mathrel{|}x_{a},x_{b})} (21) = log P ( pos , x a , x b ) P ( neg , x a , x b ) \displaystyle=\log\frac{P(\texttt{pos},x_{a},x_{b})}{P(\texttt{neg},x_{a},x_{b})} (22) = log p 𝗉𝗈𝗌 ⋅ P 𝖼𝗈𝗈𝗋 ( x a , x b ) ( 1 − p 𝗉𝗈𝗌 ) P ( x a ) P ( x b ) \displaystyle=\log\frac{p_{\mathsf{pos}}\cdot{P}_{\mathsf{coor}}(x_{a},x_{b})}{(1-p_{\mathsf{pos}})P(x_{a})P(x_{b})} (23) = log P 𝖼𝗈𝗈𝗋 ( x a , x b ) P ( x a ) P ( x b ) + log p 𝗉𝗈𝗌 1 − p 𝗉𝗈𝗌 \displaystyle=\log\frac{{P}_{\mathsf{coor}}(x_{a},x_{b})}{P(x_{a})P(x_{b})}+\log\frac{p_{\mathsf{pos}}}{1-p_{\mathsf{pos}}} (24) = K 𝖯𝖬𝖨 ( x a , x b ) + c X . \displaystyle=K_{\mathsf{PMI}}(x_{a},x_{b})+c_{X}. (25)
2. The InfoNCE loss ( Oord et al., 2018 ) is defined with randomly sampling one positive pair along with K K negative ones. With some hyperparameter τ > 0 \tau>0 , the loss is given by ℒ 𝖨𝗇𝖿𝗈𝖭𝖢𝖤 ( g ) ≜ 𝔼 ( x , x + ) ∼ P 𝖼𝗈𝗈𝗋 ( x − ( 1 ) , x − ( 2 ) , … , x − ( K ) ) ∼ i.i.d. P [ − log e g ( x , x + ) / τ e g ( x , x + ) / τ + ∑ i = 1 K e g ( x , x − ( i ) ) / τ ] . \mathcal{L}_{\mathsf{InfoNCE}}(g)\triangleq\mathbb{E}_{\begin{subarray}{c}(x,x_{+})\sim{P}_{\mathsf{coor}}\\ (x_{-}^{(1)},x_{-}^{(2)},\dots,x_{-}^{(K)})\overset{\text{i.i.d.}}{\sim}P\end{subarray}}\left[-\log\frac{e^{g(x,x_{+})/\tau}}{e^{g(x,x_{+})/\tau}+\sum_{i=1}^{K}e^{g(x,x_{-}^{(i)})/\tau}}\right]. (26) The Bayes optimal solution is given by e g ( x , x + ) / τ e g ( x , x + ) / τ + ∑ i = 1 K e g ( x , x − ( i ) ) / τ \displaystyle\frac{e^{g(x,x_{+})/\tau}}{e^{g(x,x_{+})/\tau}+\sum_{i=1}^{K}e^{g(x,x_{-}^{(i)})/\tau}} = P 𝖼𝗈𝗈𝗋 ( x + | x ) ∏ j P ( x − ( j ) ) P 𝖼𝗈𝗈𝗋 ( x + | x ) ∏ j P ( x − ( j ) ) + ∑ i P 𝖼𝗈𝗈𝗋 ( x − ( i ) | x ) P ( x + ) ∏ j ≠ i P ( x − ( j ) ) \displaystyle=\frac{{P}_{\mathsf{coor}}(x_{+}\mathrel{|}x)\prod_{j}P(x_{-}^{(j)})}{{P}_{\mathsf{coor}}(x_{+}\mathrel{|}x)\prod_{j}P(x_{-}^{(j)})+\sum_{i}{P}_{\mathsf{coor}}(x_{-}^{(i)}\mathrel{|}x)P(x_{+})\prod_{j\neq i}P(x_{-}^{(j)})} (27) = P 𝖼𝗈𝗈𝗋 ( x + | x ) / P ( x + ) P 𝖼𝗈𝗈𝗋 ( x + | x ) / P ( x + ) + ∑ i P 𝖼𝗈𝗈𝗋 ( x − ( i ) | x ) / P ( x − ( i ) ) . \displaystyle=\frac{{P}_{\mathsf{coor}}(x_{+}\mathrel{|}x)/P(x_{+})}{{P}_{\mathsf{coor}}(x_{+}\mathrel{|}x)/P(x_{+})+\sum_{i}{P}_{\mathsf{coor}}(x_{-}^{(i)}\mathrel{|}x)/P(x_{-}^{(i)})}. (28) For τ = 1 \tau=1 , this optima corresponds to g g choices where g ( x a , x b ) \displaystyle g(x_{a},x_{b}) = log P 𝖼𝗈𝗈𝗋 ( x b | x a ) P ( x b ) + c X ( x a ) \displaystyle=\log\frac{{P}_{\mathsf{coor}}(x_{b}\mathrel{|}x_{a})}{P(x_{b})}+c_{X}(x_{a}) (29) = K 𝖯𝖬𝖨 ( x a , x b ) + c X ( x a ) . \displaystyle=K_{\mathsf{PMI}}(x_{a},x_{b})+c_{X}(x_{a}). (30) For the general τ ≠ 1 \tau\neq 1 case, we have g g (and corresponding f X f_{X} ) recovers K 𝖯𝖬𝖨 K_{\mathsf{PMI}} up to an offset and a scale. Our main argument in Section 4 that f X f_{X} recovers K 𝖯𝖬𝖨 K_{\mathsf{PMI}} still holds.
### F.2 Contrastive learners can represent K 𝖯𝖬𝖨 K_{\mathsf{PMI}} exactly under smoothness conditions
We want to express K 𝖯𝖬𝖨 + C K_{\mathsf{PMI}}+C using some representation function f X : 𝒳 → ℝ n f_{X}\colon\mathcal{X}\rightarrow\mathbb{R}^{n} so that ⟨ f X ( x a ) , f X ( x b ) ⟩ = K 𝖯𝖬𝖨 ( x a , x b ) + C , for some C . \langle f_{X}(x_{a}),f_{X}(x_{b})\rangle=K_{\mathsf{PMI}}(x_{a},x_{b})+C,\qquad\text{for some $C$.} (31) For such an f X f_{X} to exist, an equivalent criterion is that K 𝖯𝖬𝖨 + C K_{\mathsf{PMI}}+C is positive semi-definite (PSD), as can be seen from eigendecomposition.
###### Proposition F.1 .
Suppose that the off-diagonal elements of K 𝖯𝖬𝖨 K_{\mathsf{PMI}} are bounded within [ log ρ 𝗆𝗂𝗇 , log ρ 𝗆𝗂𝗇 + δ ] ∈ ( − ∞ , 0 ] [\log\rho_{\mathsf{min}},\log\rho_{\mathsf{min}}+\delta]\in(-\infty,0] . We have K 𝖯𝖬𝖨 + C K_{\mathsf{PMI}}+C is positive semi-definite (PSD) for some C C if the joint distribution is sufficiently smooth: P 𝖼𝗈𝗈𝗋 ( z i | z i ) P 𝖼𝗈𝗈𝗋 ( z i ) ≥ e N δ ρ 𝗆𝗂𝗇 , ∀ i . \frac{{P}_{\mathsf{coor}}(z_{i}\mathrel{|}z_{i})}{{P}_{\mathsf{coor}}(z_{i})}\geq e^{N\delta}\rho_{\mathsf{min}}\mathrlap{,\qquad\forall i.} (32)
###### Proof.
Note that K 𝖯𝖬𝖨 + C K_{\mathsf{PMI}}+C still only has non-positive off-diagonal elements if − C ≥ log ρ 𝗆𝗂𝗇 + δ . -C\geq\log\rho_{\mathsf{min}}+\delta. (33) For such C C , it is diagonally dominant (and thus PSD) if, ∀ i , K 𝖯𝖬𝖨 ( z i , z i ) + C ≥ ∑ j ≠ i | K 𝖯𝖬𝖨 ( z i , z j ) + C | = − ( N − 1 ) C − ∑ j ≠ i K 𝖯𝖬𝖨 ( z i , z j ) , \mathllap{\forall i,\qquad}K_{\mathsf{PMI}}(z_{i},z_{i})+C\geq\sum_{j\neq i}\left\lvert K_{\mathsf{PMI}}(z_{i},z_{j})+C\right\rvert=-(N-1)C-\sum_{j\neq i}K_{\mathsf{PMI}}(z_{i},z_{j}), (34) or equivalently, ∀ i , N C + ∑ j K 𝖯𝖬𝖨 ( z i , z j ) ≥ 0 . \mathllap{\forall i,\qquad}NC+\sum_{j}K_{\mathsf{PMI}}(z_{i},z_{j})\geq 0. (35)
The following choice of C C readily satisfies the above Equation 35 : C ≜ − min i 1 N ∑ j K 𝖯𝖬𝖨 ( z i , z j ) . C\triangleq-\min_{i}\frac{1}{N}\sum_{j}K_{\mathsf{PMI}}(z_{i},z_{j}). (36)
Therefore, it remains to show that Equation 33 is true. Note that − C ≜ min i 1 N ∑ j K 𝖯𝖬𝖨 ( z i , z j ) ≥ N − 1 N log ρ 𝗆𝗂𝗇 + 1 N ( min i K 𝖯𝖬𝖨 ( z i , z i ) ) . -C\triangleq\min_{i}\frac{1}{N}\sum_{j}K_{\mathsf{PMI}}(z_{i},z_{j})\geq\frac{N-1}{N}\log\rho_{\mathsf{min}}+\frac{1}{N}(\min_{i}K_{\mathsf{PMI}}(z_{i},z_{i})). (37)
Therefore, it suffices to have log ρ 𝗆𝗂𝗇 + δ ≤ N − 1 N log ρ 𝗆𝗂𝗇 + 1 N ( min i K 𝖯𝖬𝖨 ( z i , z i ) ) . \log\rho_{\mathsf{min}}+\delta\leq\frac{N-1}{N}\log\rho_{\mathsf{min}}+\frac{1}{N}(\min_{i}K_{\mathsf{PMI}}(z_{i},z_{i})). (38) Rearranging terms gives the desired condition P 𝖼𝗈𝗈𝗋 ( z i | z i ) P 𝖼𝗈𝗈𝗋 ( z i ) ≥ e N δ ρ 𝗆𝗂𝗇 , ∀ i . \frac{{P}_{\mathsf{coor}}(z_{i}\mathrel{|}z_{i})}{{P}_{\mathsf{coor}}(z_{i})}\geq e^{N\delta}\rho_{\mathsf{min}}\mathrlap{,\qquad\forall i.} (39) ∎
###### Remark F.2 .
Proposition F.1 is one example that a sufficiently smooth world or a sufficiently high sampling rate allows the PMI kernel K 𝖯𝖬𝖨 K_{\mathsf{PMI}} to be exactly represented as inner products of a learned feature space (up to a scale). The condition here can be satisfied, for example, if the off-diagonal terms decay linearly with respect to N N and stay sufficiently close to each other. While the condition is somewhat strict, it captures the essence that smoothness and continuity allow easier learning. Nonetheless, we note that exact representation is not necessary for convergence, and thus this requirement can likely be relaxed. Please see Section 6 for discussions on practical settings.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .