To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review — part 4/6

Cleaned-text code points [47201, 63091). Bibliography omitted here; source.md includes it.

Learnable Priors : Voloshynovskiy et al. (2020) also suggests using learnable priors as an alternative to handcrafted regularization priors on the latent representation. This method involves regularizing Z Z through another IB-based regularization with two components: (i) latent space regularization and (ii) observation space regularization. In this case, an additional hidden variable M M is introduced after the representation to regulate the information flow between Z Z and Y Y . An auto-encoder q ⁡ ( m | z ) q(m|z) is employed, and the optimization process aims to compress the information flowing from Z Z to M M while retaining only label-relevant information. The IB objective is defined as: ℒ \displaystyle\mathcal{L} = D K ​ L ( q ( m | z ) | | p ( m | z ) ) − β D K ​ L ( q ( x | m ) | | p ( x | m ) ) − β y D K ​ L ( p ( y | z ) | | p ( y ) ) \displaystyle=D_{KL}(q(m|z)||p(m|z))-\beta D_{KL}(q(x|m)||p(x|m))-\beta_{y}D_{KL}(p(y|z)||p(y)) (5) ⇔ I ⁡ ( M , Z ) − β ​ I ​ ( M , X ) − β y ​ I ​ ( Y , Z ) \displaystyle\Leftrightarrow I(M;Z)-\beta I(M;X)-\beta_{y}I(Y;Z)

Here, β \beta and β y \beta_{y} are hyperparameters that balance the trade-off between the relevance of M M to the labels and the compression of Z Z into M M .

Furthermore, Voloshynovskiy et al. (2020) demonstrated that various popular semi-supervised methods can be considered special cases of the optimization problem described above. Notably, the semi-supervised AAE ( Makhzani et al., 2015 ) , CatGAN ( Springenberg, 2015 ) , SeGMA ( Smieja et al., 2019 ) , and VAE ( Kingma et al., 2014 ) can all be viewed as specific instantiations of this framework.

#### 3.2.5 Unsupervised IB learning

In the unsupervised setting, data samples are not directly labeled by classes. Voloshynovskiy et al. (2020) defined unsupervised IB as a ’compressed’ parameterized mapping of X X to Z Z , which preserves some information in Z Z about X X through the reverse decoder X ¯ = Q ⁡ ( X | Z ) \bar{X}=Q(X|Z) . Therefore, the Lagrangian of unsupervised IB can be defined as follows: I P(X),P(Z—X) ​ ( X , Z ) − β ​ I P(Z),Q(X—Z) ​ ( Z , X ¯ ) \displaystyle I_{\lx@scalerel@obj{{P(X),P(Z|X})}}(X;Z)-\beta I_{\lx@scalerel@obj{{P(Z),Q(X|Z})}}(Z;\bar{X}) where I ⁡ ( X , Z ) I(X;Z) is the information determined by the encoder q ⁡ ( z | x ) q(z|x) and I ⁡ ( Z , X ¯ ) I(Z;\bar{X}) is the information determined by the decoder q ⁡ ( x | z ) q(x|z) , i.e., the reconstruction error. In other words, unsupervised IB is a special case of supervised IB, where labels are replaced with the reconstruction performance of the training input. Alemi et al. (2016) showed that Variational Autoencoder (VAE) ( Kingma and Welling, 2019 ) and β \beta -VAE ( Higgins et al., 2017 ) are special cases of unsupervised variational IB. Voloshynovskiy et al. (2020) extended their results and showed that many models, including adversarial autoencoders ( Makhzani et al., 2015 ) , InfoVAEs ( Zhao et al., 2017c ) , and VAE/GANs ( Larsen et al., 2016 ) , could be viewed as special cases of unsupervised IB. The main difference between them is the bounds on the different mutual information of the IB. Furthermore, unsupervised IB was used by Uğur et al. (2020) to derive lower bounds for their unsupervised generative clustering framework, while Roy et al. (2018) used it to study vector-quantized autoencoders.

Voloshynovskiy et al. (2020) pointed out that for the classification task in supervised IB, the latent space Z Z should be sufficient statistics for Y Y , whose entropy is much lower than X X . This results in a highly compressed representation where sequences close in the input space might be close in the latent space, and the less significant features will be compressed. In contrast, in the unsupervised setup, the IB suggests compressing the input to the encoded representation so that each input sequence can be decoded uniquely. In this case, the latent space’s entropy should correspond to the input space’s entropy, and compression is much more difficult.

## 4 Self-Supervised Multiview Information Bottleneck Learning

How can we learn without labels and still achieve good predictive power? Is compression necessary to obtain an optimal representation? This section analyzes and discusses how to achieve optimal representation for self-supervised learning when labels are not available during training. We review recent methods for self-supervised learning and show how they can be integrated into a single framework. We compare their objective functions, implicit assumptions, and theoretical challenges. Finally, we consider the information-theoretic properties of these representations, their optimality, and different ways of learning them.

One approach to enhance deep learning methods is to apply the InfoMax principle in a multiview setting ( Linsker, 1988 ; Wiskott and Sejnowski, 2002 ) . As one of the earliest approaches, Linsker (1988) proposed maximizing information transfer from input data to its latent representation, showing its equivalence to maximizing the determinant of the output covariance under the Gaussian distribution assumption. Becker and Hinton (1992) introduced a representation learning approach based on maximizing an approximation of the mutual information between alternative latent vectors obtained from the same image. The most well-known application is the Independent Component Analysis (ICA) Infomax algorithm ( Bell and Sejnowski, 1995 ) , designed to separate independent sources from their linear combinations. The ICA-Infomax algorithm aims to maximize the mutual information between mixtures and source estimates while imposing statistical independence among outputs. The Deep Infomax approach ( Hjelm et al., 2018 ) extends this idea to unsupervised feature learning by maximizing the mutual information between input and output while matching a prior distribution for the representations. Recent work has applied this principle to a self-supervised multiview setting ( Hjelm et al., 2018 ; Henaff, 2020 ; Bachman et al., 2019 ; Tian et al., 2020a ) , wherein these works maximize the mutual information between the views Z 1 Z_{1} and Z 2 Z_{2} using the classifier q ⁡ ( z 1 | z 2 ) q(z_{1}|z_{2}) , which attempts to predict one representation from the other.

However, Tschannen et al. (2019) demonstrated that the effectiveness of InfoMax models is more attributable to the inductive biases introduced by the architecture and estimators than to the training objectives themselves, as the InfoMax objectives can be trivially maximized using invertible encoders. Moreover, a fundamental issue with the InfoMax principle is that it retains irrelevant information about the labels, contradicting the core concept of the IB principle, which advocates compressing the representation to enhance generalizability.

To resolve this problem, Sridharan and Kakade (2008) proposed the multiview IB framework . According to this framework, in the multiview without labels setting, the IB principle of preserving relevant data while compressing irrelevant data requires assumptions regarding the relationship between views and labels. They presented the MultiView assumption , which asserts that either view (approximately) would be sufficient for downstream tasks. By this assumption, they define the relevant information as the shared information between the views. Therefore, augmentations (such as changing the image style) should not affect the labels.

Additionally, the views will provide most of the information in the input regarding downstream tasks. We improve generalization without affecting performance by compressing the information not shared between the two views. Their formulation is as follows:

###### Assumption 1 .

The MultiView Assumption: There exists a ϵ info \epsilon_{\text{info}} (which is assumed to be small) such that I ⁡ ( Y ; X 2 | X 1 ) \displaystyle I(Y;X_{2}|X_{1}) ≤ ϵ info , \displaystyle\leq\epsilon_{\text{info}}, I ⁡ ( Y ; X 1 | X 2 ) \displaystyle I(Y;X_{1}|X_{2}) ≤ ϵ info . \displaystyle\leq\epsilon_{\text{info}}.

As a result, when the information sharing parameter, ϵ info \epsilon_{\text{info}} , is small, the information shared between views includes task-relevant details. For instance, in self-supervised contrastive learning for visual data ( Hjelm et al., 2018 ) , views represent various augmentations of the same image. In this scenario, the MultiView assumption is considered mild if the downstream task remains unaffected by the augmentation ( Geiping et al., 2022 ) . Image augmentations can be perceived as altering an image’s style without changing its content. Thus, Tsai et al. (2020) contends that the information required for downstream tasks should be preserved in the content rather than the style. This assumption allows us to separate the information into relevant (shared information) and irrelevant (not shared) components and to compress only the unimportant details that do not contain information about downstream tasks. Based on this assumption, we aim to maximize the relevant information I ⁡ ( X 2 , Z 1 ) I(X_{2};Z_{1}) and minimize I ⁡ ( X 1 ; Z 1 ∣ X 2 ) I(X_{1};Z_{1}\mid X_{2}) - the exclusive information that Z 1 Z_{1} contains about X 1 X_{1} , which cannot be predicted by observing X 2 X_{2} . This irrelevant information is unnecessary for the prediction task and can be discarded. In the extreme case, where X 1 X_{1} and X 2 X_{2} share only label information, this approach recovers the supervised IB method without labels. Conversely, if X 1 X_{1} and X 2 X_{2} are identical, this method collapses into the InfoMax principle, as no information can be accurately discarded.

Federici et al. (2020) used the relaxed Lagrangian objective to obtain the minimal sufficient representation Z 1 Z_{1} for X 2 X_{2} as: ℒ 1 = I P(Z_1—X_1) ​ ( Z 1 ; X 1 ∣ X 2 ) − β 1 ​ I P(Z_2—X_2),Q(Z_1—Z_2) ​ ( X 2 , Z 1 ) \displaystyle\mathcal{L}_{1}=I_{\lx@scalerel@obj{{P(Z_1|X_1})}}(Z_{1};X_{1}\mid X_{2})-\beta_{1}I_{\lx@scalerel@obj{{P(Z_2|X_2),Q(Z_1|Z_2})}}(X_{2};Z_{1}) and the symmetric loss to obtain the minimal sufficient representation Z 2 Z_{2} for X 1 X_{1} : ℒ 2 = I P(Z_2—X_2) ​ ( Z 2 ; X 2 ∣ X 1 ) − β 2 ​ I P(Z_1—X_1),Q(Z_2—Z_1) ​ I ​ ( X 1 , Z 2 ) \displaystyle\mathcal{L}_{2}=I_{\lx@scalerel@obj{{P(Z_2|X_2})}}(Z_{2};X_{2}\mid X_{1})-\beta_{2}I_{\lx@scalerel@obj{{P(Z_1|X_1),Q(Z_2|Z_1})}}I(X_{1};Z_{2}) where β 1 \beta_{1} and β 2 \beta_{2} are the Lagrangian multipliers introduced by the constraint optimization. By defining Z 1 Z_{1} and Z 2 Z_{2} on the same domain and re-parameterizing the Lagrangian multipliers, the average of the two loss functions can be upper bounded as: ℒ = − I P(Z_1—X_1),Q(Z_2—Z_1) ( Z 1 ; Z 2 ) + β D SKL [ p ( z 1 ∣ x 1 ) | | P ( z 2 ∣ x 2 ) ] \displaystyle\mathcal{L}=-I_{\lx@scalerel@obj{{P(Z_1|X_1),Q(Z_2|Z_1})}}(Z_{1};Z_{2})+\beta D_{\text{SKL}}[p(z_{1}\mid x_{1})||P(z_{2}\mid x_{2})] where D SKL D_{\text{SKL}} represents the symmetrized K ​ L KL divergence obtained by averaging the expected value of D KL ( p ( z 1 ∣ x 1 ) | | p ( z 2 ∣ x 2 ) ) D_{\text{KL}}(p(z_{1}\mid x_{1})||p(z_{2}\mid x_{2})) and D KL ( p ( z 2 ∣ x 2 ) | | p ( z 1 ∣ x 1 ) ) D_{\text{KL}}(p(z_{2}\mid x_{2})||p(z_{1}\mid x_{1})) . Note that when the mapping from X 1 X_{1} to Z 1 Z_{1} is deterministic, I ⁡ ( Z 1 ; X 1 ∣ X 2 ) I(Z_{1};X_{1}\mid X_{2}) minimization and H ⁡ ( Z 1 ∣ X 2 ) H(Z_{1}\mid X_{2}) minimization are interchangeable and the algorithms of Federici et al. (2020) and Tsai et al. (2020) minimize the same objective. Another implementation of the same idea is based on the Conditional Entropy Bottleneck (CEB) algorithm ( Fischer, 2020 ) and proposed by Lee et al. (2021b) . This algorithm adds the residual information as a compression term to the InfoMax objective using the reverse decoders q ⁡ ( z 1 ∣ x 2 ) q(z_{1}\mid x_{2}) and q ⁡ ( z 2 ∣ x 1 ) q(z_{2}\mid x_{1}) .

In conclusion, all the algorithms mentioned above are based on the Multiview assumption. Utilizing this assumption, they can distinguish relevant information from irrelevant information. As a result, all these algorithms aim to maximize the information (or the predictive ability) of one representation with respect to the other view while compressing the information between each representation and its corresponding view. The key differences between these algorithms lie in the decomposition and implementation of these information terms.

Dubois et al. (2021) offers another theoretical analysis of the IB for self-supervised learning. Their work addresses the question of the minimum bit rate required to store the input but still achieve high performance on a family of downstream tasks Y ∈ 𝒴 Y\in\mathcal{Y} . It is a rate-distortion problem, where the goal is to find a compressed representation that will give us a good prediction for every task. We require that the distortion measure is bounded: D 𝒯 ​ ( X , Z ) = sup Y ∈ 𝒴 H ⁡ ( Y ∣ Z 1 ) − H ⁡ ( Y ∣ X 1 ) ≤ δ . D_{\mathcal{T}}(X,Z)=\sup_{Y\in\mathcal{Y}}H(Y\mid Z_{1})-H(Y\mid X_{1})\leq\delta.

Accessing the downstream task is necessary to find the solution during the learning process. As a result, Dubois et al. (2021) considered only tasks invariant to some equivalence relation, which divides the input into disjoint equivalence classes. An example would be an image with labels that remain unchanged after augmentation. This is similar to the Multiview assumption where ϵ i ​ n ​ f ​ o → 0 \epsilon_{info}\to 0 . By applying Shannon’s rate-distortion theory, they concluded that the minimum achievable bit rate is the rate-distortion function with the above invariance distortion. Thus, the optimal rate can be determined by minimizing the following Lagrangian:

ℒ = min P ⁡ ( Z 1 ∣ X 1 ) ⁡ I P(Z_1—X_1) ​ ( X 1 , Z 1 ) + β ​ H ​ ( Z 2 ∣ X 1 ) . \displaystyle\mathcal{L}=\min_{P(Z_{1}\mid X_{1})}I_{\lx@scalerel@obj{P(Z_1|X_1)}}(X_{1};Z_{1})+\beta H(Z_{2}\mid X_{1}). (6)

Using this objective, the maximization of information with labels is replaced by maximizing the prediction ability of one view from the original input, regularized by direct information from the input. Similarly to the above results, we would like to find a representation Z 1 Z_{1} that compresses the input X 1 X_{1} so that Z 1 Z_{1} has the maximum information about X 2 X_{2} .

### 4.1 Implicit Compression in Self-Supervised Learning Methods

While the optimal IB representation is based on the Multiview assumption, most self-supervised learning models only use the infoMax principle and maximize the mutual information I ⁡ ( Z 1 , Z 2 ) I(Z_{1};Z_{2}) without an explicit regularization term. However, recent studies have shown that contrastive learning creates compressed representations that include only relevant information ( Wang et al., 2022 ; Tian et al., 2020b ) . The question is, why is the learned representation compressed? The maximization of I ⁡ ( Z 1 , Z 2 ) I(Z_{1};Z_{2}) could theoretically be sufficient to retain all the information from both X 1 X_{1} and X 2 X_{2} by making the representations invertible. In this section, we attempt to explain this phenomenon.

We begin with the InfoMax principle ( Linsker, 1988 ) , which maximizes the mutual information between the representations of random variables Z 1 Z^{1} and Z 2 Z^{2} of the two views. We can lower-bound it using:

I ⁡ ( Z 1 , Z 2 ) = H ⁡ ( Z ) − H ⁡ ( Z 1 ∣ Z 2 ) ≥ H ⁡ ( Z 1 ) + 𝔼 ⁡ [ log ⁡ q ⁡ ( z 1 ∣ z 2 ) ] \displaystyle I(Z_{1};Z_{2})=H(Z)-H(Z_{1}\mid Z_{2})\geq H(Z_{1})+\bm{\mathbb{E}}[\log q(z_{1}\mid z_{2})] (7)

The bound is tight when q ⁡ ( z 1 | z 2 ) = p ⁡ ( z 1 | z 2 ) q(z_{1}|z_{2})=p(z_{1}|z_{2}) , in which case the first term equals the conditional entropy H ⁡ ( Z 1 | Z 2 ) H(Z_{1}|Z_{2}) . The second term of eq. 7 can be considered a negative reconstruction error or distortion between Z 1 Z_{1} and Z 2 Z_{2} .