To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review — part 3/6

Cleaned-text code points [31210, 47201). Bibliography omitted here; source.md includes it.

Achille and Soatto (2018) explored how applying the IB objective to the network’s parameters may reduce overfitting while maintaining invariant representations. Their work showed that flat minima, which have better generalization properties, bound the information with the weights, and the information in the weights bound the information in the activations. Chelombiev et al. (2019) found that the generalization precision is positively correlated with the degree of compression of the last layer in the network. Shwartz-Ziv et al. (2018) showed that the generalization error depends exponentially on the mutual information between the model and the input once it is smaller than log ⁡ 2 ​ n \log 2n - the query sample complexity. Moreover, they demonstrated that M M bits of compression of X X are equivalent to an exponential factor of 2 M 2^{M} training examples. Piran et al. (2020) extended the original IB to the dual form, which offers several advantages in terms of compression.

These studies illustrate that the IB leads to a trade-off between prediction and complexity, even for the empirical distribution. With the IB objective, we can design estimators to find optimal solutions for different regimes with varying performance, complexity, and generalization.

## 3 Information-Theoretic Objectives

Before delving into the details, this section aims to provide an overview of the information-theoretic objectives in various learning scenarios, including supervised, unsupervised, and self-supervised settings. We will also introduce a general framework to understand better the process of learning optimal representations and explore recent methods working towards this goal.

Developing a novel algorithm entails numerous aspects, such as architecture, initialization parameters, learning algorithms, and pre-processing techniques. A crucial element, however, is the objective function. As demonstrated in Section 2.4.2 , the IB approach, originally introduced by Tishby et al. (1999b) , defines the optimal representation in supervised scenarios, enabling us to identify which terms to compress during learning. However, determining the optimal representation and deriving information-based objective functions in self-supervised settings are more challenging. In this section, we introduce a general framework to understand the process of learning optimal representations and explore recent methods striving to achieve this goal.

### 3.1 Setup and Methodology

Using a two-channel input allows us to model complex multiview learning problems. In many real-world situations, data can be observed from multiple perspectives or modalities, making it essential to develop learning algorithms capable of handling such multiview data.

Consider a two-channel input, X 1 X_{1} and X 2 X_{2} , and a single-channel label Y Y for a downstream task, all possessing a joint distribution P ⁡ ( X 1 , X 2 , Y ) P(X_{1},X_{2},Y) . We assume the availability of n n labeled examples S = ( x 1 i , x 2 i , y i ) i = 1 n S={(x^{i}_{1},x^{i}_{2},y^{i})}^{n}_{i=1} and t t unlabeled examples U = ( x 1 i , x 2 i ) i = n + 1 n + t U={(x^{i}_{1},x^{i}_{2})}^{n+t}_{i=n+1} , both independently and identically distributed. Our objective is to predict Y Y using a loss function.

In our model, we use a learned encoder with a prior P ⁡ ( Z ) P(Z) to generate a conditional representation (which may be deterministic or stochastic) Z i | X i = P θ i ​ ( Z i | X i ) Z_{i}|X_{i}=P_{\theta_{i}}(Z_{i}|X_{i}) , where i = 1 , 2 i=1,2 represents the two views. Subsequently, we utilize various decoders to ’decode’ distinct aspects of the representation:

For the supervised scenario, we have a joint embedding of the label classifiers from both views, Y ^ 1 , 2 = Q ρ ​ ( Y | Z 1 , Z 2 ) \hat{Y}_{1,2}=Q_{\rho}(Y|Z_{1},Z_{2}) , and two decoders predicting the labels of the downstream task based on each individual view, Y i ^ = Q ρ i ​ ( Y | Z i ) \hat{Y_{i}}=Q_{\rho_{i}}(Y|Z_{i}) for i = 1 , 2 i=1,2 .

For the unsupervised case, we have direct decoders for input reconstruction from the representation, X i ¯ = Q ψ i ​ ( X i | Z i ) \bar{X_{i}}=Q_{\psi_{i}}(X_{i}|Z_{i}) for i = 1 , 2 i=1,2 .

For self-supervised learning, we utilize two cross-decoders attempting to predict one representation based on the other, Z 1 ~ | Z 2 = q η 1 ​ ( Z 1 | Z 2 ) \tilde{Z_{1}}|Z_{2}=q_{\eta_{1}}(Z_{1}|Z_{2}) and Z 2 ~ | Z 1 = q η 2 ​ ( Z 2 | Z 1 ) \tilde{Z_{2}}|Z_{1}=q_{\eta_{2}}(Z_{2}|Z_{1}) . Figure 1 illustrates this structure.

The information-theoretic perspective of self-supervised networks has led to confusion regarding the information being optimized in recent work. In supervised and unsupervised learning, only one ’information path’ exists when optimizing information-theoretic terms: the input is encoded through the network, and then the representation is decoded and compared to the targets. As a result, the representation and corresponding information always stem from a single encoder and decoder.

However, in the self-supervised multiview scenario, we can construct our representation using various encoders and decoders. For instance, we need to specify the associated random variable to define the information involved in I ⁡ ( X 1 , Z 1 ) I(X_{1};Z_{1}) . This variable could either be based on the encoder of X 1 X_{1} - P θ 1 ​ ( Z 1 | X 1 ) P_{\theta_{1}}(Z_{1}|X_{1}) , or based on the encoder of X 2 X_{2} - P θ 2 ​ ( Z 2 | X 2 ) P_{\theta_{2}}(Z_{2}|X_{2}) , which is subsequently passed to the cross-decoder Q η 1 ​ ( Z 1 | Z 2 ) Q_{\eta_{1}}(Z_{1}|Z_{2}) and then to the direct decoder Q ψ 1 ​ ( X 1 | Z 1 ) Q_{\psi_{1}}(X_{1}|Z_{1}) .

To fully understand the information terms, we aim to optimize and distinguish between various ”information paths,” we marked each information path differently. For example, I , P ( X 1 ) , P ( Z 1 | X 1 ) , P ( Z 2 | Z 1 ) ( X 1 , Z 2 ) I_{,P(X_{1}),P(Z_{1}|X_{1}),P(Z_{2}|Z_{1})}\left(X_{1},Z_{2}\right) is based on the path P ⁡ ( X 1 ) → P ⁡ ( Z 1 | X 1 ) → P ⁡ ( Z 2 | Z 1 ) P(X_{1})\to P(Z_{1}|X_{1})\to P(Z_{2}|Z_{1}) . In the following section, we will ”translate” previous work into our present framework and examine the loss function.

### 3.2 Optimization with Labels

After establishing our framework, we can now incorporate various learning algorithms. We begin by examining classical single-view supervised information bottleneck algorithms for deep networks that utilize labeled data during training and extend them to the multiview scenario. Next, we broaden our perspective to include unsupervised learning, where input reconstruction replaces labels, and semi-supervised learning, where information-based regularization is applied to improve predictions.

#### 3.2.1 Single-View Supervised Learning

In classical single-view supervised learning, the task of representation learning involves finding a distribution p ⁡ ( z | x ) p(z|x) that maps data observations x ∈ 𝒳 x\in\mathcal{X} to a representation z ∈ 𝒵 z\in\mathcal{Z} , capturing only the relevant features of the input Shwartz-Ziv (2022) . The goal is to predict a label y ∈ 𝒴 y\in\mathcal{Y} using the learned representation. Achille and Soatto (2018) defined the sufficiency of Z Z for Y Y as the amount of label information retained after passing data through the encoder:

###### Definition 4 .

Sufficiency : A representation Z Z of X X is sufficient for Y Y if and only if I ⁡ ( X ; Y | Z ) = 0 I(X;Y|Z)=0 .

Federici et al. (2020) showed that Z Z is sufficient for Y Y if and only if the amount of information regarding the task remains unchanged by the encoding procedure. A sufficient representation can predict Y Y as accurately as the original data X X . In Section 2.4 , we saw a trade-off between prediction and generalization when there is a finite amount of data. To reduce the generalization gap, we aim to compress X X while retaining as much predicate information on the labels as possible. Thus, we relax the sufficiency definition and minimize the following objective:

ℒ = I ⁡ ( X , Z ) − β ​ I ​ ( Z , Y ) \displaystyle\mathcal{L}=I(X;Z)-\beta I(Z;Y) (3)

The mutual information I ⁡ ( Y , Z ) I(Y;Z) determines how much label information is accessible and reflects the model’s ability to predict performance on the target task. I ⁡ ( X , Z ) I(X;Z) represents the information that Z Z carries about the input, which we aim to compress. However, I ⁡ ( X , Z ) I(X;Z) contains both relevant and irrelevant information about Y Y . Therefore, using the chain rule of information, Federici et al. (2020) proposed splitting I ⁡ ( X , Z ) I(X,Z) into two terms: I ⁡ ( X , Z ) = I ⁡ ( X ; Z | Y ) ⏟ superfluous information + I ⁡ ( Z , Y ) ⏟ predictive information \displaystyle I(X;Z)=\underbrace{I(X;Z|Y)}_{\text{superfluous information}}+\underbrace{I(Z;Y)}_{\text{predictive information}} (4) The conditional information I ⁡ ( X , Z | Y ) I(X,Z|Y) represents information in Z Z that is not predictive of Y Y , i.e., superfluous information. The decomposition of input information enables us to compress only irrelevant information while preserving the relevant information for predicting Y Y . Several methods are available for evaluating and estimating these information-theoretic terms in the supervised case (see Section 5 for details).

#### 3.2.2 The Information Bottleneck Theory of Deep Learning

The IB hypothesis for deep learning proposes two distinct phases of training neural networks ( Shwartz-Ziv and Tishby, 2017 ) : the fitting and compression phases. The fitting phase involves extracting information from the input and converting it into learned representations, characterized by increased mutual information between inputs and hidden representations. Conversely, the compression phase, which is much longer, concentrates on discarding unnecessary information for target prediction, decreasing mutual information between learned representations and inputs. In contrast, the mutual information between representations and targets increases. For more information, see Geiger (2020) . Despite the elegance and plausibility of the IB hypothesis, empirically investigating it remains challenging ( Amjad and Geiger, 2018 ) .

The study of representation compression in Deep Neural Networks (DNNs) for supervised learning has shown inconsistent results. For instance, Chelombiev et al. (2019) discovered a positive correlation between generalization accuracy and the compression level of the network’s final layer. Shwartz-Ziv et al. (2018) also examined the relationship between generalization and compression, demonstrating that generalization error exponentially depends on mutual information, I ⁡ ( X , Z ) I(X;Z) . Furthermore, Achille et al. (2017) established that flat minima, known for their improved generalization properties, constrain the mutual information. However, Saxe et al. (2019) showed that compression was not necessary for generalization in deep linear networks. Basirat et al. (2021) revealed that the decrease in mutual information is essentially equivalent to geometrical compression. Other studies have found that the mutual information between training inputs and inferred parameters provides a concise bound on the generalization gap ( Xu and Raginsky, 2017 ; Pensia et al., 2018 ) . Lastly, Achille and Soatto (2018) explored using an information bottleneck objective on network parameters to prevent overfitting and promote invariant representations.

#### 3.2.3 Multiview IB Learning

The IB principle offers a rigorous method for learning encoders and decoders in supervised single-view problems. However, it is not directly applicable to multiview learning problems, as it assumes only one information source as the input. A common solution is to concatenate multiple views, though this neglects the unique characteristics of each view. To address this issue, Xu et al. (2014) introduced the large-margin multiview IB (LMIB) as an extension of the original IB problem. LMIB employs a communication system where multiple senders represent various views of examples. The system extracts specific components from different senders by compressing examples through a ”bottleneck,” and the linear projectors for each view are combined to create a shared representation. The large-margin principle replaces the maximization of mutual information in prediction, emphasizing the separation of samples from different classes. Limiting Rademacher complexity improves the solution’s accuracy and generalization error bounds. Moreover, the algorithm’s robustness is enhanced when accurate views counterbalance noisy views.

However, the LMIB method has a significant limitation: it utilizes linear projections for each view, which can restrict the combined representation when the relationship between different views is complex. To overcome this limitation, Wang et al. (2019) proposed using deep neural networks to replace linear projectors. Their model first extracts concise latent representations from each view using deep networks and then learns the joint representation of all views using neural networks. They minimize the objective: ℒ = α ​ I P(X_1),P(Z_1—X_1) ​ ( X 1 , Z 1 ) + β ​ I P(X_2),P(Z_2—X_2) ​ ( X 2 , Z 2 ) − I P(Z_2—X_2),P(Z_2—X_1) ​ ( Z 1 , 2 , Y ) \displaystyle\mathcal{L}=\alpha I_{\lx@scalerel@obj{{P(X_1),P(Z_1|X_1})}}(X_{1};Z_{1})+\beta I_{\lx@scalerel@obj{{P(X_2),P(Z_2|X_2})}}(X_{2};Z_{2})-I_{\lx@scalerel@obj{{P(Z_2|X_2),P(Z_2|X_1})}}(Z_{1,2};Y) Here, α \alpha and β \beta are trade-off parameters, Z 1 Z_{1} and Z 2 Z_{2} are the two neural networks’ representations, and Z 1 , 2 Z_{1,2} is the joint embedding of Z 1 Z_{1} and Z 2 Z_{2} . The first two terms decrease the mutual information between a view’s latent representation and its original data representation, resulting in a simpler and more generalizable model. The final term forces the joint representation to maximize the discrimination ability for the downstream task.

#### 3.2.4 Semi-Supervised IB Learning: Leveraging Unlabeled Data

Obtaining labeled data can be challenging or expensive in many practical scenarios, while many unlabeled samples may be readily available. Semi-supervised learning addresses this issue by leveraging the vast amount of unlabeled data during training in conjunction with a small set of labeled samples. Common strategies to achieve this involve adding regularization terms or adopting mechanisms that promote better generalization. Berthelot et al. (2019) grouped regularization methods into three primary categories: entropy minimization, consistency regularization, and generic regularization.

Voloshynovskiy et al. (2020) introduced an information-theoretic framework for semi-supervised learning based on the IB principle. In this context, the semi-supervised classification problem involves encoding input X X into the latent space Z Z while preserving only class-relevant information . A supervised classifier can achieve this if there is sufficient labeled data. However, when the number of labeled examples is limited, the standard label classifier p ⁡ ( y | z ) p(y|z) becomes unreliable and requires regularization.

To tackle this issue, the authors assumed a prior on the class label distribution p ⁡ ( y ) p(y) . They introduced a term to minimize the D K ​ L D_{KL} between the assumed marginal prior and the empirical marginal prior, effectively regularizing the conditional label classifier with the labels’ marginal distribution. This approach reduces the classifier’s sensitivity to the scarcity of labeled examples. They proposed two variational IB semi-supervised extensions for the priors:

Handcrafted Priors : These priors are predefined for regularization and can be based on domain knowledge or statistical properties of the data. Alternatively, they can be learned using other networks. Handcrafted priors in this context are similar to priors used in the Variational Information Bottleneck (VIB) formalism ( Alemi et al., 2016 ; Wang et al., 2019 ) .