To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review — part 5/6
Cleaned-text code points [63091, 78742). Bibliography omitted here; source.md includes it.
In the supervised case, where Z Z is a learned stochastic representation of the input and Y Y is the label, we aim to optimize
I ( Y , Z ) ≥ H ( Y ) + 𝔼 [ log q ( Y ∣ Z ) ] I(Y;Z)\geq H(Y)+\bm{\mathbb{E}}\left[\log q(Y\mid Z)\right] (8) . Since Y Y is constant, optimizing the information I ( Z , Y ) I(Z;Y) requires only minimizing the prediction term 𝔼 [ log q ( Y | Z ) ] \mathbb{E}\left[\log q(Y|Z)\right] by making Z Z more informative about Y Y . This term is the cross-entropy loss for classification or the square loss for regressions. Thus, we can minimize the log loss without any other regularization on the representation.
In contrast, for the self-supervised case, we have a more straightforward option to minimize H ( Z 1 | Z 2 ) H(Z_{1}|Z_{2}) : Making Z 1 Z_{1} easier to predict by Z 2 Z_{2} , which can be achieved by reducing its variance along specific dimensions. If we do not regularize H ( Z 1 ) H(Z_{1}) , it will decrease to zero, and we will observe a collapse. This is why, in contrastive methods, the variance of the representation (large entropy) is significant only in the directions with a high variance in the data, which is enforced by data augmentation ( Jing et al., 2021 ) . According to this analysis, the network benefits from making the representations ”simple” (easier to predict). Hence, even though our representation does not have explicit information-theoretical constraints, the learning process will compress the representation.
### 4.2 Beyond the Multiview Assumption
According to the Multiview IB analysis presented in Section 4 , the optimal way to create a useful representation is to maximize the mutual information between the representations of different views while compressing irrelevant information in each representation. In fact, as discussed in Section 4.1 , we can achieve this optimal compressed representation even without explicit regularization. However, this optimality is based on the Multiview assumption , which states that the relevant information for downstream tasks comes from the information shared between views. Therefore, Tian et al. (2020b) concluded that when a minimal sufficient representation has been obtained, the optimal views for self-supervised learning are determined by downstream tasks.
However, the Multiview assumption is highly constrained, as all relevant information must be shared between all views. In cases where this assumption is incorrect, such as with aggressive data augmentation or multiple downstream tasks or modalities, sharing all the necessary information can be challenging. For example, if one view is a video stream while the other is an audio stream, the shared information may be sufficient for object recognition but not for tracking. Furthermore, relevant information for downstream tasks may not be contained within the shared information between views, meaning that removing non-shared information can negatively impact performance.
Kahana and Hoshen (2022) identified a series of tasks that violate the Multiview assumption . To accomplish these tasks, the learned representation must also be invariant to unwanted attributes, such as bias removal and cross-domain retrieval. In such cases, only some attributes have labels, and the objective is to learn an invariant representation for the domain for which labels are provided while also being informative for all other attributes without labels. For example, for face images, only the identity labels may be provided, and the goal is to learn a representation that captures the unlabeled pose attribute but contains no information about the identity attribute. The task can also be applied to fair decisions, cross-domain matching, model anonymization, and image translation.
Wang et al. (2022) formalized another case where the Multiview assumption does not hold when non-shared task-relevant information cannot be ignored. In such cases, the minimal sufficient representation contains less task-relevant information than other sufficient representations, resulting in inferior performance. Furthermore, their analysis shows that in such cases, the learned representation in contrastive learning is insufficient for downstream tasks, which may overfit the shared information.
As a result of their analysis, Wang et al. (2022) and Kahana and Hoshen (2022) proposed explicitly increasing mutual information between the representation and input to preserve task-relevant information and prevent the compression of unshared information between views. In this case, the two regularization terms of the two views are incorporated into the original InfoMax objective, and the following objective is optimized:
ℒ = min P ( Z 1 | X 1 ) , p ( Z 2 | X 2 ) − I P(Z_1—X_1) ( X 1 , Z 1 ) − I P(Z_2—X_2) ( X 2 , Z 2 ) − β I P(Z_1—X_1),P(Z_2—Z_1) ( Z 1 , Z 2 ) . \displaystyle\mathcal{L}=\min_{P(Z_{1}|X_{1}),p(Z_{2}|X_{2})}-I_{\lx@scalerel@obj{P(Z_1|X_1)}}(X_{1};Z_{1})-I_{\lx@scalerel@obj{P(Z_2|X_2)}}(X_{2};Z_{2})-\beta I_{\lx@scalerel@obj{P(Z_1|X_1),P(Z_2|Z_1)}}(Z_{1};Z_{2}). (9) Wang et al. (2022) demonstrated the effectiveness of their method for SimCLR ( Chen et al., 2020a ) , BYOL ( Grill et al., 2020 ) , and Barlow Twins ( Zbontar et al., 2021 ) across classification, detection, and segmentation tasks.
### 4.3 To Compress or Not to Compress?
As seen in Eq. 9 , when the Multiview assumption is violated, the objective for obtaining an optimal representation is to maximize the mutual information between each input and its representation. This contrasts with the situation in which the Multiview assumption holds, or the supervised case, where the objective is to minimize the mutual information between the representation and the input. In both supervised and unsupervised cases, we have direct access to the relevant information, which we can use to separate and compress irrelevant information. However, in the self-supervised case, we depend heavily on the Multiview assumption . If this assumption is violated due to unshared information between views that is relevant for the downstream task, we cannot separate relevant and irrelevant information. Furthermore, the learning algorithm’s nature requires that this information be protected by explicitly maximizing it.
As datasets continue to expand in size and models are anticipated to serve as base models for various downstream tasks, the Multiview assumption becomes less pertinent. Consequently, compressing irrelevant information when the Multiview assumption does not hold presents one of the most significant challenges in self-supervised learning. Identifying new methods to separate relevant from irrelevant information based on alternative assumptions is a promising avenue for research. It is also essential to recognize that empirical measurement of information-theoretic quantities and their estimators plays a crucial role in developing and evaluating such methods.
## 5 Optimizing Information in Deep Neural Networks: Challenges and Approaches
Recent years have seen information-theoretic analyses employed to explain and optimize deep learning techniques ( Shwartz-Ziv and Tishby, 2017 ) . Despite their elegance and plausibility, empirically measuring and analyzing information in deep networks presents challenges. Two critical problems are (1) information in deterministic networks and (2) estimating information in high-dimensional spaces.
#### Information in Deterministic Networks
Information-theoretic methods have significantly impacted deep learning ( Alemi et al., 2016 ; Steinke and Zakynthinou, 2020 ; Shwartz-Ziv and Tishby, 2017 ) . However, a key challenge is addressing the source of randomness in deterministic DNNs.
The mutual information between the input and representation is infinite, leading to ill-posed optimization problems or piecewise constant outcomes ( Amjad and Geiger, 2019 ; Goldfeld et al., 2018 ) . To tackle this issue, researchers have proposed various solutions. One common approach is to discretize the input distribution and real-valued hidden representations by binning, which facilitates non-trivial measurements and prevents the mutual information from always taking the maximum value of the log of the dataset size, thus avoiding ill-posed optimization problems ( Shwartz-Ziv and Tishby, 2017 ) .
However, binning and discretization are essentially equivalent to geometrical compression and serve as clustering measures ( Goldfeld et al., 2018 ) . Moreover, this discretization depends on the chosen bin size and does not track the mutual information across varying bin sizes Goldfeld et al. (2018) ; Ross (2014) . To address these limitations, researchers have proposed alternative approaches such as interpreting binned information as a weight decay penalty Elad et al. (2019b) , estimating mutual information based on lower bounds assuming a continuous input distribution without making assumptions about the network’s output distribution properties ( Wang and Isola, 2020 ; Zimmermann et al., 2021 ; Shwartz-Ziv et al., 2022a ) , injecting additive noise, and considering data augmentation as the source of noise ( Lee et al., 2021b ; Shwartz-Ziv and Tishby, 2017 ; Goldfeld et al., 2018 ; Dubois et al., 2021 ) .
#### Measuring Information in High-Dimensional Spaces
Estimating mutual information in high-dimensional spaces presents a significant challenge when applying information-theoretic measures to real-world data. This problem has been extensively studied ( Paninski, 2003 ; Gao et al., 2015 ) , revealing the inefficiency of solutions for large dimensions and the limited scalability of known approximations with respect to sample size and dimension. Despite these difficulties, various entropy and mutual information estimation approaches have been developed, including classic methods like k-nearest neighbors (KNN) ( Kozachenko and Leonenko, 1987 ) and kernel density estimation techniques ( Hang et al., 2018 ) , as well as more recent efficient methods.
Chelombiev et al. (2019) developed adaptive mutual information estimators based on entropies-equal bins and scaled noise kernel density estimator. Generative decoder networks, such as PixelCNN++ ( Van den Oord et al., 2016 ) , have been employed to estimate a lower bound on mutual information ( Darlow and Storkey, 2020 ; Nash et al., 2018 ; Shwartz-Ziv et al., 2023 ) . Another strategy includes ensemble dependency graph estimators, adaptive mutual information estimation methods (EDGE) by merging randomized locality-sensitive hashing (LSH), dependency graphs, and ensemble bias reduction techniques ( Noshad and Hero III, 2018 ) . The Mutual Information Neural Estimator (MINE) ( Belghazi et al., 2018a ) maximizes KL divergence using the dual representation of Donsker and Varadhan (1975) and has been employed for direct mutual information estimation ( Elad et al., 2019a ) . Shwartz-Ziv and Alemi (2020) developed a controlled framework that utilized the neural tangent kernels ( Jacot et al., 2018 ) , in order to obtain tractable information measures.
Improving mutual information estimation can be achieved using larger batch sizes, although this may negatively impact generalization performance and memory requirements. Alternatively, researchers have suggested employing surrogate measures for mutual information, such as log-determinant mutual information (LDMI), based on second-order statistics ( Ozsoy et al., 2022 ; Erdogan, 2022 ) , which reflects linear dependence. Goldfeld and Greenewald (2021) proposed the Sliced Mutual Information (SMI), defined as an average of MI terms between one-dimensional projections of high-dimensional variables. SMI inherits many properties of its classic counterpart. It can be estimated with optimal parametric error rates in all dimensions by combining an MI estimator between scalar variables with an MC integrator ( Goldfeld and Greenewald, 2021 ) . The k k -SMI, introduced by Goldfeld et al. (2022) , extends the SMI by projecting to k k -dimensional subspace, which relaxes the smoothness assumptions, improves scalability, and enhances performance.
In conclusion, estimating and optimizing information in deep neural networks presents significant challenges, particularly in deterministic networks and high-dimensional spaces. Researchers have proposed various approaches to address these issues, including discretization, alternative estimators, and surrogate measures. As the field continues to evolve, it is expected that more advanced techniques will emerge to overcome these challenges and facilitate the understanding and optimization of deep learning models.
## 6 Future Research Directions
Despite the solid foundation established by existing self-supervised learning methods from an information theory perspective, several potential research directions warrant exploration:
##### Self-supervised learning with non-shared information.
As discussed in Section 4 , the separation of relevant (preserved) and irrelevant (compressed) information relies on the Multiview Assumption . This assumption, which states that only shared information is essential for downstream tasks, is rather restrictive. For example, situations may arise where each view contains distinct information relevant to a downstream task or multiple tasks necessitate different features. Some methods have been proposed to tackle this problem, but they mainly focus on maximizing the network’s information without explicit constraints. Formalizing this scenario and exploring differentiating between relevant and irrelevant data based on non-shared information represents an intriguing research direction.
##### Self-supervised learning for tabular data.
At present, the internal compression of self-supervised learning methods may compress relevant information due to improper augmentation 4.1 . Consequently, we must heavily rely on generating the two views, which must accurately represent information related to the downstream process. Custom augmentation must be developed for each domain, taking into account extensive prior knowledge on data augmentation. While some papers have attempted to extend self-supervised learning to tabular data ( Ucar et al., 2021 ; Arik and Pfister, 2021 ) , further work is necessary from both theoretical and practical standpoints to achieve high performance with self-supervised learning for tabular data ( Shwartz-Ziv and Armon, 2022 ) . The augmentation process is crucial for the performance of current vision and text models. In the case of tabular data, employing information-theoretic loss functions that do not require information compression may help harness the benefits of self-supervised learning.
##### Integrating other learning methods into the information-theoretic framework.
Prior works have investigated various supervised, unsupervised, semi-supervised, and self-supervised learning methods, demonstrating that they optimize information-theoretic quantities. However, state-of-the-art methods employ additional changes and engineering practices that may be related to information theory, such as the stop gradient operation utilized by many self-supervised learning methods today ( Grill et al., 2020 ; Chen and He, 2021 ) . The Expectation-Maximization (EM) algorithm ( Dempster et al., 1977 ) can be employed to explain this operation when one path is the E-step and the other is the M-step. Additionally, Elidan and Friedman (2012) proposed an IB-inspired version of the EM, which could help develop information-theoretic-based objectives using the stop gradient operation.
##### Expanding the analysis to usable information.