To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review — part 1/6

Cleaned-text code points [0, 15958). Bibliography omitted here; source.md includes it.

# To Compress or Not to Compress - Self-Supervised Learning and Information Theory: A Review

###### Abstract

Deep neural networks excel in supervised learning tasks but are constrained by the need for extensive labeled data. Self-supervised learning emerges as a promising alternative, allowing models to learn without explicit labels. Information theory, and notably the information bottleneck principle, has been pivotal in shaping deep neural networks. This principle focuses on optimizing the trade-off between compression and preserving relevant information, providing a foundation for efficient network design in supervised contexts. However, its precise role and adaptation in self-supervised learning remain unclear. In this work, we scrutinize various self-supervised learning approaches from an information-theoretic perspective, introducing a unified framework that encapsulates the self-supervised information-theoretic learning problem . We weave together existing research into a cohesive narrative, delve into contemporary self-supervised methodologies, and spotlight potential research avenues and inherent challenges. Additionally, we discuss the empirical evaluation of information-theoretic quantities and their estimation methods. Overall, this paper furnishes an exhaustive review of the intersection of information theory, self-supervised learning, and deep neural networks.

###### keywords

## 1 Introduction

Deep neural networks (DNNs) have revolutionized fields such as computer vision, natural language processing, and speech recognition due to their remarkable performance in supervised learning tasks ( Alam et al., 2020 ; LeCun et al., 2015 ; He et al., 2015 ) . However, the success of DNNs is often limited by the need for vast amounts of labeled data, which can be both time-consuming and expensive to acquire. Self-supervised learning (SSL) emerges as a promising alternative, enabling models to learn from data without explicit labels by leveraging the underlying structure and relationships within the data itself.

Recent advances in SSL have been driven by joint embedding architectures, such as Siamese Nets ( Bromley et al., 1993 ) , DrLIM ( Chopra et al., 2005 ; Hadsell et al., 2006 ) , and SimCLR ( Chen et al., 2020a ) . These approaches define a loss function that encourages representations of different versions of the same image to be similar while pushing representations of distinct images apart. After optimizing the surrogate objective, the pre-trained model can be employed as a feature extractor, with the learned features serving as inputs for downstream supervised tasks like image classification, object detection, instance segmentation, or pose estimation ( Caron et al., 2021 ; Chen et al., 2020a ; Misra and van der Maaten, 2020 ; Shwartz-Ziv et al., 2022b ) . Although SSL methods have shown promising results in practice, the theoretical underpinnings behind their effectiveness remain an open question ( Arora et al., 2019 ; Lee et al., 2021a ) .

Information theory has played a crucial role in understanding and optimizing deep neural networks, from practical applications like the variational information bottleneck ( Alemi et al., 2016 ) to theoretical investigations of generalization bounds induced by mutual information ( Xu and Raginsky, 2017 ; Steinke and Zakynthinou, 2020 ) . Building upon these foundations, several researchers have attempted to enhance self-supervised and semi-supervised learning algorithms using information-theoretic principles, such as the Mutual Information Neural Estimator (MINE) ( Belghazi et al., 2018b ) combined with the information maximization (InfoMax) principle ( Linsker, 1988 ) . However, the plethora of objective functions, contradicting assumptions, and various estimation techniques in the literature can make it challenging to grasp the underlying principles and their implications.

In this paper, we aim to achieve two objectives. First, we propose a unified framework that synthesizes existing research on self-supervised and semi-supervised learning from an information-theoretic standpoint. This framework allows us to present and compare current methods, analyze their assumptions and difficulties, and discuss the optimal representation for neural networks in general and self-supervised networks in particular. Second, we explore different methods and estimators for optimizing information-theoretic quantities in deep neural networks and investigate how recent models optimize various theoretical-information terms.

By reviewing the literature on various aspects of information-theoretic learning, we provide a comprehensive understanding of the interplay between information theory, self-supervised learning, and deep neural networks. We discuss the application of the information bottleneck principle ( Tishby et al., 1999a ) , connections between information theory and generalization, and recent information-theoretic learning algorithms. Furthermore, we examine how the information-theoretic perspective can offer insights into the design of better self-supervised learning algorithms and the potential benefits of using information theory in SSL across a wide range of applications.

In addition to the main structure of the paper, we dedicate a section to the challenges and opportunities in extending the information-theoretic perspective to other learning paradigms, such as energy-based models. We highlight the potential advantages of incorporating these extensions into self-supervised learning algorithms and discuss the technical and conceptual challenges that must be addressed.

The structure of the paper is as follows. Section 2 introduces the key concepts in supervised, semi-supervised, self-supervised learning, information theory, and representation learning. Section 3 presents a unified framework for multiview learning based on information theory. We first discuss what an optimal representation is and why compression is beneficial for learning. Next, we explore optimal representation in single-view supervised learning models and how they can be extended to unsupervised, semi-supervised, and multiview contexts. The focus then shifts to self-supervised learning, where the optimal representation remains an open question. Using the unified framework, we compare recent self-supervised algorithms and discuss their differences. We analyze the assumptions behind these models, their effects on the learned representation, and their varying perspectives on important information within the network.

Section 5 addresses several technical challenges, discussing both theoretical and practical issues in estimating theoretical information terms. We present recent methods for estimating these quantities, including variational bounds and estimators. Section 6 concludes the paper by offering insights into potential future research directions at the intersection of information theory, self-supervised learning, and deep neural networks. Our aim is to inspire further research that leverages information theory to advance our understanding of self-supervised learning and to develop more efficient and effective models for a broad range of applications.

## 2 Background and Fundamental Concepts

### 2.1 Multiview Representation Learning

Multiview learning has gained increasing attention and great practical success by using complementary information from multiple features or modalities. The multiview learning paradigm divides the input variable into multiple views from which the target variable should be predicted ( Zhao et al., 2017b ) . Using this paradigm, one can eliminate hypotheses that contradict predictions from other views and provide a natural semi-supervised and self-supervised learning setting. A multiview dataset consists of data captured from multiple sources, modalities, and forms but with similar high-level semantics ( Yan et al., 2021 ) . This mechanism was initially used for natural-world data, combining image, text, audio, and video measurements. For example, photos of objects are taken from various angles, and our supervised task is to identify the objects. Another example is identifying a person by analyzing the video stream as one view and the audio stream as the other.

Although these views often provide different and complementary information about the same data, directly integrating them does not produce satisfactory results due to biases between multiple views ( Yan et al., 2021 ) . Thus, multiview representation learning involves identifying the underlying data structure and integrating the different views into a common feature space, resulting in high performance. In recent decades, multiview learning has been used for many machine learning tasks and influenced many algorithms, such as co-training mechanisms ( Kumar and Daumé, 2011 ) , subspace learning methods ( Xue et al., 2019 ) , and multiple kernel learning (MKL) ( Bach and Jordan, 2002 ) . Li et al. (2018) proposed two categories for multiview representation learning: (i) multiview representation fusion, which combines different features from multiple views into a single compact representation, and (ii) alignment of multiview representation, which attempts to capture the relationships among multiple different views through feature alignment. In this case, a learned mapping function embeds the data of each view, and the representations are regularized to form a multiview-aligned space. In this research direction, an early study is the Canonical Correlation Analysis (CCA) ( Hotelling, 1936 ) and its kernel extensions ( Bach and Jordan, 2003 ; Hardoon et al., 2004 ; Sun, 2013 ) . In addition to CCA, multiview representation learning has penetrated a variety of learning methods, such as dimensionality reduction ( Sun et al., 2010 ) , clustering analysis ( Yan et al., 2015 ) , multiview sparse coding ( Jia et al., 2010 ; Cao et al., 2013 ; Liu et al., 2014 ) , and multimodal topic learning ( Pu et al., 2020 ) . However, despite their promising results, these methods use handcrafted features and linear embedding functions, which cannot capture the nonlinear properties of multiview data.

The emergence of deep learning has provided a powerful way to learn complex, nonlinear, and hierarchical representations of data. By incorporating multiple hierarchical layers, deep learning algorithms can learn complex, subtle, and abstract representations of target data. The success of deep learning in various application domains has led to a growing interest in deep multiview methods, which have shown promising results. Examples of these methods include deep multiview canonical correlation analysis ( Andrew et al., 2013 ) as an extension of CCA, multiview clustering via deep matrix factorization ( Zhao et al., 2017a ) , and the deep multiview spectral network ( Huang et al., 2019 ) . Moreover, deep architectures have been employed to generate effective representations in methods such as multiview convolutional neural networks ( Liu et al., 2021a ) , multimodal deep Boltzmann machines ( Srivastava and Salakhutdinov, 2014 ) , multimodal deep autoencoders ( Ngiam et al., 2011 ; Wang et al., 2015 ) , and multimodal recurrent neural networks ( Karpathy and Fei-Fei, 2015 ; Mao et al., 2014 ; Donahue et al., 2015 ) .

### 2.2 Self-Supervised Learning

Self-supervised learning (SSL) is a powerful technique that leverages unlabeled data to learn useful representations. In contrast to supervised learning, which relies on labeled data, SSL employs self-defined signals to establish a proxy objective between the input and the signal. The model is initially trained using this proxy objective and subsequently fine-tuned on the target task. Self-supervised signals, derived from the inherent co-occurrence relationships in the data, serve as self-supervision. Various such signals have been used to learn representations, including generative and joint embedding architectures ( Chen et al., 2020a ; Chen et al., 2020b ; Bachman et al., 2019 ; Bar et al., 2022 ) .

Two main categories of SSL architectures exist: (1) generative architectures based on reconstruction or prediction and (2) joint embedding architectures ( Liu et al., 2021b ) . Both architecture classes can be trained using either contrastive or non-contrastive methods.

We begin by discussing these two main types of architectures:

1. Generative Architecture: Generative architectures employ an objective function that measures the divergence between input data and predicted reconstructions, such as squared error. The architecture reconstructs data from a latent variable or a corrupted version, potentially with a latent variable’s assistance. Notable examples of generative architectures include auto-encoders, sparse coding, sparse auto-encoders, and variational auto-encoders ( Kingma and Welling, 2013 ; Lee et al., 2006 ; Ng et al., 2011 ) . As the reconstruction task lacks a single correct answer, most generative architectures utilize a latent variable, which, when varied, generates multiple reconstructions. The latent variable’s information content requires regularization to ensure the system reconstructs regions of high data density while avoiding a collapse by reconstructing the entire space. PCA regularizes the latent variable by limiting its dimensions, while sparse coding and sparse auto-encoders restrict the number of non-zero components. Variational auto-encoders regularize the latent variable by rendering it stochastic and maximizing the entropy of the distribution relative to a prior. Vector quantized variational auto-encoders (VQ-VAE) employ binary stochastic variables to achieve similar results ( Van Den Oord et al., 2017 ) .

2. Joint Embedding Architectures (JEA): These architectures process multiple views of an input signal through encoders, producing representations of the views. The system is trained to ensure that these representations are both informative and mutually predictable. Examples include Siamese networks, where two identical encoders share weights ( Chen and He, 2021 ; Chen et al., 2020a ; He et al., 2020 ; Grill et al., 2020 ) , and methods permitting encoders to differ ( Bardes et al., 2021 ) . A primary challenge with JEA is preventing informational collapse, in which the representations contain minimal information about the inputs, thereby facilitating their mutual prediction. JEA’s advantage lies in the encoders’ ability to eliminate noisy, unpredictable, or irrelevant information from the input within the representation space.

To effectively train these architectures, it is essential to ensure that the representations of different signals are distinct. This can be achieved through either contrastive or non-contrastive methods:

• Contrastive Methods: Contrastive methods utilize data points from the training set as positive samples and generate points outside the region of high data density as contrastive samples . The energy (e.g., reconstruction error for generative architectures or representation predictive error for JEA) should be low for positive samples and higher for contrastive samples. Various loss functions involving the energies of pairs or sets of samples can be minimized to achieve this objective.

• Non-Contrastive Methods: Non-contrastive methods prevent the energy landscape’s collapse by limiting the volume of space that can take low energy, either through architectural constraints or through a regularizer in the energy or training objective. In latent-variable generative architectures, preventing collapse is achieved by limiting or minimizing the information content of the latent variable. In JEA, collapse is prevented by maximizing the information content of the representations.

We now present a few concrete examples of popular models that employ various combinations of generative architectures, joint embedding architectures, contrastive training, and non-contrastive training: