To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review

arxiv:2304.09355

Original source · Paper JSON

Self-supervised learning and vision · 2023-04-19

Cites (3) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• To Compress or Not to Compress - Self-Supervised Learning and Information Theory: A Review
• Abstract
• keywords
• 1 Introduction
• 2 Background and Fundamental Concepts
• 2.1 Multiview Representation Learning
• 2.2 Self-Supervised Learning
• 2.3 Semi-Supervised Learning
• 2.4 Representation Learning
• 2.4.1 Minimal Sufficient Statistic
• Definition 1 .
• Theorem 2 .
• Definition 3 .
• 2.4.2 The Information Bottleneck
• 2.5 Representation Learning and the Information Bottleneck
• 3 Information-Theoretic Objectives
• 3.1 Setup and Methodology
• 3.2 Optimization with Labels
• 3.2.1 Single-View Supervised Learning
• Definition 4 .
• 3.2.2 The Information Bottleneck Theory of Deep Learning
• 3.2.3 Multiview IB Learning
• 3.2.4 Semi-Supervised IB Learning: Leveraging Unlabeled Data
• 3.2.5 Unsupervised IB learning
• 4 Self-Supervised Multiview Information Bottleneck Learning
• Assumption 1 .
• 4.1 Implicit Compression in Self-Supervised Learning Methods
• 4.2 Beyond the Multiview Assumption
• 4.3 To Compress or Not to Compress?
• 5 Optimizing Information in Deep Neural Networks: Challenges and Approaches
• Information in Deterministic Networks
• Measuring Information in High-Dimensional Spaces
• 6 Future Research Directions
• Self-supervised learning with non-shared information.
• Self-supervised learning for tabular data.
• Integrating other learning methods into the information-theoretic framework.
• Expanding the analysis to usable information.
• Extending self-supervised learning’s information-based perspective to energy-based model optimization.
• Expanding the multiview framework to accommodate more views and tasks.
• 7 Conclusion