Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture

arxiv:2301.08243

Original source · Paper JSON

Self-supervised learning and vision · 2023-01-19

Cites (4) · Cited by (7)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
• Abstract
• 1 Introduction
• 2 Background
• Joint-Embedding Architectures.
• Generative Architectures.
• Joint-Embedding Predictive Architectures.
• 3 Method
• Targets.
• Context.
• Prediction.
• Loss.
• 4 Related Work
• 5 Image Classification
• ImageNet-1K.
• Low-Shot ImageNet-1K.
• Transfer learning.
• 6 Local Prediction Tasks
• 7 Scalability
• Model Efficiency.
• Scaling data size.
• Scaling model size.
• 8 Predictor Visualizations
• 9 Ablations
• Predicting in representation space.
• Masking strategy.
• 10 Conclusion
• Appendix A Implementation Details
• A.1 Pretraining
• Architectures.
• Optimization.
• Masking.
• A.2 Downstream Tasks
• Linear evaluation.
• ImageNet evaluations.
• Low-shot evaluation.
• Appendix B Broader Related Work
• Appendix C Additional Ablations
• Multiblock masking strategy.
• Masking at the output of the target-encoder.
• Predictor depth.
• Weight decay.
• Predictor width.
• Appendix D Finetuning on the full ImageNet
• Appendix E RCDM Visualizations
• E.1 Encoder Visualization
• Instructions for reporting errors