Learning Vision from Models Rivals Learning Vision from Data — part 1/3

Cleaned-text code points [0, 15681). Bibliography omitted here; source.md includes it.

# Learning Vision from Models Rivals Learning Vision from Data

###### Abstract

We introduce SynCLR, a novel approach for learning visual representations exclusively from synthetic images and synthetic captions, without any real data. We synthesize a large dataset of image captions using LLMs, then use an off-the-shelf text-to-image model to generate multiple images corresponding to each synthetic caption. We perform visual representation learning on these synthetic images via contrastive learning, treating images sharing the same caption as positive pairs. The resulting representations transfer well to many downstream tasks, competing favorably with other general-purpose visual representation learners such as CLIP and DINO v2 in image classification tasks. Furthermore, in dense prediction tasks such as semantic segmentation, SynCLR outperforms previous self-supervised methods by a significant margin, e.g . , improving over MAE and iBOT by 6.2 and 4.3 mIoU on ADE20k for ViT-B/16.

## 1 Introduction

Representation learning extracts and organizes information from raw, often unlabeled data. The quality, quantity, and diversity of the data determines how good a representation the model can learn. The model becomes a reflection of the collective intelligence that exists in the data. We get what we feed in.

Unsurprisingly, the current best-performing visual representation learning methods [ 71 , 68 ] rely on large scale real datasets. However, the collection of real data has its own dilemmas. Collecting large scale uncurated data [ 80 ] is relatively cheap and thus quite achievable. However, for self-supervised representation learning, this approach exhibits poor scaling behavior –i.e., adding more uncurated data has little effect at large data scales [ 38 , 90 ] . Collecting small scale curated data [ 24 ] also is achievable, but models trained in this way are limited to relatively narrow tasks. The ideal would be large scale curated datasets of real images, and recent work has indeed shown that this can lead to strong performance gains at scale [ 68 ] , but this path is costly to pursue.

To alleviate the cost, in this paper we ask if synthetic data , sampled from off-the-shelf generative models, is a viable path toward large scale curated datasets that can train state-of-the-art visual representations.

We call such a paradigm learning from models , in contrast to directly learning from data . Models have several advantages as a data source for building large scale training sets: via their latent variables, conditioning variables, and hyperparameters, they provide new controls for curating data; we will make use of these controls in the method we propose. Models also can be easier to share and store (because models are more compressed than data), and can produce an unlimited number of data samples (albeit with finite diversity). A growing literature has studied these properties and other advantages (and disadvantages) of using generative models as a data source for training downstream models [ 48 , 45 , 3 , 78 , 91 , 30 ] . Some of these methods use a hybrid mode – either mixing real and synthetic datasets [ 3 ] or needing a real dataset to generate another synthetic dataset [ 91 ] . Other methods try to learn representations from purely synthetic data [ 78 ] but lag far behind the best performing models. Instead, we show that learning from models , without training on any real data, can yield representations that match the top-performing representations learnt from real data. For instance, as illustrated in Figure 1 , representations learnt by our method are able to transfer as well as OpenAI’s CLIP [ 71 ] on ImageNet (both methods using ViT-B [ 28 ] ).

Our approach leverages generative models to re-define the granularity of visual classes. As shown in Figure 2 , consider we have four images generated using two prompts: “ a golden retriever, wearing sunglasses and a beach hat, rides a bike " and “ a cute golden retriever sits in a house made of sushi ". Traditional self-supervised method such as SimCLR [ 13 ] will treat each of these images as a different class; embeddings for different images are pushed apart with no explicit consideration of the shared semantics between images. On the other extreme, supervised learning methods ( i.e . SupCE) will regard all these images as a single class (e.g., “golden retriever”). This ignores nuances in the semantics of the images, such as the fact that the dogs are riding a bike in one pair of images and sitting inside a sushi house in the other pair of images. Instead, our method, SynCLR, treats captions as classes, i.e . , each caption describes a visual class (this level of granularity was also explored in StableRep [ 91 ] ). This allows us to group images by the concepts of “riding a bike” and “sitting in a sushi house”, in addition to grouping by a coarser class label like “golden retrieval”. This level of granularity is difficult to mine in real data, since collecting multiple images described by a given caption is non-trivial, especially when scaling up the number of captions. However, text-to-image diffusion models are fundamentally built with this ability; simply by conditioning on the same caption and using different noise inputs, a text-to-image diffusion model will produce different images that all match the same caption. In our experiments, we find the caption-level granularity outperforms both SimCLR and supervised training. Another advantage is that this definition of visual classes has good scalability. Unlike ImageNet-1k/21k where a given number of classes is fixed, we can augment existing classes (or data) in an online fashion, and theoretically scale up to as many classes as needed.

Our system consists of three steps. The first step is to synthesize a large corpus of image captions. We design a scalable approach by leveraging the in-context learning capability of large language models (LLMs), where we present examples of word-to-caption translations. Next, a text-to-image diffusion model is adopted to synthesize multiple images for each synthetic caption. This yields a synthetic dataset of 600M images. Then we train visual representation models by a combination of multi-positive contrastive learning [ 50 ] and masked image modeling [ 110 ] .

Our learned representations transfer well. With SynCLR pre-training, our ViT-B and ViT-L models achieve 80.7 % \% and 83.0 % \% top-1 linear probing accuracy on ImageNet-1K, respectively, which is on par with OpenAI’s CLIP [ 71 ] . On fine-grained classification tasks, SynCLR outperforms CLIP by 3.3 % \% for ViT-B and 1.5 % \% for ViT-L, and performs similarly to DINO v2 [ 68 ] models, which are distilled from a pre-trained ViT-g model. For semantic segmentation on ADE20k, SynCLR outperforms MAE pre-trained on ImageNet by 6.2 and 4.1 in mIoU for ViT-B and ViT-L under the same setup, showing strong transfer ability for dense prediction tasks similar to DINO v2, which additionally involves a training period on 518x518 resolution images that SynCLR does not have.

## 2 Related Works

Self-supervised representation learning approaches in vision develop domain-specific pre-text tasks, such as colorization [ 106 ] , rotation prediction [ 36 ] , and solving jigsaw puzzles [ 65 ] . Domain-agnostic approaches have been popular, such as contrastive learning [ 6 , 40 , 66 , 97 , 88 , 43 , 13 ] and masked image modeling [ 5 , 100 , 44 , 96 , 2 , 110 , 4 , 33 ] . Contrastive learning promotes invariance [ 89 ] for two views of the same image and pushes apart representations for different images [ 95 ] (or only invariance [ 39 , 11 ] ); the resulting representations yield strong performance for linear or zero-shot transfer. Masked image modeling reconstructs the pixels [ 44 , 100 ] or local features [ 4 ] , often producing excellent fine-tuning transfer performance, especially in dense prediction tasks [ 44 ] . The state of the art DINO v2 [ 68 ] leverages both approaches, and our approach shares a similar spirit.

Supervised learning [ 52 , 84 , 41 ] used to be the dominant approach for learning transferable visual representations for various tasks [ 37 , 26 , 81 ] . Recent studies [ 57 , 42 ] has shown that, the transferability of representations learned in this way is limited, e.g . , pre-training has no improvement over random initialization for dense prediction tasks ( e.g . , object detection) when the fine-tuning is long enough. Such limitation continues when the model has been scaled up to 22B [ 23 ] . An alternative paradigm learns visual representations from text supervision [ 71 , 49 ] , e.g . , CLIP [ 71 ] . This approach is more flexible ( i.e . , not requiring classes) and provides richer supervision, often learning generalizable representations.

Generative models as representation learners. A number of papers have explored the representations that are learned by generative models for various recognition tasks [ 25 , 56 ] . As might be expected intuitively, such models indeed learn especially good representations for dense tasks, such as optical flow estimation [ 79 ] , semantic segmentation [ 8 , 101 ] , and depth estimation [ 107 ] . Another line of work [ 55 , 19 ] adapt pre-trained diffusion models for zero-shot image recognition via analysis-by-synthesis. These approaches may need to be adapted when the architectures of the generative models change or a new family of generative model emerge. Our approach treats images as universal interfaces with the hope of better generality.

Learning from synthetic data from generative models. Synthetic data has been explored to train machine learning models in various domains [ 83 , 74 , 75 , 63 , 53 , 87 , 102 , 62 , 31 ] . In computer vision, the utilization of synthetic data for training models is common, ranging from optical flow [ 61 ] and autonomous driving [ 1 ] to semantic segmentation [ 15 ] and human pose estimation [ 94 ] . Others [ 58 , 48 ] have explored synthetic data for representation learning, with the predominant approach of altering the latent variables of deep generative models. Our approach aligns with this research paradigm, but it diverges in its use of text-to-image models, which have also been investigated by other researchers [ 78 , 111 , 45 ] . But they use synthetic data for supervised learning [ 78 , 30 ] . The closet work is StableRep [ 91 ] , which also conducts representation learning but still needs a real text dataset.

## 3 Approach

In this paper, we study the problem of learning a visual encoder f f in the absence of real images or textual data. Our approach hinges on the utilization of three key resources: a language generation model ( g 1 g_{1} ), a text-to-image generative model ( g 2 g_{2} ), and a curated list of visual concepts ( C C ). Our exploration include three steps: (1) we employ g 1 g_{1} to synthesize a comprehensive set of image descriptions T T , which encompass the range of visual concepts in C C ; (2) for each caption in T T , we generate multiple images using g 2 g_{2} , culminating in an extensive synthetic image dataset X X ; (3) we train on X X to obtain a visual representation encoder f f .

We use Llama-2 7B [ 93 ] and Stable Diffusion 1.5 [ 73 ] as g 1 g_{1} and g 2 g_{2} , respectively, because of their fast inference speed. We anticipate that better g 1 g_{1} and g 2 g_{2} in the future will further enhance the effectiveness of this approach.

### 3.1 Synthesizing captions

To harness the capability of powerful text-to-image models for generating a substantial dataset of training images, we initially require a collection of captions that not only precisely depict an image but also exhibit diversity to encompass a broad spectrum of visual concepts.

We have developed a scalable approach to create such a large collection of captions, leveraging the in-context learning capability of LLMs [ 9 ] . Our method involves crafting specific prompt engineering templates that guide the LLM to produce the required captions. We start by gathering the concept list C C from some existing datasets, such as ImageNet-21k [ 24 ] and Places-365 [ 108 ] . For each concept c ∈ C c\in C , we consider three straightforward templates to generate captions effectively.

• c c –> caption . As the most direct and simple approach, we have the Llama-2 model sample a sentence for the concept c c .

• c c , b ​ g bg –> caption . We combine the visual concept c c with a background or setting b ​ g bg . A naïve approach would randomly select both c c and b ​ g bg , where b ​ g bg may correspond to a class name from a places dataset like [ 108 ] . However, this method often leads to unlikely combinations in the real world, such as a blue whale in a football field. Our ablation experiments demonstrate that this strategy results in suboptimal performance, likely because the generated captions fall far outside the training distribution of g 2 g_{2} . Instead, we employ GPT-4 [ 67 ] to generate a list of suitable backgrounds for the chosen concepts. This approach increases the likelihood of generating more plausible combinations, such as a tiger in a forest or a cat in a kitchen, enhancing the overall quality of the results.

• c c , r ​ e ​ l rel –> caption . Given a visual concept c c , we consider pairing it with a positional relationship word, r ​ e ​ l rel . Take for instance, if c c signifies cat and r ​ e ​ l rel translates to in front of , our objective is to prompt the LLM to create captions such as a cute yellow cat is enjoying the fish in front of the sofa . To add variety, we have a selection of 10 different positional relationship words that we randomly choose from.

For each of the three templates, we have prepared multiple demonstration examples that serve as instructions for the LLM to complete the caption synthesis task. Table 1 shows a couple of examples for each template. In total, we have 106 examples for c c –>prompt , 50 examples for c , b ​ g c,bg –>prompt , and 20 examples for c , r ​ e ​ l c,rel –>prompt . Such examples are mostly collected by prompting GPT-4, with a handful from human. In a pilot study, we do not observe difference between including or excluding human generated examples.

In the stage of generating captions in-context, we select a concept and one of the three templates. Next, we randomly pick three examples from the chosen template and frame the caption generation as a text completion task. This process is illustrated in Figure 3 .

### 3.2 Synthesizing Images

For each text caption, we generate a variety of images by initiating the reverse diffusion process with different random noise. The Classifier-Free Guidance (CFG) scale is a crucial factor in this process. A higher CFG scale enhances the quality of the samples and the alignment between text and image, whereas a lower scale results in more diverse samples and better adherence to the original conditional distribution of images based on the given text. Following the approach used in StableRep [ 91 ] , we opt for a lower CFG scale, specifically 2.5, and produce 4 images for each caption. Examples of these images can be seen in Figure 4 .

### 3.3 Representation Learning

Our representation learning method is built upon StableRep [ 91 ] . The key component of our approach is the multi-positive contrastive learning loss [ 50 ] which works by aligning (in the embedding space) images generated from the same caption. We additionally combine multiple techniques from other self-supervised learning methods, including a patch-level masked image modeling objective. We briefly review StableRep and elaborate on the added modules.