Learning Vision from Models Rivals Learning Vision from Data β€” part 2/3

Cleaned-text code points [15681, 31371). Bibliography omitted here; source.md includes it.

StableRep [ 91 ] minimizes the cross-entropy loss between a ground-truth assignment distribution and a contrastive assignment distribution. Consider an encoded anchor sample 𝒂 \bm{a} and a set of encoded candidates { 𝒃 1 , 𝒃 2 , … , 𝒃 K } \{\bm{b}_{1},\bm{b}_{2},...,\bm{b}_{K}\} . The contrastive assignment distribution πͺ {\mathbf{q}} describes how likely the model predicts 𝒂 \bm{a} and each 𝒃 \bm{b} to be generated from the same caption, and the ground-truth distribution is the actual match between 𝒂 \bm{a} and 𝒃 \bm{b} ( 𝒂 \bm{a} is allowed to match multiple 𝒃 \bm{b} ): πͺ i = exp ⁑ ( 𝒂 β‹… 𝒃 i / Ο„ ) βˆ‘ j = 1 K exp ⁑ ( 𝒂 β‹… 𝒃 j / Ο„ ) \displaystyle{\mathbf{q}}_{i}=\frac{\exp(\bm{a}\cdot\bm{b}_{i}/\tau)}{\sum_{j=1}^{K}\exp(\bm{a}\cdot\bm{b}_{j}/\tau)} (1) 𝐩 i = πŸ™ match ​ ( 𝒂 , 𝒃 i ) βˆ‘ j = 1 K πŸ™ match ​ ( 𝒂 , 𝒃 j ) \displaystyle{\mathbf{p}}_{i}=\frac{\mathbbm{1}_{\text{match}(\bm{a},\bm{b}_{i})}}{\sum_{j=1}^{K}\mathbbm{1}_{\text{match}(\bm{a},\bm{b}_{j})}} (2) where Ο„ ∈ β„› + \tau\in\mathcal{R}_{+} is the scalar temperature, 𝒂 \bm{a} and all 𝒃 \bm{b} have been β„“ 2 \ell_{2} normalized, and the indicator function πŸ™ match ​ ( β‹… , β‹… ) \mathbbm{1}_{\text{match}(\cdot,\cdot)} indicates whether two samples are from the same caption. The contrastive loss for 𝒂 \bm{a} is given as β„’ ( 𝒂 ) = H ( 𝐩 , πͺ ) = βˆ’ βˆ‘ i = 1 K 𝐩 i log πͺ i \displaystyle\mathcal{L}(\bm{a})=H({\mathbf{p}},{\mathbf{q}})=-\sum_{i=1}^{K}{\mathbf{p}}_{i}\log{\mathbf{q}}_{i} (3)

iBOT [ 110 ] is a masked image modeling objective, wherein a localized patch is masked, and the model is tasked with predicting the tokenized representation of said masked patch. It adapts the DINO [ 11 ] objective from the image level into the patch level. We follow [ 76 ] to replace the softmax-centering method with the iterative Sinkhorn-Knopp (SK) algorithm [ 22 ] . We run SK for 3 iterations to build the prediction target.

Exponential Moving Average (EMA) is firstly introduced into self-supervised learning by MoCo [ 43 ] . We use EMA to encode crops as 𝒃 \bm{b} and to produce the targets for iBOT loss. We update the EMA model as ΞΈ e ​ m ​ a ← Ξ» ​ ΞΈ e ​ m ​ a + ( 1 βˆ’ Ξ» ) ​ ΞΈ \theta_{ema}\leftarrow\lambda\theta_{ema}+(1-\lambda)\theta , following a cosine schedule for Ξ» \lambda from 0.994 to 1 during training [ 39 , 68 ] . We find the EMA module not only increases the final performance, but also improves the training stability for long training schedules.

Multi-crop strategy is introduced by [ 10 ] as a smart way to improve computation efficiency, and is adopted in this paper. For these local crops, we only employ the contrastive loss, omitting the iBOT loss. Local crops are encoded only by the student network, and matched to global crops from the same caption encoded by the EMA model. Such reuse of global crops saves computation. For each image x x , where we generate a single global crop x g x^{g} alongside n n local crops x l x^{l} , the final loss can be expressed as follows: β„’ ⁑ ( x g ) + 1 n ​ βˆ‘ i = 1 n β„’ ⁑ ( x i l ) + β„’ i ​ B ​ O ​ T ​ ( x g ) \displaystyle\mathcal{L}(x^{g})+\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}(x^{l}_{i})+\mathcal{L}^{iBOT}(x^{g}) (4)

### 3.4 Implementation

Concept list. We concatenate class names from various datasets, including IN-1k [ 24 ] , IN-21k (we keep the most frequent 13k classes), Aircraft [ 60 ] , Cars [ 51 ] , DTD [ 18 ] , Flowers [ 64 ] , Pets [ 69 ] , Sun397 [ 98 ] , Caltech-101 [ 34 ] , Food-101 [ 7 ] , and Places-365 [ 108 ] . If the concept is a place ( i.e . SUN397 and Places) or a texture ( i.e . DTD), we only apply the c c –> caption template. For fine-grained classes such as pets or flowers, we employ GPT-4 to generate a consolidated list of probable backgrounds, rather than producing distinct lists for each specific class. We favor more frequent sampling from IN-1k, Food101, Cars, Aircraft, and Flowers.

Batches. For each training batch, we sample 2048 captions (except when noted), and use all of the 4 images generated by each caption. We generate 1 global and 4 local crops for each image. As a result, each batch contains 8192 global crops, which is similar with prior work [ 13 , 39 , 14 , 91 ] .

Masking. For the iBOT loss, we randomly choose 50 % 50\% images inside a batch to mask, and randomly mask 50 % 50\% of the tokens in each chosen image. We use 65536 prototypes. While the target from the EMA model is ascertained using the SK algorithm, we apply softmax normalization to the output of the student model.

Projection heads. We follow the design in MoCo v3 [ 14 ] and DINO [ 11 ] for the contrastive and iBOT loss heads, respectively, ensuring consistency with established methods.

Other hyper-parameters. We set the temperature in the contrastive loss to 0.08 0.08 . For the temperature used in the iBOT loss, we linearly increase it from 0.04 to 0.07 over 4000 iterations, and keep it as 0.07 afterwards, as in DINO [ 11 ] . Additionally, the weight decay parameter is incrementally adjusted from 0.04 to 0.2, adhering to a cosine schedule.

## 4 Experiment

We first perform an ablation study to evaluate the efficacy of various designs and modules within our pipeline. Then we proceed to scale up the volume of synthetic data.

### 4.1 Study different components

We analyze each component of SynCLR, and ablate their effectiveness in two measurements: (1) linear probing performance on IN-1k; (2) average accuracy of linear transfer on fine-grained datasets Aircraft [ 60 ] , Cars [ 51 ] , DTD [ 18 ] , Flowers [ 64 ] , Pets [ 69 ] , Sun397 [ 98 ] , Caltech-101 [ 34 ] , Food-101 [ 7 ] , and Pascal VOC [ 29 ] . For analysis conducted in this subsection, we train ViT-B/16 [ 28 ] models for 85000 iterations, and use the cls token as image representation.

Synthesize captions. Following [ 91 ] , we use cc12m [ 12 ] real captions as our baseline, which has 10M sentences. To synthesize captions, we design the following variants: (a) IN+h+Places randomly combines one IN class plus its hypernyms in WordNet graph, with one place class; (b) IN+Places+LLM uses the c , b ​ g c,bg –> caption in-context synthesis template with c c from IN and b ​ g bg from places; (c) IN+ourBG+LLM uses the background classes output by GPT-4, instead of Places; (d) ours means our full configuration specified in Section 3.1 . For each of the config, we generate 10M captions. If not enough, we do duplication.

Results are summarized in Table 2 , where we train both StableRep and SynCLR to avoid biases favored by a single method. Compared to a real caption dataset cc12m, simply concatenating IN and Places class names improves the ImageNet linear accuracy but reduces the fine-grained classification performance. Interestingly, naively asking Llama to combine IN and Places classes into captions yields the worst performance. Replacing random background from places with GPT generated background improves the accuracy. This shows the importance of synthesizing captions that follow the distribution of real captions, which were used to train the text-to-image model. Finally, our full configuration achieves the best accuracy on both ImageNet and fine-grained classification. Another advantage of our synthesis method is its scalability – scale up to hundreds of millions of captions with little duplication. In contrast, if we concatenate IN classes with Places classes, there are at most 365k unique captions.

Synthesize images. There are two major parameters in this process: number of images per caption and classifier free guidance scale. For the former, we find generating 4 images is almost able to reproduce StableRep [ 91 ] ’s performance (10 images) when using cc12m captions (ours 73.0 % \% v.s. StableRep 73.5 % \% on ImageNet). Thus we stick to 4. For guidance scale, we briefly find the contrastive loss is not very sensitive to CFG in a pilot study, as shown in Table 3 . Thus we stick to 2.5, similar as StableRep [ 91 ] .

ImageNet

Aircraft

Cars

DTD

Flowers

Pets

SUN397

Caltech-101

Food-101

VOC2007

Average

Model components. We present the improvement of accuracy brought by different modules in Table 4 . Compared to the baseline StableRep, adding a teacher EMA model improves the IN linear accuracy by 0.9 % \% . Further adding iBOT local objective or the multi-crop strategy increases the accuracy by 0.9 % \% and 1.9 % \% , respectively. Combining all of them results in our full SynCLR model, which achieves 78.8 % \% top-1 IN linear accuracy. The fine-grained classification performance follows a similar trend, and reaches 88.1 % \% . Besides, we test the transfer ability to semantic segmentation on ADE20k. The iBOT objective brings 1.0 more mIoU than multi-crop strategy, demonstrating the effectiveness of masked image modeling for dense prediction tasks.

Compare to SimCLR and supervised training. We compare the three different representation learning objectives shown in Figure 2 , which classify images at different levels of granularity. Since supervised cross-entropy training requires a fixed set of balanced classes (indeed both fixed set and balance are limitations of such method), we use the IN+ourBG+LLM configuration where we have 1000 balanced classes ( i.e . , each class has 40k images). The supervised training recipe follows [ 86 ] . For a fair comparison with SimCLR, we remove all unmatched modules ( i.e . , EMA, iBOT, and MC) to make sure that the only difference between SimCLR and our SynCLR is the classification granularity defined by the contrastive loss. For all of them, we do pre-training and then linear probing on the target dataset.

Table 5 presents the comparison. Our multi-positive objective, which defines images as the same class if they are generated by the same caption, achieves the best performance. It outperforms supervised cross-entropy training and SimCLR by 3.4 % \% and 11.7 % \% for top-1 accuracy on ImageNet linear evaluation, and by 3.5 % \% and 10.6 % \% on fine-grained classification tasks. Besides, our objective does not require balance between samples from a fixed set of classes, making it easier to scale up.

### 4.2 Scaling up

After we have ablated different components, we scale up our experiments. Specifically, we synthesize a dataset of 150M captions, called SynCaps-150M , from which we generate 600M images. We train both ViT-B/16 and ViT-L/14 (no SwiGLU [ 82 ] or LayerScale [ 92 ] ), and extend the training schedules to 500k steps with a batch size of 8192 captions. We use 224x224 resolution for all pre-training tasks.

We compare SynCLR with OpenAI’s CLIP [ 71 ] , OpenCLIP [ 17 ] , and DINO v2 [ 68 ] , which represent learning from data . We note that ViT-B/14 and ViT-L/14 from DINO v2 are distilled from a ViT-g [ 104 ] model, which makes DINO v2 advantageous in our comparison. We also includes StableRep [ 91 ] , which uses the hybrid paradigm.

ImageNet linear evaluation. For fair comparison, cls token from the last block is used as representation across all models (whereas in DINO v2, results are from concatenating multiple layers). As shown in Table 6 , SynCLR achieves 80.7 % \% with ViT-B and 83.0 % \% with ViT-L. This is similar as CLIP, but still lags behind DINO v2 by 3.2 % \% and 2.7 % \% , respectively, partially because of the extra distillation in DINO v2. We note SynCLR has already outperformed other self-supervised methods pre-trained directly on ImageNet-1k ( e.g . , DINO achieves 78.2 % \% with ViT-B/16 and iBOT reaches 81.0 % \% with ViT-L/16).

Fine-grained classification. On the nine fine-grained datasets we have evaluated in Table 6 , SynCLR achieves very similar average accuracy as DINO v2, e.g . , 89.0 % \% v.s. 89.0 % \% for ViT-B, and 90.1 % \% vs 90.4 % \% for ViT-L. Both SynCLR and DINO v2 have curated the pre-training data to include the distribution for these datasets (but in different ways and portions), and end up with similar performance. Interestingly, SynCLR outperforms others on Aircraft and Cars, possibly because we favor more frequent sampling towards them. This can be an advantage for synthetic data when we know what downstream tasks to solve. Besides, SynCLR outperforms CLIP and StableRep by 3.3 % \% and by 5.6 % \% for ViT-B, respectively.

Semantic segmentation. To evaluate the pixel-level understanding ability of SynCLR, we fine-tune the pre-trained models on ADE20k [ 109 ] , following the setup in [ 44 , 5 ] . UperNet [ 99 ] is used as the task layer, and we evaluate with a single-scale, i.e . 512x512. Besides CLIP and DINO v2, we also compare to self-supervised methods pre-trained on ImageNet, as well as BEiT v2 [ 70 ] , which distills from CLIP. Table 7 shows that our SynCLR outperforms self-supervised methods trained on IN-1k by a clear marge, e.g . , 4.3 higher mIoU than iBOT. Despite not involving a high resolution pre-training period like DINO v2 ( e.g . , 518x518), SynCLR performs similarly with DINO v2 (0.1 lower for ViT-B possibly because DINO v2 uses a smaller patch size of 14x14, but 0.2 higher for ViT-L). This suggests SynCLR pre-training is suitable for dense prediction tasks.

ImageNet fine-tuning. We evaluate the fine-tuning transfer ability of SynCLR on ImageNet. We compare with other state of the art self-supervised methods [ 14 , 5 , 100 , 44 , 27 , 4 , 110 ] in Table 8 . Our SynCLR outperforms models trained on ImageNet images or large scale image datasets. Specifically, SynCLR outperforms OpenCLIP ViT-L trained on Laion-2B, which is the dataset Stable Diffusion (the text2image model we used) is trained on. This contrasts with [ 78 , 30 ] , which shows that directly training a classifier on synthetic images yields bad classification accuracy. Our finding suggests synthetic images are good for training representations, which later can be easily adapted to a downstream task with limited amount of real data.

### 4.3 Further analysis

SynCLR requires a list of concepts C C to start off. But how will SynCLR transfer to concepts outside our list?

EuroSAT

GTSRB

Country211

MNIST

RESISC45

KITTI

Average

Generalize to unseen concepts. We consider additional datasets whose classes are outside the synthesis list, including EuroSAT [ 46 ] , GTSRB [ 85 ] , Country211 [ 71 ] , MNIST [ 54 ] , RESISC45 [ 16 ] , and KITTI distances [ 35 ] . These datasets, except for KITTI, are also outside the curation list of DINO v2. Therefore, it is also a generalization test for DINO v2. Table 9 shows the linear probing results. SynCLR outperforms DINO v2 by 1.5 % \% for ViT-B and 1.1 % \% for ViT-L, respectively. This suggests the representations of SynCLR generalize. CLIP outperforms SynCLR and DINO v2, with most gains coming from Country211. An explanation is CLIP’s training data contains similar country flags which are not in the training sets of SynCLR and DINO v2.

Given that both captions and images are synthesized, a natural question arises: how would CLIP training perform on such data?

Compare to CLIP training. We use the same data to train a ViT-B CLIP model. For each caption, we randomly choose 1 out of the 4 synthesized images in each iteration. Following common practice [ 71 ] , we train for 32 epochs with a batch size of 32768. This model achieves 44.4 % \% zero-shot accuracy on IN-1k. The SynCaps-150M row in Table 10 presents the linear probing results. Synthetic CLIP learns reasonably good features, reaching 78.3 % \% on IN-1k and 87.7 % \% on fine-grained datasets. However, SynCLR is still better.

We have also repeated our experiments with Laion-400M captions, i.e . , generate 4 images for each caption and train SynCLR and CLIP. The comparison between rows SynCaps-150M and Laion-400M in Table 10 suggests synthetic captions are also favorable on a large scale.