Learning Vision from Models Rivals Learning Vision from Data — part 3/3

Cleaned-text code points [31371, 38789). Bibliography omitted here; source.md includes it.

PCA visualization. Following the method used in DINO v2 [ 68 ] , we present visualizations derived from the Principal Component Analysis (PCA) conducted on patch features extracted using our model SynCLR. As depicted in Figure 5 , a comparative analysis is conducted between SynCLR and DINO v2, both utilizing the ViT-L/14 architecture. The results demonstrate that SynCLR effectively accentuates the features of cars and planes, while efficiently minimizing background clutter.

Scaling behavior. We train ViT-BViT-L models using random subsets of varying sizes: 1M, 3M, 10M, 40M, and the comprehensive 150M (measured in the number of captions). These models are trained over a reduced schedule of 300,000 steps and utilizes a smaller batch size of 2048. The outcomes of linear probing are illustrated in Figures 6 and 7 . These results indicate that the ViT-B model delivers robust performance at the 10M scale, with diminishing returns observed beyond this point. In contrast, the ViT-L model exhibits a greater demand for data (i.e., it underperforms ViT-B at the 3M scale) and scales better with data.

## 5 Discussions and Conclusion

Why learn from generative models? One compelling reason is that a generative model can act like hundreds of datasets simultaneously . Traditionally, researchers have to spend separate effort collecting datasets for different image categories, e.g . , cars, flowers, cats, dogs, and so on. DINO v2 [ 68 ] achieves robust representations by curating and amalgamating numerous such datasets. Such a process introduces complexities such as clustering and search challenges. In contrast, advanced text-to-image generative models like Stable Diffusion [ 72 ] or Imagen [ 77 ] have the capability to generate many diverse datasets. These models provide the flexibility to produce an infinite number of samples (albeit finite diversity) and control the generation process through textual input. Thus, generative models offer a convenient and effective method for curating training data. In our study, we harness this advantage to synthesize images encompassing a broad spectrum of visual concepts.

What can be further improved? Enhanced caption sets can be achieved through various methods, such as enriching the set of in-context examples, optimizing the sampling ratios among different concepts, and utilizing more advanced LLMs. In terms of the learning process, one approach is to distill knowledge from a larger model, and incorporate an additional high-resolution training phase (as discussed in [ 68 ] ) or an intermediate IN-21k fine-tuning stage (as per [ 5 , 70 ] ). Regarding architectural improvements, the integration of SwiGLU and LayerScale, coupled with superior model initialization strategies (referenced in [ 32 ] ), can be beneficial. However, due to limited resources and the scope of this paper not being focused on achieving the highest possible metrics, we propose these areas for further exploration in future research endeavors.

In summary, this paper studies a new paradigm for visual representation learning – learning from generative models . Without using any real data, SynCLR learns visual representations that are comparable with those achieved by state of the art general-purpose visual representation learners.

## Appendix A Concept Sampling

The concepts used to synthesize captions are randomly sampled from the names of various datasets. The rough ratios are presented in Table 11 . It is likely that different combinations of these ratios lead to different results, but we do not optimize over this dimension. For example, we simply concatenate IN-21k concepts with the classes of other datasets ( e.g . , Caltech-101, Pets), and do uniform sampling from the concatenated list. This may lead to under-sampling for other datasets, as the list is dominated by IN-21 classes.

## Appendix B Implementation Details

### B.1 Pre-training

The setting for our final long schedule training in Section 4.2 is summarized in Table 12 , where models are trained for 500k steps with a batch size of 8192 captions. For ablation study present in Section 4.1, we only train for 85k steps with a batch size of 2048 captions; for the scaling plots in Section 4.3, we train all models for 300k steps with a batch size of 2048.

### B.2 ImageNet linear probing

We use the cls token from the final transformer block as the image representation. This is different from DINO v2, which tries to concatenate cls token with average pooled patch tokens and sweep over whether to use multiple layers.

We follow prior work [ 14 , 11 ] to train the linear classifier. It has been generally observed that regularization such as weight decay hurts the performance [ 88 , 43 ] . Therefore, we set weight decay as 0, and we sweep the b ​ a ​ s ​ e ​ _ ​ l ​ r base\_lr over { 0.1 , 0.2 , 0.5 , 1 , 2 , 5 , 10 , 20 , 50 } × 10 − 2 \{0.1,0.2,0.5,1,2,5,10,20,50\}\times 10^{-2} .

### B.3 End-to-End ImageNet fine-tuning

Following common practice [ 5 , 44 ] , we append a linear classifier on top of the CLS token of the last transformer block, and fine-tune the whole network. We use layer-wise lr decay [ 20 ] . Table 14 shows the settings.

### B.4 Semantic segmentation on ADE20k

We conduct the experiments on ADE20k [ 109 ] . Following [ 5 , 44 ] , we use UperNet [ 99 ] as the task adaptation layer. We use the common single-scale [ 5 ] setup, with a resolution of 512 × \times 512 for models with a patch size of 16 × \times 16 and a resolution of 518 × \times 518 for models with a patch size of 14 × \times 14. The hyper-parameters are summarized in Table 15 .

### B.5 Fine-grained linear classification

Following prior works [ 13 , 39 ] , we train a regularized multi-nomial logistic regression model upon the output CLS token. In training and testing, we do not perform any data augmentation; images are resized to 224 pixels along the shorter side, followed by a center crop of 224 × \times 224. We minimize the cross-entropy objective using L-BFGS with ℓ 2 \ell_{2} -regularization. We select this ℓ 2 \ell_{2} -regularization constant on the validation set over 45 logarithmically spaced values between 10 − 6 10^{-6} and 10 5 10^{5} . The maximum number of L-BFGS iterations is set to 1000 1000 , similar as that in DINO v2 [ 68 ] .

## Appendix C In-context Learning Examples

All of the three types of in-context examples are summarized in Table 16 , Table 17 , and Table 18 , respectively.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .