Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks — part 3/5

Cleaned-text code points [31468, 47034). Bibliography omitted here; source.md includes it.

We train our models with a mini-batch size of 2048/3072 (base/large) and an image size of 384 × \times 384 until reaching 3 billion effective training samples. Similar to [ 15 , 64 , 29 , 95 , 92 ] , we further conduct high-resolution tuning with an image size of 768 × \times 768 for 0.5 billion samples for the base model and 0.1 billion samples for the large model.

### 6.2 Zero-shot Evaluation Across Tasks

COCO Cap.

NoCaps

TextCaps

COCO Det.

Flickr30k

Refcoco

Refcoco+

Refcocog

Refcoco RES

We present a powerful vision foundation model that does not require task-specific supervised annotations for fine-tuning. The zero-shot performance of our model is shown in Table 4 . For image-level tasks, Florence-2-L achieves a 135.6 CIDEr score on the COCO caption benchmark [ 48 ] , utilizing less than 1% of the parameters compared to the 80B Flamingo [ 2 ] model (which has an 84.3 CIDEr score). For region-level grounding and referring expression comprehension tasks, Florence-2-L establishes a new record in zero-shot performance achieving a 5.7 improvement in Flickr30k [ 61 ] Recall@1, and approximately 4%, 8%, and 8% absolute improvements on Refcoco, Refcoco+, and Refcocog [ 94 ] , respectively, compared to the Kosmos-2 [ 60 ] model, which has 1.6B parameters. Additionally, our pre-trained model attains a 35.8% mIOU in the Refcoco referring expression segmentation (RES) [ 94 ] task, a capability not supported by prior foundation models.

### 6.3 Generalist Model with Public Supervised Data

We demonstrate the versatility and effectiveness of our model as a vision foundation that can be transferred to various downstream tasks. We fine-tune Florence-2 models by adding a collection of public datasets that cover image-level, region-level, pixel-level tasks, yielding one generalist model for various vision tasks. The details of the dataset collection are provided in Table 14 . Tables 5 and 6 compare our model with other state-of-the-art models. Our key findings are:

#### Simple design for strong performance.

Florence-2 demonstrates strong performance with standard multi-modality Transformer encoder-decoder without special designs, particularly for region-level and pixel-level tasks. For example, Florence-2-L outperforms PolyFormer [ 49 ] on both RefCOCO REC task and RES task by 3.0 Accuracy@0.5 and 3.54 mIOU respectively, where PolyFormer [ 49 ] adapts specifically designed regression-based prediction head for coordinates. Florence-2-L also outperforms previous SOTA method UNINEXT [ 84 ] on RefCOCO by 0.8 Accuracy@0.5, where UNINEXT [ 84 ] is based on advanced object detector Deformable DETR [ 100 ] and DINO [ 97 ] .

#### Competitive performance with fewer parameters.

Florence-2-L achieves competitive performance without the need for LLMs, showcasing efficiency in handling diverse tasks while maintaining a compact size. For instance, Florence-2-L attains a CIDEr score of 140.0 on the COCO Caption karpathy test split [ 30 ] , outperforming models with significantly more parameters, such as Flamingo (80B parameters, 138.1 CIDEr score).

#### Adaptable generalization across task levels.

Florence-2 demonstrates competitive performance across image-level, pixel-level, and region-level tasks, emphasizing its adaptability and effectiveness in addressing various challenges in computer vision and natural language processing. For example, in the TextVQA task, Florence-2-L sets a new state-of-the-art performance with an accuracy of 81.5 without any external OCR token input, surpassing previous SOTA methods [ 15 , 12 ] .

These achievements emphasize Florence-2 ’s efficiency in handling diverse tasks while maintaining a compact size, making it a unique and valuable asset in the ever-evolving landscape of AI research and applications.

### 6.4 Downstream Tasks Fine-tuning

In this section, we investigate the performance of our single model fine-tuning on downstream tasks. This experiment highlights the superiority of Florence-2 pre-training over previous approaches, as it demonstrates the effectiveness of the learned universal image representation. We use the base size model with about 80M parameters in our experiments to ensure fair comparison with other methods.

#### Object detection and segmentation.

We conduct COCO object detection and instance segmentation [ 48 ] experiments with Mask R-CNN [ 26 ] , and COCO object detection [ 48 ] experiments with DINO [ 97 ] to further demonstrate the effectiveness of Florence-2 pre-training. We train on the train2017 split and evaluate on the val2017 split.

For Mask R-CNN [ 26 ] experiments, we follow the common setup used in [ 51 , 97 ] , we use the standard 1 × \times (12 epochs) schedule with multi-scale training for all experiments. The learning rate is stepped down by a factor of 0.1 at the 67% and 89% of training epochs. We do not use any additional augmentation (such as random crop, mosaic, etc) or optimization techniques (such as EMA, weight normalization) during training to ensure a fair comparison. We do not use any test time augmentation (TTA) either. Thanks to the strong universal representation learned by Florence-2 pre-training, we do not require longer training epochs, such as 36 epochs in [ 81 , 51 , 86 , 85 ] , or 100 epochs in [ 46 ] , to achieve better results.

For DINO [ 97 ] experiments, we train DINO-4scale [ 97 ] detector for 12 epochs (1 × \times ) using the same data augmentation strategy as employed by [ 7 ] .

First, our base model achieves a strong performance improvement compared to other approaches. As shown in Table 7 , our DaViT-B model pre-trained by Florence-2 surpasses previous best base model (ConvNext v2-B), which is pre-trained by FCMAE [ 81 ] , by 0.7 A ​ P b AP_{b} using Mask RCNN. Importantly, while ConvNeXt v2-B leverages a 3 × \times schedule (36 epochs), our model efficiently employs a 1 × \times schedule (12 epochs) thanks to our powerful pre-trained universal representation. For DINO framework, our model significantly outperforms the ViT-B, achieving a notable improvement of 4.2 AP.

Second, our pre-training demonstrates higher training efficiency. As shown in Table 8 and Figure 6 , compared to the model with supervised ImageNet-1k pre-training, our model with Florence-2 pre-training achieves 4x efficiency and a significant improvement of 6.9 AP and 5.5 AP with Mask-RCNN and DINO framework, respectively.

Third, our pre-training provides a good generic representation without extensive fine-tuning. Table 8 indicates that the models with Florence-2 pre-training maintains competitive performances when the first two stages are frozen with only 0.3 and 0.2 drops for Mask-RCNN and DINO, respectively. Moreover, our approach with completely frozen backbone can outperform the model with supervised ImageNet-1k pre-training by 1.6 and 2.4 for Mask-RCNN and DINO.

#### Semantic segmentation.

We conduct semantic segmentation experiments with UperNet [ 82 ] framework on ADE20k [ 98 ] dataset. We mostly follow the training and evaluation protocols from Swin [ 51 ] . Specifically, we use input size 512 × \times 512 and train the model for 40k iterations with a batch size of 64. We adopt the AdamW [ 54 ] optimizer with the optimal learning rate searched from {8e-4,4e-4,2e-4,1e-4}.

Our results show a similar trend to the object detection experiments. As illustrated in Table 9 , our base model outperforms the previous SoTA model, which is BEiT pre-trained ViT-B [ 4 ] , by 1.3 and 1.4 points in single-scale and multi-scale testing protocol, respectively. With the same backbone architecture of DaViT-B [ 20 ] , Florence-2 pre-trained model achieves a remarkable improvement of 4.9 points and 4 × \times efficiency compared to the ImageNet-1k pre-trained counterpart as demonstrated in Tables 8 and 6 .

### 6.5 Ablation Studies

#### Multitask transfer.

In this study, we aimed to identify the most effective pre-trained model for transfer learning across various downstream tasks in computer vision. We compared three different models, each pre-trained on a different combination of tasks: • Image-level Model: pre-trained on image-level tasks only

• Image-Region Model: pre-trained on image-level and region-level tasks

• Image-Region-Pixel Model: pre-trained on image-level, region-level, and pixel-level tasks

For pre-training, we optimize all models for the same number of effective samples (72M) on a subset of our FLD-5B dataset.

These models are then transferred to a combined dataset with four downstream tasks, each representing a different level of task granularity: COCO caption (image-level task), COCO object detection (region-level task), Flickr30k grounding (region-level task), RefCOCO referring segmentation (pixel-level task).

The results are shown in Figure 7 . The results demonstrate that Image-Region-Pixel Model, pre-trained on all three levels of tasks, consistently demonstrated competitive performance across the four downstream tasks.

For the COCO caption task, Image-Region-Pixel Model initially performs worse than Image-level Model and Image-Region Model but eventually achieve a final performance (133.4 CIDEr) that is only slightly worse than the other models (134.6 CIDEr).

For the COCO object detection task, Image-Region-Pixel Model outperforms Image-level Model by a significant margin (28.3 vs. 0.1) and was only slightly worse than Image-Region Model (29.7).

For the Flickr30k grounding task, Image-Region-Pixel Model shows strong performance (78.1 recall@1), comparable to Image-Region Model (79.1 recall@1) and significantly better than Image-level Model (62.0 recall@1).

For the RefCOCO referring segmentation task, Image-Region-Pixel Model clearly outperforms both Image-level Model and Image-Region Model, achieving the highest performance (31.6 mIoU) compared to the other models (28.4 and 18.2 mIoU).

Our findings suggest that the Image-Region-Pixel Model, which is pre-trained on tasks at the image, region, and pixel levels, is the most effective base model for transfer learning across various computer vision tasks. This model shows strong performance on all four downstream tasks we evaluated, and consistently outperforms the Image-level Model and matches or exceeds the Image-Region Model in performance. By pre-training a model on tasks at different levels of granularity, we can ensure that the base model is better prepared to handle a diverse range of downstream tasks, offering a versatile and robust solution for transfer learning in computer vision.

#### Model scaling.

We aimed to investigate the impact of increasing model capacity on zero-shot performance on various downstream tasks in computer vision. We compared two models: Florence-2-B and Florence-2-L , which have 232M and 771M parameters, respectively. The model architectures are described in Table 15 . We show the zero-shot performance on four downstream tasks in Table 10 . The large model clearly outperforms the base model across various downstream tasks.

#### Data scaling.

We conducted experiments to study how zero-shot performance on various computer vision tasks is affected by the scale of pre-training data. We used four different data sizes for pre-training: 0.12M, 0.36M, 1.2M, and 12M images. All models were trained with the same effective sample size (72M) on a subset of FLD-5B data.

Table 11 presents the zero-shot performance results on COCO caption, COCO object detection, Flickr30k grounding, and RefCoco referring segmentation (RES) tasks. We can observe a trend of improved zero-shot performance on the downstream tasks as the pre-training data size increases (except for RES, 1.2M data has slightly better performance compared to 12M).

Our experiments on data scaling demonstrate that larger pre-training data sizes generally lead to improved zero-shot performance across a variety of downstream tasks in computer vision. This finding suggests that investing in larger pre-training datasets can provide a more effective and versatile foundation for handling a wide range of downstream tasks.

Our approach to scaling data is significantly more efficient than relying solely on human annotations, as most of the annotation generation is performed using model inference. By leveraging specialist models to generate annotations, we can substantially reduce the time and cost associated with manual annotation efforts, which often involve labor-intensive processes and may be subject to human errors or inconsistencies.

Furthermore, utilizing model-generated annotations enables us to scale the pre-training datasets more rapidly and efficiently, allowing us to explore the impact of larger data sizes on model performance across various downstream tasks in computer vision. This not only facilitates the development of more effective and versatile foundation models but also ensures that the annotation process remains sustainable and scalable as the need for high-quality labeled data continues to grow.

In summary, our data scaling approach offers a more efficient alternative to traditional human annotation methods by harnessing the power of specialist models for annotation generation. This strategy enables us to accelerate the pre-training process, optimize model performance, and effectively manage the ever-increasing demand for labeled data in the field of computer vision.

#### Training settings.

We analyze the basic model training settings for the two primary components of our model, namely the vision encoder and the multi-modality encoder-decoder. The experiment results are presented in Table 12

We observe that freezing the vision encoders does not affect the performance on tasks that require image-level understanding, but it significantly degrades the performance on tasks that require region-level or pixel-level understanding (e.g., AP on COCO object detection drops from 19.7 to 6.9). Previous methods for pre-training vision foundation models mainly focus on image-level tasks (e.g., image classification [ 38 , 27 ] , image-text contrastive learning [ 64 , 95 ] ), which may not provide them with sufficient region-level and pixel-level skills for downstream tasks. Therefore, it is important to unfreeze the vision backbone, enabling it to learn region-level and pixel-level features for various downstream tasks.

The effect of language pre-training weights on multi-modal encoder-decoder tasks varies depending on the task. Tasks that require more text understanding, such as captioning and grounding, benefit slightly from using language pre-training weights (e.g., COCO caption, Flickr30k grounding). Tasks that are mostly vision-focused, such as object detection and region segmentation, do not gain much from using language pre-training weights (for COCO object detection, the gain is only 0.1; for RES tasks, which use only localization tokens, the drop is 2.91 mIOU).

We investigate the effects of different training configurations on the performance of a foundation model in region-level and pixel-level tasks. We find that unfreezing the vision backbone is crucial for enhancing the model’s ability to learn from regions and pixels, which is beneficial for transferring to various downstream tasks. Moreover, we observe that using language pre-training weights can help the model in tasks that require text understanding, but have less impact on tasks that are purely vision-based. These results offer useful guidance for choosing the best training settings for different computer vision tasks.

## 7 Related Works

### 7.1 Vision-Language Foundation Models