Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks — part 1/5

Cleaned-text code points [0, 15518). Bibliography omitted here; source.md includes it.

# Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks

###### Abstract

We introduce Florence-2 , a novel vision foundation model with a unified, prompt-based representation for a variety of computer vision and vision-language tasks. While existing large vision models excel in transfer learning, they struggle to perform a diversity of tasks with simple instructions, a capability that implies handling the complexity of various spatial hierarchy and semantic granularity. Florence-2 was designed to take text-prompt as task instructions and generate desirable results in text forms, whether it be captioning, object detection, grounding or segmentation. This multi-task learning setup demands large-scale, high-quality annotated data. To this end, we co-developed FLD-5B that consists of 5.4 billion comprehensive visual annotations on 126 million images, using an iterative strategy of automated image annotation and model refinement. We adopted a sequence-to-sequence structure to train Florence-2 to perform versatile and comprehensive vision tasks. Extensive evaluations on numerous tasks demonstrated Florence-2 to be a strong vision foundation model contender with unprecedented zero-shot and fine-tuning capabilities.

## 1 Introduction

In the realm of Artificial General Intelligence (AGI) systems, there has been a notable shift towards utilizing pre-trained, versatile representations, acknowledged for task-agnostic benefits accross diverse applications. This trend is evident in natural language processing (NLP), where advanced models [ 5 , 19 , 43 , 66 , 6 , 65 ] show adaptability with comprehensive knowledge spanning various domains and tasks with simple instructions. The success of NLP motivates a parallel approach in computer vision.

Universal representation for diverse vision-related tasks presents unique challenges, notably the need for comprehensive perceptual abilities. Unlike NLP, which deals mainly with text, computer vision requires handling intricate visual data like object location, masked contours, and attributes. Attaining universal representation in computer vision demands adept management of a spectrum of complex tasks, organized two-dimensionally as illustrated in Figure 1 : • Spatial Hierarchy : The model must discern spatial details across varying scales, understanding image-level concepts and fine-grained pixel specifics. Accommodating the intricate spatial hierarchy within vision demands the model’s proficiency in handling diverse levels of granularity.

• Semantic Granularity : Universal representation in computer vision should span a spectrum of semantic granularity. The model transitions from high-level captions to nuanced descriptions, enabling versatile understanding for diverse applications.

This pursuit is characterized by distinctiveness and substantial challenges. A key hurdle is the scarcity of comprehensive visual annotations , hindering the development of a foundational model capable of capturing the intricate nuances of spatial hierarchy and semantic granularity. Existing datasets, such as ImageNet [ 18 ] , COCO [ 48 ] , and Flickr30k Entities [ 61 ] , tailored for specialized applications, are extensively labeled by humans. To overcome this constraint, it is imperative to generate extensive annotations for each image on a larger scale.

Another challenge is the absence of a unified pre-training framework with a singular network architecture that seamlessly integrates spatial hierarchy and semantic granularity in computer vision. Traditional models excel in tasks like object detection [ 26 , 97 ] , semantic segmentation [ 16 , 82 ] , and image captioning [ 45 , 78 ] with task-specific design. However, it is essential to develop a comprehensive, unified model that is capable of adapting across various vision tasks in a task-agnostic manner, even accommodating new tasks with minimal or no task-specific fine-tuning.

The model Florence [ 95 ] pioneers the integration of spatial, temporal, and multi-modal aspects in computer vision through unified pre-training and network architecture. The first evolutionary version [ 95 ] excels in transfer learning via pre-training with noisy text-image pairs and task-specific fine-tuning using specialized adapters. However, it relies on large task-specific datasets and adapters, leaving gaps in addressing the above dual key challenges.

In this paper, we introduce Florence-2 , a universal backbone achieved through multitask learning with extensive visual annotations. This results in a unified, prompt-based representation for diverse vision tasks, effectively addressing the challenges of limited comprehensive data and the absence of a unified architecture.

Multitask learning necessitates large-scale, high-quality annotated data. Our data engine, instead of relying on labor-intensive manual annotation, autonomously generates a comprehensive visual dataset called FLD-5B , encompassing a total of 5.4B annotations for 126M images. This engine consists of two efficient processing modules. The first module uses specialized models to collaboratively and autonomously annotate images, moving away from the traditional single and manual annotation approach. Multiple models work together to reach a consensus, reminiscent of the wisdom of crowds concept [ 89 , 80 , 33 ] , ensuring a more reliable and unbiased image understanding. The second module iteratively refines and filters these automated annotations using well-trained foundational models.

By utilizing this extensive dataset, our model employs a sequence-to-sequence (seq2seq) architecture [ 76 , 17 , 66 , 19 ] , which integrates an image encoder and a multi-modality encoder-decoder. This design accommodates a spectrum of vision tasks without the need for task-specific architectural modifications, aligning with the ethos of the NLP community for versatile model development with a consistent underlying structure. All annotations in the dataset FLD-5B , are uniformly standardized into textual outputs, facilitating a unified multi-task learning approach with consistent optimization with the same loss function as the objective. The outcome is a versatile vision foundation model, Florence-2 , capable of performing a variety of tasks, such as object detection, captioning, and grounding, all within a single model governed by a uniform set of parameters. Task activation is achieved through textual prompts, reflecting the approach used by Large Language Models (LLMs) [ 65 ] .

Our approach attains a universal representation, demonstrating broad applicability across various visual tasks. Key results include:

• As a versatile vision foundation model, Florence-2 achieves new state-of-the-art zero-shot performance in tasks such as captioning on COCO [ 48 ] , visual grounding on Flick30k [ 61 ] , and referring expression comprehension on RefCOCO/+/g [ 31 , 93 , 56 ] .

• After fine-tuning with public human-annotated data, Florence-2 , despite its compact size, competes with larger specialist models. Notably, the fine-tuned Florence-2 establishes new state-of-the-art results on the benchmarks on RefCOCO/+/g.

• The pre-trained Florence-2 backbone enhances performance on downstream tasks, e.g . COCO object detection and instance segmentation, and ADE20K semantic segmentation, surpassing both supervised and self-supervised models. Compared to pre-trained models on ImageNet, ours improves training efficiency by 4 × \times and achieves substantial improvements of 6.9, 5.5, and 5.9 points on COCO [ 48 ] and ADE20K [ 98 ] datasets, using Mask-RCNN [ 26 ] , DINO [ 97 ] , and UperNet [ 82 ] frameworks respectively.

## 2 Rethinking Vision Model Pre-training

In pursuit of a versatile vision foundation model, we revisit three predominant pre-training paradigms: supervised ( e.g . , ImageNet classification [ 18 ] ), self-supervised ( e.g . , SimCLR [ 9 ] , MoCo [ 25 ] , BEiT [ 4 ] , MAE [ 24 ] ), and weakly supervised ( e.g . , CLIP [ 64 ] , Florence [ 95 ] , SAM [ 32 ] ). Each paradigm captures unique aspects of visual data but is inherently limited by the constraints of single-task learning frameworks. Supervised pre-training excels in object recognition but lacks adaptability [ 38 ] ; self-supervised algorithms reveal intricate features but may overemphasize certain attributes [ 8 ] ; weakly supervised methods leverage unstructured textual annotations but yield only image-level understanding [ 64 ] . To build a unified vision foundation model suitable for various applications, we must explore innovative pre-training strategies that overcome single-task limitations and integrate both textual and visual semantics.

Image understanding necessitates capturing multiple levels of granularity, from global semantics to local details, and comprehending spatial relationships between objects and entities in their semantic context. To address these core aspects of image understanding, our approach incorporates a diverse set of annotations, effectively capturing visual understanding nuances and bridging the gap between vision and language understanding.

### 2.1 Comprehensive Multitask Learning

To develop a versatile vision foundation model, we formulate a range of multitask learning objectives, each tailored to address specific aspects of visual comprehension. These objectives align with our predefined criteria: spatial hierarchy and semantic granularity, inspired by recent research on multitask learning [ 2 , 79 , 55 , 15 , 12 , 14 ] . Our multitask learning approach incorporates three distinct learning objectives, each addressing a different level of granularity and semantic understanding:

• Image-level understanding tasks capture high-level semantics and foster a comprehensive understanding of images through linguistic descriptions [ 13 , 91 , 18 , 34 ] . They enable the model to comprehend the overall context of an image and grasp semantic relationships and contextual nuances in the language domain. Exemplar tasks include image classification, captioning, and visual question answering.

• Region/pixel-level recognition tasks facilitate detailed object and entity localization within images, capturing relationships between objects and their spatial context. Tasks include object detection, segmentation, and referring expression comprehension.

• Fine-grained visual-semantic alignment tasks require fine-grained understanding of both text and image. It involves locating the image regions that correspond to the text phrases, such as objects, attributes, or relations. These tasks challenge the ability to capture the local details of visual entities and their semantic contexts, as well as the interactions between textual and visual elements.

By combining these three learning objectives in a multitask learning framework, our foundation model learns to handle different levels of detail and semantic understanding. This strategic alignment enables our model to deal with various spatial details, distinguish levels of detail in understanding, and go beyond surface-level recognition—ultimately learning a universal representation for vision understanding.

## 3 Model

We present the foundation model Florence-2 , designed for universal representation learning, capable of handling various vision tasks with a single set of weights and a unified architecture. As depicted in Figure 2 , Florence-2 employs a sequence-to-sequence learning paradigm [ 77 ] , integrating all tasks, described in Section 2 , under a common language modeling objective. The model takes images coupled with task-prompt as task instructions, and generates the desirable results in text forms. It uses a vision encoder to convert images into visual token embeddings, which are then concatenated with text embeddings and processed by a transformer-based multi-modal encoder-decoder to generate the response. In the following sections, we will provide a detailed explanation of each model component.

#### Task formulation.

We adopt a sequence-to-sequence framework [ 77 , 55 , 15 , 10 ] to address various vision tasks in a unified manner. As shown in Table 13 , we formulate each task as a translation problem: Given an input image and a task-specific prompt, we generate the corresponding output response. Depending on the task, the prompt and response can be either text or region:

• Text : When the prompt or answer is plain text without special formatting, we maintain it in our final sequence-to-sequence format.

• Region : For region-specific tasks, we add location tokens to the tokenizer’s vocabulary list, representing quantized coordinates. We create 1,000 1,000 bins, similar to [ 10 , 55 , 11 , 79 ] , and represent regions using formats tailored to task requirements: – Box representation ( x 0 , y 0 , x 1 , y 1 ) (x_{0},y_{0},x_{1},y_{1}) : Utilized in tasks such as object detection and dense region captioning, with location tokens corresponding to the box coordinates. The location tokens are the coordinates of the top-left and bottom-right corners of the box.

– Quad box representation ( x 0 , y 0 , … , x 3 , y 3 ) (x_{0},y_{0},...,x_{3},y_{3}) : For text detection and recognition tasks, using location tokens for each coordinate of the quadrilateral enclosing the text. The location tokens are the coordinates of each corner of the quad box, starting from the top-left and going clockwise.

– Polygon Representation ( x 0 , y 0 , … , x n , y n ) (x_{0},y_{0},...,x_{n},y_{n}) : For referring segmentation tasks, with location tokens representing the vertices of the polygon. The location tokens are the coordinates of the vertices of the polygon, in clockwise order.

By extending the tokenizer’s vocabulary to include location tokens, we enable the model to process region-specific information in a unified learning format. This eliminates the need to design task-specific heads for different tasks and allows for a more data-centric approach.

#### Vision encoder.

We employ DaViT [ 20 ] as the vision encoder. It processes an input image 𝐈 ∈ ℝ H × W × 3 \mathbf{I}\in\mathbb{R}^{H\times W\times 3} (with H H and W W denoting height and width, respectively) into flattened visual token embeddings 𝐕 ∈ ℝ N v × D v \mathbf{V}\in\mathbb{R}^{N_{v}\times D_{v}} , where N v N_{v} and D v D_{v} represent the number and dimensionality of vision tokens, respectively.

#### Multi-modality encoder decoder.

We use a standard encoder-decoder transformer architecture to process visual and language token embeddings. We first obtain prompt text embeddings 𝐓 p ​ r ​ o ​ m ​ p ​ t ∈ 𝐑 N t × D \mathbf{T}_{prompt}\in\mathbf{R}^{N_{t}\times D} using our extended language tokenizer and word embedding layer [ 43 ] . Then, we concatenate vision token embeddings with prompt embeddings to form the multi-modality encoder module input, 𝐗 = [ 𝐕 ′ , 𝐓 p ​ r ​ o ​ m ​ p ​ t ] \mathbf{X}=[\mathbf{V}^{\prime},\mathbf{T}_{prompt}] , where 𝐕 ′ ∈ ℝ N v × D \mathbf{V}^{\prime}\in\mathbb{R}^{N_{v}\times D} is obtained by applying a linear projection and LayerNorm layer [ 3 ] to 𝐕 \mathbf{V} for dimensionality alignment.

#### Optimization objective.

Given the input x x combined from the image and the prompt, and the target y y , we use the standard language modeling with cross-entropy loss for all the tasks.

ℒ = − ∑ i = 1 | y | l o g P θ ( y i | y < i , x ) , \mathcal{L}=-\sum_{i=1}^{|y|}logP_{\theta}(y_{i}|y_{<i},x), (1) where θ \theta are the network parameters, | y | |y| is the number of target tokens.

## 4 Data Engine