Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks — part 4/5

Cleaned-text code points [47034, 62124). Bibliography omitted here; source.md includes it.

Recent vision-language pre-training models [ 64 , 29 , 95 ] have demonstrated impressive zero-shot transfer abilities to vision-language alignment and image classification tasks, thanks to the alignment of vision and text embeddings extracted from respective encoders through contrastive learning objectives [ 74 , 58 ] . These models ( e.g . , [ 95 ] ), trained on weakly large-scale image-text data, have been further extended to more downstream tasks such as object detection, achieving state-of-the-art performance with task-specific adaptation heads.

In contrast, other studies [ 92 , 45 , 78 , 2 ] propose using a multi-modality decoder to predict text in an autoregressive manner with language modeling pre-training objectives. Techniques for fusing vision and language embeddings vary: GIT [ 78 ] concatenates vision and text tokens as decoder input and designs a casual attention mask, CoCa [ 92 ] uses attentional poolers with learnable queries to select task-specific vision representations which are then cross-attended via the decoder, and Flamingo [ 2 ] pools a fixed number of vision tokens with a Perceiver Resampler and adds new learnable cross-attention layers to the decoder while freezing the pre-trained vision encoder and text decoder.

Beyond image captioning pre-training task, some research [ 55 , 15 , 79 ] attempts to formulate more vision tasks in a unified sequence-to-sequence learning paradigm, including object detection and image segmentation. Customized special tokens accommodate representations beyond pure text, such as bounding boxes [ 55 , 10 , 79 ] . This approach uses the same architecture for pre-training and downstream tasks, potentially using the same set of weights for all tasks. Our method, which falls into this category, aims to obtain foundation models that understand dense information beyond simple image-level captions. It shares the same encoder-decoder design as other multi-modality encoder-decoder models [ 15 , 55 ] adapted for sequence-to-sequence learning, but uses our built large-scale comprehensive annotation data instead of combining existing sparse annotated data.

### 7.2 Vision Datasets

#### Comprehensive annotations.

The quest for comprehensive understanding of visual scenes, the holy grail of computer vision [ 36 ] , has evolved from focusing on individual datasets each targeting a single perspective, e.g . , image classification [ 18 ] , to providing multi-perspective [ 48 , 36 , 40 ] , comprehensive annotations for every visual data point. Notable datasets like MS-COCO [ 48 , 13 ] and Visual Genome [ 36 ] integrate various types of annotations, enabling richer understanding in spatial and semantic granularities and better model interactions across annotations. However, due to the high cost of human verification, these annotations are limited in size. Our datasets, while large-scale, maintain comprehensive annotations covering text, region-text pairs, and text-phrase-region triplets, with reduced human involvement.

#### Scalable annotations.

: Over the past decade, vision datasets have rapidly scaled up from thousands [ 42 , 37 ] to billion examples [ 29 , 96 ] to encompass more visual concepts for better generalization. This shift is evident in recent foundation models that employ massive quantities of data [ 5 ] . These large datasets typically collect images from the web and parse noisy annotations from the corresponding metadata, such as category label from query [ 75 , 96 ] , short description from alt-text [ 64 , 29 ] , as well as detailed description from interleaved text [ 2 , 41 ] . Despite their diversity, these annotations suffer from randomness and limited types ( i.e . , texts only). Some works [ 32 , 45 ] attempt to scale up annotations using pseudo-label generation with iteratively trained models, which offer higher quality without significant diversity loss. Our data pipeline extends these large-scale, web-crawled noisy annotations with higher-quality, autonomous annotations generated from multiple specialist models. The pipeline iteratively refines labels and completes missing pieces, resulting in a scalable and comprehensive dataset for learning a unified visual representation.

## 8 Conclusion

The Florence Project endeavors to develop a foundational vision model endowed with a diverse array of perceptual capabilities, encompassing spatial hierarchy and semantic granularity. To this end, we construct FLD-5B dataset containing an extensive collection of 126M images paired with 5B comprehensive annotations, which are collected by the Florence data engine. Subsequently, we pre-train Florence-2 on this rich dataset through comprehensive multitask learning in a unified manner. Florence-2 has exhibited remarkable zero-shot capabilities that extend across a wide spectrum of visual tasks, such as captioning, object detection, visual grounding, and referring segmentation, among others. The experimental findings underscore the potency of the universal representation pre-trained by Florence-2 , revealing its substantial contributions to the enhancement of a multitude of downstream tasks.

#### Acknowledgment.

We would like to express our heartfelt gratitude to all the contributors from the Azure AI team who worked on the Florence project. We sincerely appreciate Misha Bilenko for the invaluable guidance and support. Our thanks are extended to Yi-Ling Chen, Mengchen Liu, Yen-Chun Chen and Dongdong Chen for engaging in helpful discussions and to Yunsheng Li for their assistance with segmentation annotations. Deep appreciation is also expressed to Qingfen Lin, Ryan Menezes, Kuan Lu, Gabe Blanco, Shohei Ono, Ping Jin, Jiahe Zhou, Xiong Qiao, Tong Bai, Xingchao Peng, Pei Guo, Lihang Li for providing valuable feedback in downstream applications discussions. Special thanks to Cha Zhang, Jinyu Li, Min Gao, Christina Sun, Oliver Ernst, Kevin Pan, Mei Gao for their work on data annotation support and insightful discussions in data pipeline. Furthermore, we would like to thank Thomas Soemo, Nguyen Bach for their constructive feedback.

## Appendix A Supported Tasks and Annotations in Florence-2

## Appendix B Supervised Data Collection for Generalist Model Fine-tuning

## Appendix C Model Configuration

## Appendix D More Examples of Annotations in FLD-5B

## Appendix E Qualitative Evaluation and Visualization Results

### E.1 Detailed Image Caption

The image shows the interior of a grocery store with a large variety of fruits and vegetables on display. The store has a high ceiling with green and white striped awnings, and the floor is made of wood. There are rows of wooden crates filled with different types of fruits, including apples, oranges, lemons, limes, and limes. The crates are arranged in neat rows, and there are price tags attached to each crate. In the background, there are shelves stocked with various items such as fruits, vegetables, and other grocery items. The overall atmosphere of the store is bright and colorful, with a sense of freshness and abundance. The image is a movie poster for the film "Harry Potter and the Sorcerer’s Stone" starring Ron Weasley. The poster features a young boy, Ron Weasley, standing in front of a dark background with a castle-like structure in the background. He is wearing a school uniform with a Gryffindor crest on his jacket and a red tie. He has blonde hair and is looking directly at the camera with a serious expression on his face. The title of the film is written in white text at the top of the poster, with the tagline "20 years of movie magic" written in smaller text below. The image is a digital illustration of a girl hugging a white cat. The girl is wearing a pink sweater and has long brown hair. She is sitting on a green surface with several potted plants and flowers around her. The plants have green leaves and pink and white flowers. There are also two butterflies fluttering around the scene. The background is white. The overall style of the illustration is cartoon-like and playful.

### E.2 Visual Grounding

The image shows a cluttered room with a black and white checkered floor 1 . On the right side of the image, there is a small white cabinet 2 with a television 3 on top of it. Next to the cabinet, there are several items 4 scattered on the floor, including a red blanket 5 , a wooden stool 6 , and a pile of trash. On top of the cabinet is a picture frame 7 and a hat 8 . In the center of the room is a white refrigerator 9 with a few items on top. The walls 10 are painted white and there are a few clothes 11 hanging on a rack 12 on the left wall. The room appears to be in disarray, with some items strewn about and others scattered around. The image shows a kitchen countertop with various kitchen items on it. On the left side of the countertop, there is a microscope with a black body and a white lens 1 . Next to the microscope, there are two bottles of condiments 2 - one with a red label 3 4 and the other with green. On top of the microscope is a yellow banana 5 , a blue spatula 6 , a red plate 7 , and a yellow corn 8 9 on the cob. In the center of the image, there appears to be a frying pan 10 with a fried egg 11 on it, and on the right side is a white sink 12 with a white faucet 13 . The countertop 14 is made of wood and has a gray tile backsplash.

The image shows a modern kitchen with a large window on the left side. The window 1 has a view of trees and greenery outside. On the left side of the image, there is a blue sofa 2 with a wooden coffee table in front of it. Above the table, there are three copper pendant lights 3 hanging from the ceiling. There is a large island 4 with a white countertop. There are two bar stools 5 next to the table. In the center of the kitchen, there is a bottle green plants 6 on the table. The floor 7 is made of light-colored wood and the walls 8 are painted in a dark blue color. The image shows a man 1 standing in a kitchen with a small dog. The man 1 is wearing a plaid shirt 2 and jeans 3 and is holding a red cup 4 in his hand. The dog 5 is a light brown color and is standing on a tiled floor 6 . The kitchen 7 has wooden cabinets 8 and a countertop 9 with various kitchen utensils hanging on the wall. There is a window 10 with yellow curtains 11 in the background. On the right side of the image, there is a wooden cutting board 12 and a wooden stool 13 .

### E.3 Dense Region Caption

### E.4 Open Vocabulary Detection

Prompt : Locate giraffe in the image. Prompt : Locate Mercedes-Benz ⟨ \langle and ⟩ \rangle M2 ⟨ \langle and ⟩ \rangle Audi in the image. Prompt : Locate the objects with category name in the image. Prompt : Locate the objects with category name in the image.

### E.5 OCR

COFFEE+TEA 1 BLENDED 2 $1.69/$1.89/$2.09 3 $3.49/$3.99 4 Hot Coffee/Tea 5 Taro 6 Iced Coffee/ Tea 7 Mango 8 Hot Chocolate 9 Honeydew 10 $3,49/$ 3.99 11 Strawberry 12 Mocha 14 Thai Iced Tea / Coffee 13 Caramel 15 $1,99/$2,29/$2:59 16 SPECIALTY Brew !! 17 Jasmine GreenTea 18 LEONARDO 1 DiCAPRIO 2 ROBERT 3 DE NIRO 4 LILY 5 GLADSTONE 6 A MARTIN SCORSESE PICTURE 7 KILLERS 8 OF 9 FLOWER 10 MOON 11 SCREENLY ERIC ROTH AND MARTIIN SCORSESE DIRECTED BYMARTIN SORSESE 12 ONLY IN THEATRES OCTOBER 20 13

### E.6 Region to segmentation

Prompt : What is the polygon mask of region ⟨ \langle loc_541 ⟩ \rangle ⟨ \langle loc_266 ⟩ \rangle ⟨ \langle loc_692 ⟩ \rangle ⟨ \langle loc_627 ⟩ \rangle Prompt : What is the polygon mask of region ⟨ \langle loc_583 ⟩ \rangle ⟨ \langle loc_66 ⟩ \rangle ⟨ \langle loc_794 ⟩ \rangle ⟨ \langle loc_331 ⟩ \rangle Prompt : What is the polygon mask of region ⟨ \langle loc_386 ⟩ \rangle ⟨ \langle loc_53 ⟩ \rangle ⟨ \langle loc_759 ⟩ \rangle ⟨ \langle loc_998 ⟩ \rangle Prompt : What is the polygon mask of region ⟨ \langle loc_102 ⟩ \rangle ⟨ \langle loc_7 ⟩ \rangle ⟨ \langle loc_375 ⟩ \rangle ⟨ \langle loc_648 ⟩ \rangle

## Appendix F Comparision with LMMs on Detailed Caption Task

Florence-2 : The image is a photograph of a handwritten note from a person named Angelica. The note is written in black ink on a white background. The handwriting is neat and legible, and it appears to be a handwritten letter. The person’s name, Jonathan, is written at the top of the note in a cursive font. Below the name, there is a note that reads, "I’ve got to admit, London’s lights have nothing on our family get together. I miss that, I miss you guys too. Let’s not get too happy about it though I’m keeping my fingers crossed for a trip back stateside. Give a big hug to Amelia for me. Will you?" GPT-4V 1 1 1 https://chat.openai.com : The image shows a handwritten note addressed to Jonathan. The note conveys a personal message from the sender, Angelica, who admits to missing family gatherings and the feeling of home, contrasting it with the lights of London. Angelica expresses a longing to return stateside and asks Jonathan to give Amelia a big hug on her behalf. The tone of the note is casual and warm, with a sense of fondness and a hint of homesickness. The note is written informally, suggesting a close relationship between the sender and the recipients. LLaVA 2 2 2 https://llava.hliu.cc : The image is a handwritten letter on a piece of paper, written by a person named Jonathan. The letter is addressed to someone named Angela, and it appears to be a heartfelt message expressing love and affection. The letter is written in cursive, giving it a personal and intimate touch. MiniGPT-4 3 3 3 https://huggingface.co/spaces/Vision-CAIR/minigpt4 : The image is a handwritten letter from an unknown person to someone named Angel. The letter is written in black ink on a piece of white paper. The letter is addressed to Angel and begins with the words, “Dear Angel,”. The letter goes on to describe the writer’s feelings about a recent event or situation, but the details are unclear. The letter ends with the words, “Sincerely, [Writer’s Name].”

Florence-2 : The image is a close-up of a red apple with a green label on it. The label has the text "FUJI 94131 USA" and a barcode on it, indicating that the apple is from the United States. The apple appears to be fresh and has a shiny texture. The background is white, making the apple stand out. GPT-4V 4 4 4 https://chat.openai.com : The image displays a Fuji apple with a sticker indicating it is organic, along with the identifying number 94131, and specifying that it is from the USA. The apple has a reddish-pink skin with some yellow and green coloration, which is characteristic of Fuji apples. The apple is placed on a white background, which appears to be a textured fabric. LLaVA 5 5 5 https://llava.hliu.cc : The image features a large, fresh, and ripe red apple sitting on a white surface. The apple has a sticker on it, which is likely a barcode or a label providing information about the product. The apple is prominently displayed, taking up a significant portion of the image. MiniGPT-4 6 6 6 https://huggingface.co/spaces/Vision-CAIR/minigpt4 : The image shows a red apple with a label on it that reads “Organic” in white letters. The apple is sitting on a white surface.