Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks

arxiv:2311.06242

Original source · Paper JSON

LLMs: architectures, context, training · 2023-11-10

Cites (1) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks
• Abstract
• 1 Introduction
• 2 Rethinking Vision Model Pre-training
• 2.1 Comprehensive Multitask Learning
• 3 Model
• Task formulation.
• Vision encoder.
• Multi-modality encoder decoder.
• Optimization objective.
• 4 Data Engine
• 4.1 Image Collection
• 4.2 Data Annotation
• Initial annotation with specialist models.
• Data filtering and enhancement.
• Iterative data refinement.
• 4.3 Annotation-specific Variations
• Text.
• Region-text pairs.
• Text-phrase-region triplets.
• 5 Dataset
• 5.1 Overview
• 5.2 Data Analysis
• Annotation statistics.
• Semantic coverage.
• Spatial coverage.
• 6 Experiments
• 6.1 Setup
• 6.2 Zero-shot Evaluation Across Tasks
• 6.3 Generalist Model with Public Supervised Data
• Simple design for strong performance.
• Competitive performance with fewer parameters.
• Adaptable generalization across task levels.
• 6.4 Downstream Tasks Fine-tuning
• Object detection and segmentation.
• Semantic segmentation.
• 6.5 Ablation Studies
• Multitask transfer.
• Model scaling.
• Data scaling.
• Training settings.
• 7 Related Works
• 7.1 Vision-Language Foundation Models
• 7.2 Vision Datasets
• Comprehensive annotations.
• Scalable annotations.
• 8 Conclusion
• Acknowledgment.
• Appendix A Supported Tasks and Annotations in Florence-2
• Appendix B Supervised Data Collection for Generalist Model Fine-tuning
• Appendix C Model Configuration
• Appendix D More Examples of Annotations in FLD-5B
• Appendix E Qualitative Evaluation and Visualization Results
• E.1 Detailed Image Caption
• E.2 Visual Grounding
• E.3 Dense Region Caption
• E.4 Open Vocabulary Detection
• E.5 OCR
• E.6 Region to segmentation
• Appendix F Comparision with LMMs on Detailed Caption Task
• Appendix G Comarison with Kosmos-2 on Detailed Caption and Grounding Tasks
• Instructions for reporting errors