Panoptic Captioning: Seeking An Equivalency Bridge for Image and Text
This addresses the challenge of bridging image and text equivalence for applications in computer vision and multimodal AI, though it appears incremental as it builds on existing MLLM frameworks.
The paper tackles the problem of generating comprehensive textual descriptions that capture all entities, locations, attributes, relationships, and global state in images, revealing that state-of-the-art MLLMs have limited performance. The proposed PancapChain-13B model beats open-source MLLMs like InternVL-2.5-78B and surpasses proprietary models like GPT-4o and Gemini-2.0-Pro.
This work introduces panoptic captioning, a novel task striving to seek the minimum text equivalence of images. We take the first step towards panoptic captioning by formulating it as a task of generating a comprehensive textual description for an image, which encapsulates all entities, their respective locations and attributes, relationships among entities, as well as global image state. Through an extensive evaluation, our work reveals that state-of-the-art Multi-modal Large Language Models (MLLMs) have limited performance in solving panoptic captioning. To address this, we propose an effective data engine named PancapEngine to produce high-quality data and a novel method named PancapChain to improve panoptic captioning. Specifically, our PancapEngine first detects diverse categories of entities in images by an elaborate detection suite, and then generates required panoptic captions using entity-aware prompts. Additionally, our PancapChain explicitly decouples the challenging panoptic captioning task into multiple stages and generates panoptic captions step by step. More importantly, we contribute a comprehensive metric named PancapScore and a human-curated test set for reliable model evaluation. Experiments show that our PancapChain-13B model can beat state-of-the-art open-source MLLMs like InternVL-2.5-78B and even surpass proprietary models like GPT-4o and Gemini-2.0-Pro, demonstrating the effectiveness of our data engine and method. Project page: https://visual-ai.github.io/pancap/