Chain-of-Description: What I can understand, I can put into words
This addresses performance bottlenecks in multi-modal AI systems for tasks like audio and vision understanding, representing an incremental improvement over standard prompting methods.
The paper tackles the problem of improving multi-modal large language models by proposing Chain-of-Description Prompting, which involves having the model first describe multi-modal inputs before answering questions, resulting in a nearly 4% improvement on an audio benchmark and a 5.3% improvement on a vision benchmark.
In this paper, we propose a novel strategy defined as Chain-of-Description (CoD) Prompting, tailored for Multi-Modal Large Language Models. This approach involves having the model first provide a detailed description of the multi-modal input before generating an answer to the question. When applied to models such as Qwen2-Audio, Qwen2-VL, and Qwen2.5-VL, CoD Prompting significantly enhances performance compared to standard prompting methods. This is demonstrated by nearly a 4\% improvement in the speech category of the audio benchmark AIR-Bench-Chat and a 5.3\% improvement in the hard-level portion of the vision benchmark MMMU\_Pro. Our ablation study further validates the effectiveness of CoD Prompting.