Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data
This addresses the need for more interpretable and accurate multimodal AI in domain-specific applications, representing an incremental improvement.
The paper tackles the problem of large multimodal models struggling with fine-grained visual reasoning and explainability by proposing a visual rejection sampling framework that uses self-synthesized data for iterative fine-tuning, resulting in improved accuracy and explainability in specialized visual classification tasks.
Large Multimodal Models (LMMs), or Vision-Language Models (VLMs), have shown impressive capabilities in a wide range of visual tasks. However, they often struggle with fine-grained visual reasoning, failing to identify domain-specific objectives and provide justifiable explanations for their predictions. To address the above challenge, we propose a novel visual rejection sampling framework to improve the cognition and explainability of LMMs using self-synthesized data. Specifically, visual fine-tuning requires images, queries, and target answers. Our approach begins by synthesizing interpretable answers that include human-verifiable visual features. These features are based on expert-defined concepts, and carefully selected based on their alignment with the image content. After each round of fine-tuning, we apply a reward model-free filtering mechanism to select the highest-quality interpretable answers for the next round of tuning. This iterative process of synthetic data generation and fine-tuning progressively improves the model's ability to generate accurate and reasonable explanations. Experimental results demonstrate the effectiveness of our method in improving both the accuracy and explainability of specialized visual classification tasks.