Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Models
This work addresses a specific bottleneck in multi-modal reasoning for AI researchers and practitioners, offering an incremental enhancement to existing Chain-of-Thought methods.
The paper tackles the challenge of selecting optimal demonstration examples for multi-modal reasoning in large language models by introducing a retrieval-augmented approach that dynamically selects examples based on cross-modal and intra-modal similarities, resulting in performance improvements of 6% on ScienceQA and 12.9% on MathVista for GPT-4.
The advancement of Large Language Models (LLMs) has brought substantial attention to the Chain of Thought (CoT) approach, primarily due to its ability to enhance the capability of LLMs on complex reasoning tasks. Moreover, the significance of CoT approaches extends to the application of LLMs for multi-modal tasks. However, the selection of optimal CoT demonstration examples in multi-modal reasoning remains less explored for LLMs due to the inherent complexity of multi-modal examples. In this paper, we introduce a novel approach that addresses this challenge by using retrieval mechanisms to dynamically and automatically select demonstration examples based on cross-modal and intra-modal similarities. Furthermore, we employ a Stratified Sampling method of categorising demonstration examples into groups based on their types and then retrieving examples from different groups respectively to promote the diversity of demonstration examples. Through a series of experiments on two popular benchmark datasets: ScienceQA and MathVista, we demonstrate that our approach significantly improves the performance of GPT-4 by 6% on ScienceQA and 12.9% on MathVista, and enhances the performance of GPT-4V on two datasets by 2.7%, substantially improving the performance of the most advanced LLMs and LMMs for complex multi-modal reasoning tasks.