Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models
This addresses the need for better multi-modal reasoning in language models to handle complex real-world problems, representing an incremental improvement over existing methods.
The paper tackles the problem of multi-modal chain-of-thought reasoning in language models by introducing a latent space learning approach using diffusion processes to generate image features aligned with language thoughts, achieving state-of-the-art performance on the ScienceQA benchmark.
Chain-of-thought (CoT) reasoning has exhibited impressive performance in language models for solving complex tasks and answering questions. However, many real-world questions require multi-modal information, such as text and images. Previous research on multi-modal CoT has primarily focused on extracting fixed image features from off-the-shelf vision models and then fusing them with text using attention mechanisms. This approach has limitations because these vision models were not designed for complex reasoning tasks and do not align well with language thoughts. To overcome this limitation, we introduce a novel approach for multi-modal CoT reasoning that utilizes latent space learning via diffusion processes to generate effective image features that align with language thoughts. Our method fuses image features and text representations at a deep level and improves the complex reasoning ability of multi-modal CoT. We demonstrate the efficacy of our proposed method on multi-modal ScienceQA and machine translation benchmarks, achieving state-of-the-art performance on ScienceQA. Overall, our approach offers a more robust and effective solution for multi-modal reasoning in language models, enhancing their ability to tackle complex real-world problems.