Differential Multimodal Transformers
This work addresses noise and hallucination issues in multimodal models for users of efficient small language models, but it is incremental as it extends an existing method to a new domain.
The authors tackled the problem of noisy information retrieval and hallucinations in multimodal small language models by adapting the Differential Attention mechanism to the text-vision model PaliGemma, resulting in enhanced performance in these areas.
Small language models have gained significant popularity due to their efficiency and growing capabilities. However, incorporating additional modalities, such as vision, can exacerbate the challenge of limited context windows by introducing noise. Recent studies have highlighted that Transformer attention mechanisms often disproportionately focus on irrelevant contexts. In this work, we extend the Differential Attention mechanism, originally designed for text-only models, to the text-vision model PaliGemma. Our aim is to evaluate its ability to mitigate noisy information retrieval and reduce hallucinations. To this end, we fine-tuned the PaliGemma 3B model using LoRA, incorporating Differential Attention, and experimented with various parameter settings and configurations. We demonstrate that Differential Attention can be adapted and integrated into the fine-tuning of existing models to enhance noisy information retrieval and question-answering capabilities.