Multimodal Commonsense Knowledge Distillation for Visual Question Answering
This addresses a specific bottleneck in VQA for AI systems, but it is incremental as it builds on existing teacher-student and graph-based methods.
The paper tackles the problem of Visual Question Answering (VQA) questions requiring external commonsense knowledge by proposing a graph-based multimodal commonsense knowledge distillation framework, achieving competitive performances on the ScienceQA dataset.
Existing Multimodal Large Language Models (MLLMs) and Visual Language Pretrained Models (VLPMs) have shown remarkable performances in the general Visual Question Answering (VQA). However, these models struggle with VQA questions that require external commonsense knowledge due to the challenges in generating high-quality prompts and the high computational costs of fine-tuning. In this work, we propose a novel graph-based multimodal commonsense knowledge distillation framework that constructs a unified relational graph over commonsense knowledge, visual objects and questions through a Graph Convolutional Network (GCN) following a teacher-student environment. This proposed framework is flexible with any type of teacher and student models without further fine-tuning, and has achieved competitive performances on the ScienceQA dataset.