Dealing with Missing Modalities in the Visual Question Answer-Difference Prediction Task through Knowledge Distillation
This addresses a modality-missing issue in visual question answering for researchers, but it is incremental as it builds on existing knowledge distillation techniques.
The paper tackles the problem of missing ground truth answers in the Visual Question Answer-Difference prediction task by using a privileged knowledge distillation scheme, where a teacher model with full modalities trains a student model that only uses image and question inputs, achieving performance improvements on VizWiz and VQA-V2 datasets.
In this work, we address the issues of missing modalities that have arisen from the Visual Question Answer-Difference prediction task and find a novel method to solve the task at hand. We address the missing modality-the ground truth answers-that are not present at test time and use a privileged knowledge distillation scheme to deal with the issue of the missing modality. In order to efficiently do so, we first introduce a model, the "Big" Teacher, that takes the image/question/answer triplet as its input and outperforms the baseline, then use a combination of models to distill knowledge to a target network (student) that only takes the image/question pair as its inputs. We experiment our models on the VizWiz and VQA-V2 Answer Difference datasets and show through extensive experimentation and ablation the performances of our method and a diverse possibility for future research.