Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering
This work addresses catastrophic forgetting in multimodal AI systems, which is an incremental improvement for applications like visual question answering.
The study tackled catastrophic forgetting in neural multimodal Visual Question Answering by designing linguistically-informed tasks based on psycholinguistics, finding that task difficulty and order significantly impact forgetting, with current continual learning methods providing only limited mitigation.
We study the issue of catastrophic forgetting in the context of neural multimodal approaches to Visual Question Answering (VQA). Motivated by evidence from psycholinguistics, we devise a set of linguistically-informed VQA tasks, which differ by the types of questions involved (Wh-questions and polar questions). We test what impact task difficulty has on continual learning, and whether the order in which a child acquires question types facilitates computational models. Our results show that dramatic forgetting is at play and that task difficulty and order matter. Two well-known current continual learning methods mitigate the problem only to a limiting degree.