I2I: Initializing Adapters with Improvised Knowledge
This addresses the issue of missed cross-task knowledge transfer in continual learning for multimodal tasks like visual question answering, though it is incremental as it builds on existing Adapter methods.
The paper tackles the problem of catastrophic forgetting in continual learning by proposing I2I, an algorithm that initializes new Adapters using knowledge distilled from previous tasks, resulting in consistently better task accuracy than independently-trained Adapters and outperforming AdapterFusion without added parameters.
Adapters present a promising solution to the catastrophic forgetting problem in continual learning. However, training independent Adapter modules for every new task misses an opportunity for cross-task knowledge transfer. We propose Improvise to Initialize (I2I), a continual learning algorithm that initializes Adapters for incoming tasks by distilling knowledge from previously-learned tasks' Adapters. We evaluate I2I on CLiMB, a multimodal continual learning benchmark, by conducting experiments on sequences of visual question answering tasks. Adapters trained with I2I consistently achieve better task accuracy than independently-trained Adapters, demonstrating that our algorithm facilitates knowledge transfer between task Adapters. I2I also results in better cross-task knowledge transfer than the state-of-the-art AdapterFusion without incurring the associated parametric cost.