Is Fast Adaptation All You Need?
This work addresses a specific bottleneck in meta-learning for incremental learning scenarios, offering an incremental improvement over existing methods.
The paper tackles the problem of catastrophic interference in meta-learning by proposing a training signal based on robustness to interference, showing that representations learned this way improve incremental learning compared to those focused solely on fast adaptation.
Gradient-based meta-learning has proven to be highly effective at learning model initializations, representations, and update rules that allow fast adaptation from a few samples. The core idea behind these approaches is to use fast adaptation and generalization -- two second-order metrics -- as training signals on a meta-training dataset. However, little attention has been given to other possible second-order metrics. In this paper, we investigate a different training signal -- robustness to catastrophic interference -- and demonstrate that representations learned by directing minimizing interference are more conducive to incremental learning than those learned by just maximizing fast adaptation.