Differentiable Weight Masks for Domain Transfer
This work addresses the issue of catastrophic forgetting in computer vision models for researchers, but it is incremental as it builds on existing weight masking techniques.
The paper tackles the problem of domain transfer in deep learning by evaluating three weight masking methods to mitigate forgetting on a source task while fine-tuning on a target task, finding trade-offs in knowledge retention and target performance.
One of the major drawbacks of deep learning models for computer vision has been their inability to retain multiple sources of information in a modular fashion. For instance, given a network that has been trained on a source task, we would like to re-train this network on a similar, yet different, target task while maintaining its performance on the source task. Simultaneously, researchers have extensively studied modularization of network weights to localize and identify the set of weights culpable for eliciting the observed performance on a given task. One set of works studies the modularization induced in the weights of a neural network by learning and analysing weight masks. In this work, we combine these fields to study three such weight masking methods and analyse their ability to mitigate "forgetting'' on the source task while also allowing for efficient finetuning on the target task. We find that different masking techniques have trade-offs in retaining knowledge in the source task without adversely affecting target task performance.