Curbing Task Interference using Representation Similarity-Guided Multi-Task Feature Sharing
This addresses a challenge in multi-task learning for dense prediction tasks, offering incremental improvements in performance and robustness.
The paper tackles the problem of task interference in multi-task learning when sharing both encoder and decoder, which can hinder generalization and robustness. The proposed Progressive Decoder Fusion method improves in-distribution and out-of-distribution generalization, adversarial robustness, and inter-task prediction consistency.
Multi-task learning of dense prediction tasks, by sharing both the encoder and decoder, as opposed to sharing only the encoder, provides an attractive front to increase both accuracy and computational efficiency. When the tasks are similar, sharing the decoder serves as an additional inductive bias providing more room for tasks to share complementary information among themselves. However, increased sharing exposes more parameters to task interference which likely hinders both generalization and robustness. Effective ways to curb this interference while exploiting the inductive bias of sharing the decoder remains an open challenge. To address this challenge, we propose Progressive Decoder Fusion (PDF) to progressively combine task decoders based on inter-task representation similarity. We show that this procedure leads to a multi-task network with better generalization to in-distribution and out-of-distribution data and improved robustness to adversarial attacks. Additionally, we observe that the predictions of different tasks of this multi-task network are more consistent with each other.