Rethinking Hard-Parameter Sharing in Multi-Domain Learning
This work provides a stronger baseline for multi-domain learning model design, potentially improving efficiency and accuracy for researchers and practitioners in computer vision.
The paper challenges the conventional hard-parameter sharing approach in multi-domain learning by showing that using separate bottom-layer parameters significantly improves performance on image classification tasks, achieving competitive results with independent models while using only a small proportion of domain-specific parameters.
Hard parameter sharing in multi-domain learning (MDL) allows domains to share some of the model parameters to reduce storage cost while improving prediction accuracy. One common sharing practice is to share the bottom layers of a deep neural network among domains while using separate top layers for each domain. In this work, we revisit this common practice via an empirical study on image classification tasks from a diverse set of visual domains and make two surprising observations. (1) Using separate bottom-layer parameters could achieve significantly better performance than the common practice and this phenomenon holds with different experimental settings. (2) A multi-domain model with a small proportion of domain-specific parameters from bottom layers can achieve competitive performance with independent models trained on each domain separately. Our observations suggest that people adopt the new strategy of using separate bottom-layer parameters as a stronger baseline for model design in MDL.