Efficient Multi-Domain Network Learning by Covariance Normalization
This addresses the problem of parameter efficiency in multi-domain learning for deep learning practitioners, though it appears incremental as it builds on existing normalization techniques.
The paper tackles multi-domain learning by introducing Covariance Normalization (CovNorm), a method that reduces parameters in adaptive layers per target domain, achieving performance comparable to full fine-tuning with only 0.13% of parameters per domain.
The problem of multi-domain learning of deep networks is considered. An adaptive layer is induced per target domain and a novel procedure, denoted covariance normalization (CovNorm), proposed to reduce its parameters. CovNorm is a data driven method of fairly simple implementation, requiring two principal component analyzes (PCA) and fine-tuning of a mini-adaptation layer. Nevertheless, it is shown, both theoretically and experimentally, to have several advantages over previous approaches, such as batch normalization or geometric matrix approximations. Furthermore, CovNorm can be deployed both when target datasets are available sequentially or simultaneously. Experiments show that, in both cases, it has performance comparable to a fully fine-tuned network, using as few as 0.13% of the corresponding parameters per target domain.