Domain Specific Sub-network for Multi-Domain Neural Machine Translation
This addresses the challenge of efficiently adapting machine translation models to multiple domains with reduced computational overhead, though it is incremental over existing pruning and fine-tuning techniques.
This paper tackles the problem of multi-domain neural machine translation by proposing Domain-Specific Sub-network (DoSS), which uses pruning masks to create domain-specific sub-networks and fine-tunes them on domain data, achieving BLEU score improvements of 1.47 and 1.52 points over strong baselines while drastically reducing parameters.
This paper presents Domain-Specific Sub-network (DoSS). It uses a set of masks obtained through pruning to define a sub-network for each domain and finetunes the sub-network parameters on domain data. This performs very closely and drastically reduces the number of parameters compared to finetuning the whole network on each domain. Also a method to make masks unique per domain is proposed and shown to greatly improve the generalization to unseen domains. In our experiments on German to English machine translation the proposed method outperforms the strong baseline of continue training on multi-domain (medical, tech and religion) data by 1.47 BLEU points. Also continue training DoSS on new domain (legal) outperforms the multi-domain (medical, tech, religion, legal) baseline by 1.52 BLEU points.