Modelling Latent Skills for Multitask Language Generation
This work addresses efficient adaptation and positive transfer across diverse language generation tasks, though it is incremental as it builds on existing latent variable and sequence-to-sequence models.
The authors tackled the problem of multitask language generation by hypothesizing that a shared set of latent skills underlies various tasks, and they developed a latent variable model to explicitly model these skills. Their model outperformed sequence-to-sequence baselines in multitask settings and showed robust adaptation in few-shot learning on unseen tasks.
We present a generative model for multitask conditional language generation. Our guiding hypothesis is that a shared set of latent skills underlies many disparate language generation tasks, and that explicitly modelling these skills in a task embedding space can help with both positive transfer across tasks and with efficient adaptation to new tasks. We instantiate this task embedding space as a latent variable in a latent variable sequence-to-sequence model. We evaluate this hypothesis by curating a series of monolingual text-to-text language generation datasets - covering a broad range of tasks and domains - and comparing the performance of models both in the multitask and few-shot regimes. We show that our latent task variable model outperforms other sequence-to-sequence baselines on average across tasks in the multitask setting. In the few-shot learning setting on an unseen test dataset (i.e., a new task), we demonstrate that model adaptation based on inference in the latent task space is more robust than standard fine-tuning based parameter adaptation and performs comparably in terms of overall performance. Finally, we examine the latent task representations learnt by our model and show that they cluster tasks in a natural way.