Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling
This work provides a method for improving NLG and NLU models for researchers and practitioners facing low-resource data scenarios, offering strong specific gains on established benchmarks.
This paper addresses the data scarcity in neural natural language generation (NLG) and understanding (NLU) by synthesizing weak labels at scale using a fine-tuned GPT-2. The proposed semi-supervised framework jointly trains NLG and NLU models, adapting parameter updates based on estimated label quality, and achieves state-of-the-art performance on E2E and Weather benchmarks when using 100% of the training data.
Neural natural language generation (NLG) and understanding (NLU) models are data-hungry and require massive amounts of annotated data to be competitive. Recent frameworks address this bottleneck with generative models that synthesize weak labels at scale, where a small amount of training labels are expert-curated and the rest of the data is automatically annotated. We follow that approach, by automatically constructing a large-scale weakly-labeled data with a fine-tuned GPT-2, and employ a semi-supervised framework to jointly train the NLG and NLU models. The proposed framework adapts the parameter updates to the models according to the estimated label-quality. On both the E2E and Weather benchmarks, we show that this weakly supervised training paradigm is an effective approach under low resource scenarios and outperforming benchmark systems on both datasets when 100% of training data is used.