Crowd-sourcing NLG Data: Pictures Elicit Better Data
This addresses the data bottleneck for NLG researchers by providing a more efficient method to collect high-quality training data, though it is incremental as it builds on existing crowdsourcing approaches.
The paper tackles the problem of costly training data for corpus-based Natural Language Generation by proposing a crowdsourcing framework with automatic quality control, showing that pictorial meaning representations elicit significantly more natural, informative, and better-phrased utterances than logic-based ones, with an average quality increase of about 0.5 points on a 6-point scale.
Recent advances in corpus-based Natural Language Generation (NLG) hold the promise of being easily portable across domains, but require costly training data, consisting of meaning representations (MRs) paired with Natural Language (NL) utterances. In this work, we propose a novel framework for crowdsourcing high quality NLG training data, using automatic quality control measures and evaluating different MRs with which to elicit data. We show that pictorial MRs result in better NL data being collected than logic-based MRs: utterances elicited by pictorial MRs are judged as significantly more natural, more informative, and better phrased, with a significant increase in average quality ratings (around 0.5 points on a 6-point scale), compared to using the logical MRs. As the MR becomes more complex, the benefits of pictorial stimuli increase. The collected data will be released as part of this submission.