Underreporting of errors in NLG output, and what to do about it
This addresses a critical transparency issue for the NLG research community, though it is incremental as it focuses on reporting practices rather than new methods.
The paper identifies severe under-reporting of error types in Natural Language Generation systems, which hinders improvement by obscuring specific weaknesses, and provides recommendations for better error identification, analysis, and reporting.
We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by `state-of-the-art' research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.