Exploring the Influence of Relevant Knowledge for Natural Language Generation Interpretability
This work addresses the need for interpretable, knowledge-enhanced NLG systems, though it is incremental as it builds on existing datasets and models.
The paper tackled the problem of how external knowledge affects natural language generation by creating a benchmark (KITGI) and testing T5-Large with full vs. filtered knowledge, finding that full knowledge achieved 91% correctness while filtering reduced it to 6%.
This paper explores the influence of external knowledge integration in Natural Language Generation (NLG), focusing on a commonsense generation task. We extend the CommonGen dataset by creating KITGI, a benchmark that pairs input concept sets with retrieved semantic relations from ConceptNet and includes manually annotated outputs. Using the T5-Large model, we compare sentence generation under two conditions: with full external knowledge and with filtered knowledge where highly relevant relations were deliberately removed. Our interpretability benchmark follows a three-stage method: (1) identifying and removing key knowledge, (2) regenerating sentences, and (3) manually assessing outputs for commonsense plausibility and concept coverage. Results show that sentences generated with full knowledge achieved 91\% correctness across both criteria, while filtering reduced performance drastically to 6\%. These findings demonstrate that relevant external knowledge is critical for maintaining both coherence and concept coverage in NLG. This work highlights the importance of designing interpretable, knowledge-enhanced NLG systems and calls for evaluation frameworks that capture the underlying reasoning beyond surface-level metrics.