Evaluation of Unsupervised Entity and Event Salience Estimation
This work addresses methodological issues in salience estimation evaluation for NLP researchers, though it appears incremental as it builds on existing frameworks.
The authors tackled the problem of unreliable evaluation protocols for unsupervised entity and event salience estimation, proposing a new protocol with syntactic dependency parsing and dependency-based heterogeneous graphs, which resulted in baseline methods and a novel GNN method consistently outperforming previous state-of-the-art models across all metrics.
Salience Estimation aims to predict term importance in documents. Due to few existing human-annotated datasets and the subjective notion of salience, previous studies typically generate pseudo-ground truth for evaluation. However, our investigation reveals that the evaluation protocol proposed by prior work is difficult to replicate, thus leading to few follow-up studies existing. Moreover, the evaluation process is problematic: the entity linking tool used for entity matching is very noisy, while the ignorance of event argument for event evaluation leads to boosted performance. In this work, we propose a light yet practical entity and event salience estimation evaluation protocol, which incorporates the more reliable syntactic dependency parser. Furthermore, we conduct a comprehensive analysis among popular entity and event definition standards, and present our own definition for the Salience Estimation task to reduce noise during the pseudo-ground truth generation process. Furthermore, we construct dependency-based heterogeneous graphs to capture the interactions of entities and events. The empirical results show that both baseline methods and the novel GNN method utilizing the heterogeneous graph consistently outperform the previous SOTA model in all proposed metrics.