Do Models Learn the Directionality of Relations? A New Evaluation: Relation Direction Recognition
This addresses interpretability concerns in relation classification for NLP researchers, but it is incremental as it focuses on evaluation rather than a new method.
The paper tackles the problem of whether deep neural models like BERT learn the directionality of relations in relation classification, proposing a new evaluation task called Relation Direction Recognition (RDR) and showing clear performance gaps among state-of-the-art models on a real-world dataset, even when they have similar traditional metric scores.
Deep neural networks such as BERT have made great progress in relation classification. Although they can achieve good performance, it is still a question of concern whether these models recognize the directionality of relations, especially when they may lack interpretability. To explore the question, a novel evaluation task, called Relation Direction Recognition (RDR), is proposed to explore whether models learn the directionality of relations. Three metrics for RDR are introduced to measure the degree to which models recognize the directionality of relations. Several state-of-the-art models are evaluated on RDR. Experimental results on a real-world dataset indicate that there are clear gaps among them in recognizing the directionality of relations, even though these models obtain similar performance in the traditional metric (e.g. Macro-F1). Finally, some suggestions are discussed to enhance models to recognize the directionality of relations from the perspective of model design or training.