Learning Locality and Isotropy in Dialogue Modeling
This addresses the issue of poor dialogue structure retention for researchers and practitioners in conversational AI, offering an incremental improvement over existing methods.
The paper tackled the problem of anisotropic and non-conversational context representations in dialogue modeling, proposing SimDRC to calibrate representations for locality and isotropy, which significantly outperformed state-of-the-art models on three dialogue tasks in automatic and human evaluations.
Existing dialogue modeling methods have achieved promising performance on various dialogue tasks with the aid of Transformer and the large-scale pre-trained language models. However, some recent studies revealed that the context representations produced by these methods suffer the problem of anisotropy. In this paper, we find that the generated representations are also not conversational, losing the conversation structure information during the context modeling stage. To this end, we identify two properties in dialogue modeling, i.e., locality and isotropy, and present a simple method for dialogue representation calibration, namely SimDRC, to build isotropic and conversational feature spaces. Experimental results show that our approach significantly outperforms the current state-of-the-art models on three dialogue tasks across the automatic and human evaluation metrics. More in-depth analyses further confirm the effectiveness of our proposed approach.