Topic Shift Detection in Chinese Dialogues: Corpus and Benchmark
This work addresses a gap in Chinese natural conversation data and improves topic shift detection for dialogue systems, though it is incremental as it builds on existing methods for a known bottleneck.
The paper tackles the challenge of detecting topic shifts in dialogues without access to the response, by introducing a Chinese corpus of 1308 dialogues and proposing a teacher-student framework with hierarchical contrastive learning, achieving effective results on both Chinese and English benchmarks.
Dialogue topic shift detection is to detect whether an ongoing topic has shifted or should shift in a dialogue, which can be divided into two categories, i.e., response-known task and response-unknown task. Currently, only a few investigated the latter, because it is still a challenge to predict the topic shift without the response information. In this paper, we first annotate a Chinese Natural Topic Dialogue (CNTD) corpus consisting of 1308 dialogues to fill the gap in the Chinese natural conversation topic corpus. And then we focus on the response-unknown task and propose a teacher-student framework based on hierarchical contrastive learning to predict the topic shift without the response. Specifically, the response at high-level teacher-student is introduced to build the contrastive learning between the response and the context, while the label contrastive learning is constructed at low-level student. The experimental results on our Chinese CNTD and English TIAGE show the effectiveness of our proposed model.