Uncovering the Potential of ChatGPT for Discourse Analysis in Dialogue: An Empirical Study
This work addresses the problem of assessing ChatGPT's discourse understanding for researchers in NLP, providing empirical insights into its limitations and potential, though it is incremental as it applies existing methods to new tasks.
The paper systematically evaluates ChatGPT's performance on discourse analysis tasks, finding it proficient in topic segmentation for general-domain conversations but struggling with specific-domain dialogues and complex rhetorical structures, with results showing it can outperform human annotations in some cases but only linearly parses hierarchical structures.
Large language models, like ChatGPT, have shown remarkable capability in many downstream tasks, yet their ability to understand discourse structures of dialogues remains less explored, where it requires higher level capabilities of understanding and reasoning. In this paper, we aim to systematically inspect ChatGPT's performance in two discourse analysis tasks: topic segmentation and discourse parsing, focusing on its deep semantic understanding of linear and hierarchical discourse structures underlying dialogue. To instruct ChatGPT to complete these tasks, we initially craft a prompt template consisting of the task description, output format, and structured input. Then, we conduct experiments on four popular topic segmentation datasets and two discourse parsing datasets. The experimental results showcase that ChatGPT demonstrates proficiency in identifying topic structures in general-domain conversations yet struggles considerably in specific-domain conversations. We also found that ChatGPT hardly understands rhetorical structures that are more complex than topic structures. Our deeper investigation indicates that ChatGPT can give more reasonable topic structures than human annotations but only linearly parses the hierarchical rhetorical structures. In addition, we delve into the impact of in-context learning (e.g., chain-of-thought) on ChatGPT and conduct the ablation study on various prompt components, which can provide a research foundation for future work. The code is available at \url{https://github.com/yxfanSuda/GPTforDDA}.