Conversations Gone Awry, But Then? Evaluating Conversational Forecasting Models
This work addresses the need for reliable benchmarks in conversational forecasting to assist human-human interactions, but it is incremental as it focuses on evaluation rather than new model development.
The authors tackled the problem of forecasting whether conversations will derail by introducing a uniform evaluation framework for the Conversations Gone Awry task, enabling direct comparisons and providing an up-to-date overview of current model progress.
We often rely on our intuition to anticipate the direction of a conversation. Endowing automated systems with similar foresight can enable them to assist human-human interactions. Recent work on developing models with this predictive capacity has focused on the Conversations Gone Awry (CGA) task: forecasting whether an ongoing conversation will derail. In this work, we revisit this task and introduce the first uniform evaluation framework, creating a benchmark that enables direct and reliable comparisons between different architectures. This allows us to present an up-to-date overview of the current progress in CGA models, in light of recent advancements in language modeling. Our framework also introduces a novel metric that captures a model's ability to revise its forecast as the conversation progresses.