Findings of the First Workshop on Simulating Conversational Intelligence in Chat
It addresses the challenge of creating realistic conversational AI for open-domain dialogue, but is incremental as it builds on prior workshops.
The paper provides an overview and analysis of the SCI-CHAT shared task, which focused on simulating intelligent conversation through multi-turn dialogues with reasoning and argumentation, resulting in publicly released evaluation data and code.
The aim of the workshop was to bring together experts working on open-domain dialogue research. In this speedily advancing research area many challenges still exist, such as learning information from conversations, and engaging in a realistic and convincing simulation of human intelligence and reasoning. SCI-CHAT follows previous workshops on open domain dialogue but in contrast the focus of the shared task is simulation of intelligent conversation as judged in a live human evaluation. Models aim to include the ability to follow a challenging topic over a multi-turn conversation, while positing, refuting and reasoning over arguments. The workshop included both a research track and shared task. The main goal of this paper is to provide an overview of the shared task, and an in depth analysis of the shared task results following presentation at the workshop. The current paper is an extension of that made available prior to presentation of results at the workshop at EACL Malta (Graham et al., 2024). The data collected in the evaluation was made publicly available to aide future research. The code was also made available for the same purpose.