The Second Conversational Intelligence Challenge (ConvAI2)
This addresses the problem of improving open-domain chatbots for researchers and developers, but it is incremental as it builds on existing competition frameworks and methods.
The paper describes the ConvAI2 NeurIPS competition, which aimed to advance open-domain chatbots, finding that pretrained Transformer variants performed best but highlighting the need for better multi-turn conversation metrics beyond single-word measures.
We describe the setting and results of the ConvAI2 NeurIPS competition that aims to further the state-of-the-art in open-domain chatbots. Some key takeaways from the competition are: (i) pretrained Transformer variants are currently the best performing models on this task, (ii) but to improve performance on multi-turn conversations with humans, future systems must go beyond single word metrics like perplexity to measure the performance across sequences of utterances (conversations) -- in terms of repetition, consistency and balance of dialogue acts (e.g. how many questions asked vs. answered).