Bipartite-play Dialogue Collection for Practical Automatic Evaluation of Dialogue Systems
This is an incremental improvement for developers of dialogue systems, enabling more practical and robust automated evaluation.
The paper tackled the problem of automating dialogue system evaluation by introducing the bipartite-play method to address limitations like inability to compare with non-public systems and vulnerability to cheating, showing it correlates as strongly with human subjectivity as existing methods.
Automation of dialogue system evaluation is a driving force for the efficient development of dialogue systems. This paper introduces the bipartite-play method, a dialogue collection method for automating dialogue system evaluation. It addresses the limitations of existing dialogue collection methods: (i) inability to compare with systems that are not publicly available, and (ii) vulnerability to cheating by intentionally selecting systems to be compared. Experimental results show that the automatic evaluation using the bipartite-play method mitigates these two drawbacks and correlates as strongly with human subjectivity as existing methods.