Towards Collaborative Question Answering: A Preliminary Study
This work addresses the problem of disjointed knowledge in complex QA for AI researchers, but it is a preliminary study with incremental contributions.
The authors introduced CollabQA, a new question-answering task where multiple expert agents collaborate to answer complex questions that single agents cannot handle alone, and they created a synthetic dataset and evaluation metrics to demonstrate the challenge of the task without prior collaboration structure.
Knowledge and expertise in the real-world can be disjointedly owned. To solve a complex question, collaboration among experts is often called for. In this paper, we propose CollabQA, a novel QA task in which several expert agents coordinated by a moderator work together to answer questions that cannot be answered with any single agent alone. We make a synthetic dataset of a large knowledge graph that can be distributed to experts. We define the process to form a complex question from ground truth reasoning path, neural network agent models that can learn to solve the task, and evaluation metrics to check the performance. We show that the problem can be challenging without introducing prior of the collaboration structure, unless experts are perfect and uniform. Based on this experience, we elaborate extensions needed to approach collaboration tasks in real-world settings.