UKP-SQuARE v3: A Platform for Multi-Agent QA Research
This work provides a tool for researchers in natural language processing to more easily experiment with multi-agent QA systems, though it is incremental as it builds on an existing platform.
The paper tackles the challenge of facilitating research on multi-agent question answering systems by extending the UKP-SQuARE platform to support three types of multi-agent systems, and it evaluates their inference speed and performance trade-offs compared to multi-dataset models.
The continuous development of Question Answering (QA) datasets has drawn the research community's attention toward multi-domain models. A popular approach is to use multi-dataset models, which are models trained on multiple datasets to learn their regularities and prevent overfitting to a single dataset. However, with the proliferation of QA models in online repositories such as GitHub or Hugging Face, an alternative is becoming viable. Recent works have demonstrated that combining expert agents can yield large performance gains over multi-dataset models. To ease research in multi-agent models, we extend UKP-SQuARE, an online platform for QA research, to support three families of multi-agent systems: i) agent selection, ii) early-fusion of agents, and iii) late-fusion of agents. We conduct experiments to evaluate their inference speed and discuss the performance vs. speed trade-off compared to multi-dataset models. UKP-SQuARE is open-source and publicly available at http://square.ukp-lab.de.