"Is This It?": Towards Ecologically Valid Benchmarks for Situated Collaboration
This work addresses the need for ecologically valid benchmarks for AI researchers and developers, but it is incremental as it builds on existing embodied question answering frameworks.
The paper tackles the problem of creating more realistic benchmarks for evaluating large multimodal models in situated collaboration by proposing an interactive system-driven approach where users generate questions in context, revealing differences in form and content from existing benchmarks and highlighting new real-world challenges.
We report initial work towards constructing ecologically valid benchmarks to assess the capabilities of large multimodal models for engaging in situated collaboration. In contrast to existing benchmarks, in which question-answer pairs are generated post hoc over preexisting or synthetic datasets via templates, human annotators, or large language models (LLMs), we propose and investigate an interactive system-driven approach, where the questions are generated by users in context, during their interactions with an end-to-end situated AI system. We illustrate how the questions that arise are different in form and content from questions typically found in existing embodied question answering (EQA) benchmarks and discuss new real-world challenge problems brought to the fore.