Improving the State of the Art for Training Human-AI Teams: Technical Report #3 -- Analysis of Testbed Alternatives
This work addresses the need for effective testbeds in Human-AI teaming research, particularly for military applications, but it is incremental as it focuses on evaluating existing options rather than creating new ones.
The researchers tackled the problem of developing a testbed for Human-AI teaming research by systematically evaluating existing alternatives based on criteria from expert surveys, resulting in the identification of five candidate testbeds for further assessment.
Sonalysts is working on an initiative to expand our current expertise in teaming to Human-Artificial Intelligence (AI) teams by developing original research in this area. To provide a foundation for that research, Sonalysts is investigating the development of a Synthetic Task Environment (STE). In a previous report, we documented the findings of a recent outreach effort in which we asked military Subject Matter Experts (SMEs) and other researchers in the Human-AI teaming domain to identify the qualities that they most valued in a testbed. A surprising finding from that outreach was that several respondents recommended that our team look into existing human-AI teaming testbeds, rather than creating something new. Based on that recommendation, we conducted a systematic investigation of the associated landscape. In this report, we describe the results of that investigation. Building on the survey results, we developed testbed evaluation criteria, identified potential testbeds, and conducted qualitative and quantitative evaluations of candidate testbeds. The evaluation process led to five candidate testbeds for the research team to consider. In the coming months, we will assess the viability of the various alternatives and begin to execute our program of research.