Test and Evaluation Framework for Multi-Agent Systems of Autonomous Intelligent Agents
This work addresses the need for robust evaluation in autonomous intelligent systems, but it is incremental as it builds on existing test and evaluation concepts without introducing a new method or paradigm.
The paper tackles the problem of testing and evaluating complex multi-agent systems with embedded AI by proposing a unifying framework that integrates testing throughout the development and operational life cycles, addressing challenges like limited resources and hierarchical integration.
Test and evaluation is a necessary process for ensuring that engineered systems perform as intended under a variety of conditions, both expected and unexpected. In this work, we consider the unique challenges of developing a unifying test and evaluation framework for complex ensembles of cyber-physical systems with embedded artificial intelligence. We propose a framework that incorporates test and evaluation throughout not only the development life cycle, but continues into operation as the system learns and adapts in a noisy, changing, and contended environment. The framework accounts for the challenges of testing the integration of diverse systems at various hierarchical scales of composition while respecting that testing time and resources are limited. A generic use case is provided for illustrative purposes and research directions emerging as a result of exploring the use case via the framework are suggested.