Autonomous Capability Assessment of Sequential Decision-Making Systems in Stochastic Settings (Extended Version)
This addresses the understudied issue of safe AI usage by providing a method for capability assessment, though it is incremental as it builds on existing active-learning and modeling techniques.
The paper tackles the problem of enabling users to assess the capabilities of black-box AI systems with sequential decision-making in stochastic settings, presenting an active-learning approach that learns an interpretable probabilistic model and is shown to be few-shot generalizable and sample-efficient in empirical evaluations.
It is essential for users to understand what their AI systems can and can't do in order to use them safely. However, the problem of enabling users to assess AI systems with sequential decision-making (SDM) capabilities is relatively understudied. This paper presents a new approach for modeling the capabilities of black-box AI systems that can plan and act, along with the possible effects and requirements for executing those capabilities in stochastic settings. We present an active-learning approach that can effectively interact with a black-box SDM system and learn an interpretable probabilistic model describing its capabilities. Theoretical analysis of the approach identifies the conditions under which the learning process is guaranteed to converge to the correct model of the agent; empirical evaluations on different agents and simulated scenarios show that this approach is few-shot generalizable and can effectively describe the capabilities of arbitrary black-box SDM agents in a sample-efficient manner.