Automated Testing with Temporal Logic Specifications for Robotic Controllers using Adaptive Experiment Design
This addresses the challenge of costly and infeasible exhaustive testing in robotics for researchers and engineers, though it is incremental as it builds on existing techniques like smooth robustness metrics and adaptive experiment design.
The paper tackles the problem of efficiently testing robotic controllers with black-box components against temporal logic specifications by developing an adaptive experiment design method that selects the most informative inputs to build a surrogate model of robustness, resulting in sample-efficient descriptions of system performance and enabling assessment under domain shift.
Many robot control scenarios involve assessing system robustness against a task specification. If either the controller or environment are composed of "black-box" components with unknown dynamics, we cannot rely on formal verification to assess our system. Assessing robustness via exhaustive testing is also often infeasible if the space of environments is large compared to experiment cost. Given limited budget, we provide a method to choose experiment inputs which give greatest insight into system performance against a given specification across the domain. By combining smooth robustness metrics for signal temporal logic with techniques from adaptive experiment design, our method chooses the most informative experimental inputs by incrementally constructing a surrogate model of the specification robustness. This model then chooses the next experiment to be in an area where there is either high prediction error or uncertainty. Our experiments show how this adaptive experimental design technique results in sample-efficient descriptions of system robustness. Further, we show how to use the model built via the experiment design process to assess the behaviour of a data-driven control system under domain shift.