RoboVAST: Automated Scenario-Based Validation of Robots at Scale
For roboticists and validation engineers, this provides a scalable, reproducible methodology for systematic robot testing, though it is an incremental improvement over existing scenario-based approaches.
The paper tackles the problem of manual and unscalable scenario-based validation of robots. They propose RoboVAST, a framework that enables automated, compositional scenario generation and execution, demonstrated with over 100,000 runs across 5 maps, achieving 5480 scenario configurations and 1800+ hours of simulation.
Validation of robotic systems critically depends on the operating conditions under which they are assessed. Scenario selection and variation are often manual, experience-driven, and difficult to scale, which harms reproducibility and weakens validation conclusions. We propose a scenario-based methodology that models scenarios compositionally and formalizes how these dimensions are varied, instantiated, executed, and interpreted. Building on this, we introduce RoboVAST, a framework that realizes declarative campaign specifications, plugin-based scenario generation, and scalable containerized execution with integrated result analysis. We demonstrate the approach with a navigation dataset comprising 5480 scenario configurations and over 100000 execution runs across five indoor maps with varied paths, sensor noise, software parameters, and obstacle settings, totaling more than 1800 hours of simulated operation and 1873 km traveled. Twenty repetitions per configuration allow us to distinguish systematic failures from stochastic anomalies.