LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
This work exposes critical robustness issues in VLA models for robotics, challenging benchmark assumptions and highlighting the need for more reliable evaluation practices.
The paper tackled the problem of hidden weaknesses in Vision-Language-Action models by conducting a systematic vulnerability analysis with controlled perturbations across seven dimensions, revealing that performance drops from 95% to below 30% under modest perturbations and models often ignore language instructions.
Visual-Language-Action (VLA) models report impressive success rates on robotic manipulation benchmarks, yet these results may mask fundamental weaknesses in robustness. We perform a systematic vulnerability analysis by introducing controlled perturbations across seven dimensions: objects layout, camera viewpoints, robot initial states, language instructions, light conditions, background textures and sensor noise. We comprehensively analyzed multiple state-of-the-art models and revealed consistent brittleness beneath apparent competence. Our analysis exposes critical weaknesses: models exhibit extreme sensitivity to perturbation factors, including camera viewpoints and robot initial states, with performance dropping from 95% to below 30% under modest perturbations. Surprisingly, models are largely insensitive to language variations, with further experiments revealing that models tend to ignore language instructions completely. Our findings challenge the assumption that high benchmark scores equate to true competency and highlight the need for evaluation practices that assess reliability under realistic variation.