How VLAs Fail Differently: Black-Box Action Monitoring Reveals Architecture-Specific Failure Signatures
For practitioners deploying VLA models in safety-critical robotics, this work provides actionable guidelines for selecting failure monitors based on architecture type, revealing that common velocity-based safety checks are ineffective for continuous VLAs.
The paper discovers that Vision-Language-Action (VLA) models fail in architecture-specific ways, showing that direction reversal rate is a universal failure predictor (AUROC up to 0.93) while jerk monitoring works only for discrete-token architectures, and velocity monitoring is ineffective for continuous ones. This demonstrates that no single monitor works universally, requiring architecture-matched selection.
We discover that VLA architectures fail in fundamentally different, predictable ways at the motor-command level. Running VQ-BeT, Diffusion Policy, and ACT on identical evaluation protocols (n=450 episodes across PushT and ALOHA 14-DOF bimanual manipulation), we find: (1) direction reversal rate is a universal failure predictor across all three architectures (AUROC=0.93, 0.79, 0.91; p<0.001); (2) jerk monitoring is predictive only for discrete-token architectures, following a discrete-to-continuous gradient (0.88, 0.69, 0.41); (3) velocity violations alone are non-predictive everywhere (AUROC 0.41-0.69), yet velocity checking is the most common safety mechanism in VLA deployment code; and (4) for continuous-family VLAs, velocity monitoring provides effectively zero predictive signal (AUROC=0.52 on ACT, 0.41 on Diffusion), proving that architecture-matched monitor selection is essential. These results quantify a monitoring consequence of the well-known discrete/continuous VLA distinction: the two families produce qualitatively different failure signatures that require different monitors. No single monitor works universally; architecture-matched selection is required. This finding was enabled by SafeContract, a training-free, black-box action monitoring toolkit with conformal calibration. Code: https://github.com/krishnam94/vla-edge