Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failures
This work addresses safety-critical failures in robotic systems using generative world models, representing an incremental improvement by enhancing existing latent safety filters with uncertainty awareness.
The paper tackles the problem of latent safety filters missing novel hazards and overconfidently misclassifying risky out-of-distribution situations as safe by introducing an uncertainty-aware latent safety filter that uses epistemic uncertainty to identify unseen potential hazards. The result is a filter that reliably safeguards arbitrary policies from both known and unseen safety hazards, as demonstrated in simulation and hardware experiments on vision-based control tasks with a Franka manipulator.
Recent advances in generative world models have enabled classical safe control methods, such as Hamilton-Jacobi (HJ) reachability, to generalize to complex robotic systems operating directly from high-dimensional sensor observations. However, obtaining comprehensive coverage of all safety-critical scenarios during world model training is extremely challenging. As a result, latent safety filters built on top of these models may miss novel hazards and even fail to prevent known ones, overconfidently misclassifying risky out-of-distribution (OOD) situations as safe. To address this, we introduce an uncertainty-aware latent safety filter that proactively steers robots away from both known and unseen failures. Our key idea is to use the world model's epistemic uncertainty as a proxy for identifying unseen potential hazards. We propose a principled method to detect OOD world model predictions by calibrating an uncertainty threshold via conformal prediction. By performing reachability analysis in an augmented state space-spanning both the latent representation and the epistemic uncertainty-we synthesize a latent safety filter that can reliably safeguard arbitrary policies from both known and unseen safety hazards. In simulation and hardware experiments on vision-based control tasks with a Franka manipulator, we show that our uncertainty-aware safety filter preemptively detects potential unsafe scenarios and reliably proposes safe, in-distribution actions. Video results can be found on the project website at https://cmu-intentlab.github.io/UNISafe