Pixels to Proofs: Probabilistically-Safe Latent World Model Control via Parallel Conformal Robust MPC
For vision-based control tasks requiring safety guarantees, this framework provides a probabilistic safety approach that outperforms existing latent world-model and safe-planning methods.
SLS^2 enables safe feedback motion planning from pixels by combining learned latent world models with robust MPC and conformal prediction, achieving improved goal-reaching and safety over baselines in vision-based control tasks.
We present SLS^2, a framework for safe feedback motion planning from pixels using robust model predictive control (MPC) in learned latent world models. Our approach trains an action-conditioned joint-embedding world model with compact Markovian latent states, enabling efficient gradient-based trajectory optimization through learned latent dynamics. To enforce safety for the true system despite imperfect latent predictions, we inform a GPU-accelerated system level synthesis (SLS) robust MPC scheme with conformal prediction to obtain calibrated latent error bounds and robust latent-space constraint sets. We further learn and conformalize a latent constraint checker, allowing the SLS planner to impose probabilistic safety constraints during closed-loop execution. We evaluate our method on vision-based control tasks, where it improves both goal-reaching performance and safety over latent world-model and safe-planning baselines.