Lyapunov Density Models: Constraining Distribution Shift in Learning-Based Control
This addresses distribution shift issues in deploying learning-based control algorithms, offering a mechanism to enhance safety and reliability, though it appears incremental as it builds on existing concepts from control theory and machine learning.
The paper tackles the problem of unpredictable outputs from learned models and policies on out-of-distribution inputs in learning-based control by proposing Lyapunov density models, which combine Lyapunov stability and density estimation to constrain agents to in-distribution states and actions, providing guarantees on staying in-distribution over trajectories.
Learned models and policies can generalize effectively when evaluated within the distribution of the training data, but can produce unpredictable and erroneous outputs on out-of-distribution inputs. In order to avoid distribution shift when deploying learning-based control algorithms, we seek a mechanism to constrain the agent to states and actions that resemble those that it was trained on. In control theory, Lyapunov stability and control-invariant sets allow us to make guarantees about controllers that stabilize the system around specific states, while in machine learning, density models allow us to estimate the training data distribution. Can we combine these two concepts, producing learning-based control algorithms that constrain the system to in-distribution states using only in-distribution actions? In this work, we propose to do this by combining concepts from Lyapunov stability and density estimation, introducing Lyapunov density models: a generalization of control Lyapunov functions and density models that provides guarantees on an agent's ability to stay in-distribution over its entire trajectory.