A Test-Function Approach to Incremental Stability
This work addresses a foundational problem in control theory and reinforcement learning integration, offering a new perspective distinct from traditional Lyapunov methods, but it is incremental in nature as it builds on existing stability concepts.
The paper tackles the challenge of analyzing incremental stability in control systems by introducing a novel framework that connects reinforcement learning value functions to stability properties, establishing an equivalence between incremental input-to-state stability and the regularity of value functions under adversarial reward selection.
This paper presents a novel framework for analyzing Incremental-Input-to-State Stability ($δ$ISS) based on the idea of using rewards as "test functions." Whereas control theory traditionally deals with Lyapunov functions that satisfy a time-decrease condition, reinforcement learning (RL) value functions are constructed by exponentially decaying a Lipschitz reward function that may be non-smooth and unbounded on both sides. Thus, these RL-style value functions cannot be directly understood as Lyapunov certificates. We develop a new equivalence between a variant of incremental input-to-state stability of a closed-loop system under given a policy, and the regularity of RL-style value functions under adversarial selection of a Hölder-continuous reward function. This result highlights that the regularity of value functions, and their connection to incremental stability, can be understood in a way that is distinct from the traditional Lyapunov-based approach to certifying stability in control theory.