Koopman Spectrum Nonlinear Regulators and Efficient Online Learning
This offers a new paradigm for improving motion efficiency and predictability in robotics and control systems, though it is incremental in extending classical methods to nonlinear decision making.
The paper tackles the problem of unnatural and inefficient motions in reinforcement learning by introducing a Koopman spectrum cost for controlling nonlinear systems, resulting in smoother behaviors like oscillators and closed loops, and provides an online learning algorithm with sub-linear regret.
Most modern reinforcement learning algorithms optimize a cumulative single-step cost along a trajectory. The optimized motions are often 'unnatural', representing, for example, behaviors with sudden accelerations that waste energy and lack predictability. In this work, we present a novel paradigm of controlling nonlinear systems via the minimization of the Koopman spectrum cost: a cost over the Koopman operator of the controlled dynamics. This induces a broader class of dynamical behaviors that evolve over stable manifolds such as nonlinear oscillators, closed loops, and smooth movements. We demonstrate that some dynamics characterizations that are not possible with a cumulative cost are feasible in this paradigm, which generalizes the classical eigenstructure and pole assignments to nonlinear decision making. Moreover, we present a sample efficient online learning algorithm for our problem that enjoys a sub-linear regret bound under some structural assumptions.