The Power of Learned Locally Linear Models for Nonlinear Policy Optimization
This work addresses control optimization for nonlinear systems, offering a rigorous analysis with potential improvements in sample efficiency, though it appears incremental in its approach.
The paper tackles the problem of learning-based control for nonlinear systems by analyzing an algorithm that iteratively estimates local linear models and performs policy updates, achieving polynomial sample complexity and overcoming exponential dependence on the problem horizon.
A common pipeline in learning-based control is to iteratively estimate a model of system dynamics, and apply a trajectory optimization algorithm - e.g.~$\mathtt{iLQR}$ - on the learned model to minimize a target cost. This paper conducts a rigorous analysis of a simplified variant of this strategy for general nonlinear systems. We analyze an algorithm which iterates between estimating local linear models of nonlinear system dynamics and performing $\mathtt{iLQR}$-like policy updates. We demonstrate that this algorithm attains sample complexity polynomial in relevant problem parameters, and, by synthesizing locally stabilizing gains, overcomes exponential dependence in problem horizon. Experimental results validate the performance of our algorithm, and compare to natural deep-learning baselines.