A Differential Game Theoretic Neural Optimizer for Training Residual Networks
This work addresses the challenge of applying trajectory-optimization frameworks to modern neural network architectures, offering an incremental improvement in optimization methods for deep learning practitioners.
The authors tackled the problem of extending a differential dynamic programming (DDP) neural optimizer to modern architectures like residual networks and convolution layers, resulting in a Game Theoretic DDP (GT-DDP) optimizer that improved training convergence and reduced variance on image classification datasets such as MNIST and CIFAR100.
Connections between Deep Neural Networks (DNNs) training and optimal control theory has attracted considerable attention as a principled tool of algorithmic design. Differential Dynamic Programming (DDP) neural optimizer is a recently proposed method along this line. Despite its empirical success, the applicability has been limited to feedforward networks and whether such a trajectory-optimization inspired framework can be extended to modern architectures remains unclear. In this work, we derive a generalized DDP optimizer that accepts both residual connections and convolution layers. The resulting optimal control representation admits a game theoretic perspective, in which training residual networks can be interpreted as cooperative trajectory optimization on state-augmented dynamical systems. This Game Theoretic DDP (GT-DDP) optimizer enjoys the same theoretic connection in previous work, yet generates a much complex update rule that better leverages available information during network propagation. Evaluation on image classification datasets (e.g. MNIST and CIFAR100) shows an improvement in training convergence and variance reduction over existing methods. Our approach highlights the benefit gained from architecture-aware optimization.