Contracting Neural Networks: Sharp LMI Conditions with Applications to Integral Control and Deep Learning
This work addresses the need for stable and efficient neural network designs in control and deep learning, though it is incremental in building on existing contractivity theory.
The paper tackled the problem of ensuring contractivity in recurrent neural networks by deriving sharp LMI conditions for synaptic matrices, and applied these results to design integral controllers for reference tracking and improve the expressivity of Implicit Neural Networks, achieving competitive performance on image classification benchmarks with fewer parameters.
This paper studies contractivity of firing-rate and Hopfield recurrent neural networks. We derive sharp LMI conditions on the synaptic matrices that characterize contractivity of both architectures, for activation functions that are either non-expansive or monotone non-expansive, in both continuous and discrete time. We establish structural relationships among these conditions, including connections to Schur diagonal stability and the recovery of optimal contraction rates for symmetric synaptic matrices. We demonstrate the utility of these results through two applications. First, we develop an LMI-based design procedure for low-gain integral controllers enabling reference tracking in contracting firing rate networks. Second, we provide an exact parameterization of weight matrices that guarantee contraction and use it to improve the expressivity of Implicit Neural Networks, achieving competitive performance on image classification benchmarks with fewer parameters.