Least Squares Training of Quadratic Convolutional Neural Networks with Applications to System Theory
This work addresses the need for analyzable and efficient neural networks in safety-critical systems such as aircraft or autonomous vehicles, though it is incremental as it focuses on shallow networks.
The paper tackles the training of 2-layer convolutional neural networks with quadratic activations using a least squares formulation, resulting in an analytic expression for globally optimal weights and a quadratic input-output equation, which reduces training time significantly with minimal accuracy loss in applications like system identification and GPS position estimation.
This paper provides a least squares formulation for the training of a 2-layer convolutional neural network using quadratic activation functions, a 2-norm loss function, and no regularization term. Using this method, an analytic expression for the globally optimal weights is obtained alongside a quadratic input-output equation for the network. These properties make the network a viable tool in system theory by enabling further analysis, such as the sensitivity of the output to perturbations in the input, which is crucial for safety-critical systems such as aircraft or autonomous vehicles. The least squares method is compared to previously proposed strategies for training quadratic networks and to a back-propagation-trained ReLU network. The proposed method is applied to a system identification problem and a GPS position estimation problem. The least squares network is shown to have a significantly reduced training time with minimal compromises on prediction accuracy alongside the advantages of having an analytic input-output equation. Although these results only apply to 2-layer networks, this paper motivates the exploration of deeper quadratic networks in the context of system theory.