Local Observability and Moving Horizon Estimation-based Training of Feedforward Neural Networks
This work provides a control-theoretic framework for training neural networks with convergence guarantees, which is relevant for researchers interested in rigorous analysis of neural network training.
The authors propose a moving horizon estimation (MHE)-based training method for feedforward neural networks with ReLU activations, reformulating the network as a dynamical system to analyze local observability. They derive conditions for observability in two-layer networks and show that multi-layer networks generally fail the observability rank condition, then design a persistently exciting input to guarantee convergence of the MHE-based training.
In this paper, we propose a moving horizon estimation (MHE)-based training method for feedforward neural networks (FNNs) with rectified linear unit (ReLU) activation functions to determine their ideal weights from a control-theoretic perspective. This allows for a rigorous theoretical analysis of the trained network. First, we reformulate the FNN as a dynamical system with the weights as states. Then, we investigate the local observability of such a system. For two-layer FNNs with fixed output weights, we derive a sufficient condition under which the observability rank condition holds, ensuring a locally observable state. We also show that multi-layer FNNs in general fail to satisfy the observability rank condition. Based on this analysis, we develop a persistently exciting (PE) input design method, which renders a state distinguishable from its neighbors. The resulting local observability provides convergence guarantees for the proposed MHE-based training, where only the projection of the state onto the observable subspace is updated using a fixed-length window of input-output data. The effectiveness of the approach is illustrated via numerical examples.