Representing Neural Network Layers as Linear Operations via Koopman Operator Theory
This provides a linear perspective for analyzing neural networks, which is incremental as it applies existing dynamical systems theory to a new context.
The paper tackles the problem of understanding and controlling neural networks by reframing them as dynamical systems and using Koopman operator theory to replace nonlinear layers with linear operators, achieving model accuracies of up to 97.3% on the Yin-Yang dataset and 95.8% on MNIST, compared to original accuracies of 98.4% and 97.2% respectively.
The strong performance of simple neural networks is often attributed to their nonlinear activations. However, a linear view of neural networks makes understanding and controlling networks much more approachable. We draw from a dynamical systems view of neural networks, offering a fresh perspective by using Koopman operator theory and its connections with dynamic mode decomposition (DMD). Together, they offer a framework for linearizing dynamical systems by embedding the system into an appropriate observable space. By reframing a neural network as a dynamical system, we demonstrate that we can replace the nonlinear layer in a pretrained multi-layer perceptron (MLP) with a finite-dimensional linear operator. In addition, we analyze the eigenvalues of DMD and the right singular vectors of SVD, to present evidence that time-delayed coordinates provide a straightforward and highly effective observable space for Koopman theory to linearize a network layer. Consequently, we replace layers of an MLP trained on the Yin-Yang dataset with predictions from a DMD model, achieving a mdoel accuracy of up to 97.3%, compared to the original 98.4%. In addition, we replace layers in an MLP trained on the MNIST dataset, achieving up to 95.8%, compared to the original 97.2% on the test set.