Extraction of nonlinearity in neural networks with Koopman operator
This work addresses the interpretability and compression of neural networks for researchers and practitioners, but it is incremental as it applies an existing physics method to a new context.
The paper tackles the problem of understanding the essential nonlinearity in deep neural networks by using the Koopman operator to replace nonlinear layers with linear ones, achieving sufficient accuracy in classification tasks and high compression ratios through pruning.
Nonlinearity plays a crucial role in deep neural networks. In this paper, we investigate the degree to which the nonlinearity of the neural network is essential. For this purpose, we employ the Koopman operator, extended dynamic mode decomposition, and the tensor-train format. The Koopman operator approach has been recently developed in physics and nonlinear sciences; the Koopman operator deals with the time evolution in the observable space instead of the state space. Since we can replace the nonlinearity in the state space with the linearity in the observable space, it is a hopeful candidate for understanding complex behavior in nonlinear systems. Here, we analyze learned neural networks for the classification problems. As a result, the replacement of the nonlinear middle layers with the Koopman matrix yields enough accuracy in numerical experiments. In addition, we confirm that the pruning of the Koopman matrix gives sufficient accuracy even at high compression ratios. These results indicate the possibility of extracting some features in the neural networks with the Koopman operator approach.