Interpretable Convolutional Neural Networks via Feedforward Design
This work addresses the need for more interpretable and efficient neural network design methods, particularly for researchers and practitioners in machine learning, though it is incremental as it builds on existing transforms like PCA.
The authors tackled the problem of designing interpretable convolutional neural networks without backpropagation by proposing a feedforward design that uses data statistics and a new Saab transform to derive network parameters layer by layer. They compared classification and robustness performance on MNIST and CIFAR-10 datasets against backpropagation-based CNNs, showing competitive results.
The model parameters of convolutional neural networks (CNNs) are determined by backpropagation (BP). In this work, we propose an interpretable feedforward (FF) design without any BP as a reference. The FF design adopts a data-centric approach. It derives network parameters of the current layer based on data statistics from the output of the previous layer in a one-pass manner. To construct convolutional layers, we develop a new signal transform, called the Saab (Subspace Approximation with Adjusted Bias) transform. It is a variant of the principal component analysis (PCA) with an added bias vector to annihilate activation's nonlinearity. Multiple Saab transforms in cascade yield multiple convolutional layers. As to fully-connected (FC) layers, we construct them using a cascade of multi-stage linear least squared regressors (LSRs). The classification and robustness (against adversarial attacks) performances of BP- and FF-designed CNNs applied to the MNIST and the CIFAR-10 datasets are compared. Finally, we comment on the relationship between BP and FF designs.