Regularization of polynomial networks for image recognition
This work addresses the need for more interpretable models in image recognition, though it is incremental as it focuses on closing an existing performance gap rather than introducing a new paradigm.
The authors tackled the performance gap between Polynomial Networks (PNs) and Deep Neural Networks (DNNs) by introducing a class of PNs that match ResNet performance across six benchmarks, achieving similar results through strong regularization and more parameter-efficient D-PolyNets.
Deep Neural Networks (DNNs) have obtained impressive performance across tasks, however they still remain as black boxes, e.g., hard to theoretically analyze. At the same time, Polynomial Networks (PNs) have emerged as an alternative method with a promising performance and improved interpretability but have yet to reach the performance of the powerful DNN baselines. In this work, we aim to close this performance gap. We introduce a class of PNs, which are able to reach the performance of ResNet across a range of six benchmarks. We demonstrate that strong regularization is critical and conduct an extensive study of the exact regularization schemes required to match performance. To further motivate the regularization schemes, we introduce D-PolyNets that achieve a higher-degree of expansion than previously proposed polynomial networks. D-PolyNets are more parameter-efficient while achieving a similar performance as other polynomial networks. We expect that our new models can lead to an understanding of the role of elementwise activation functions (which are no longer required for training PNs). The source code is available at https://github.com/grigorisg9gr/regularized_polynomials.