Controlling the Complexity and Lipschitz Constant improves polynomial nets
This work addresses theoretical and practical robustness issues for Polynomial Nets, an alternative to neural networks, though it is incremental as it builds on existing models with new regularization.
The paper tackles the lack of theoretical generalization and robustness guarantees for Polynomial Nets by deriving complexity bounds and Lipschitz constant bounds for CCP and NCP models, and proposes a regularization scheme that improves accuracy and robustness on six datasets, with further gains when combined with adversarial training.
While the class of Polynomial Nets demonstrates comparable performance to neural networks (NN), it currently has neither theoretical generalization characterization nor robustness guarantees. To this end, we derive new complexity bounds for the set of Coupled CP-Decomposition (CCP) and Nested Coupled CP-decomposition (NCP) models of Polynomial Nets in terms of the $\ell_\infty$-operator-norm and the $\ell_2$-operator norm. In addition, we derive bounds on the Lipschitz constant for both models to establish a theoretical certificate for their robustness. The theoretical results enable us to propose a principled regularization scheme that we also evaluate experimentally in six datasets and show that it improves the accuracy as well as the robustness of the models to adversarial perturbations. We showcase how this regularization can be combined with adversarial training, resulting in further improvements.