Learning Parities with Neural Networks
This addresses the gap in understanding why neural networks outperform linear methods, focusing on non-linear learnability for machine learning researchers.
The paper tackles the problem of learning inherently non-linear models with neural networks, specifically showing that sparse parities are learnable via gradient descent on a depth-two network under certain distributions, while linear methods cannot efficiently learn them under the same conditions.
In recent years we see a rapidly growing line of research which shows learnability of various models via common neural network algorithms. Yet, besides a very few outliers, these results show learnability of models that can be learned using linear methods. Namely, such results show that learning neural-networks with gradient-descent is competitive with learning a linear classifier on top of a data-independent representation of the examples. This leaves much to be desired, as neural networks are far more successful than linear methods. Furthermore, on the more conceptual level, linear models don't seem to capture the "deepness" of deep networks. In this paper we make a step towards showing leanability of models that are inherently non-linear. We show that under certain distributions, sparse parities are learnable via gradient decent on depth-two network. On the other hand, under the same distributions, these parities cannot be learned efficiently by linear methods.