Neural Networks Should Be Wide Enough to Learn Disconnected Decision Regions
This addresses the problem of understanding neural network expressiveness for researchers, showing width is crucial for learning complex patterns, though it is incremental as it builds on prior depth-focused studies.
The paper investigates conditions under which neural networks produce connected decision regions, finding that pyramidal networks with certain activation functions like leaky ReLU cannot create disconnected regions, implying sufficient width is necessary for this capability. This result has implications for network design and adversarial robustness.
In the recent literature the important role of depth in deep learning has been emphasized. In this paper we argue that sufficient width of a feedforward network is equally important by answering the simple question under which conditions the decision regions of a neural network are connected. It turns out that for a class of activation functions including leaky ReLU, neural networks having a pyramidal structure, that is no layer has more hidden units than the input dimension, produce necessarily connected decision regions. This implies that a sufficiently wide hidden layer is necessary to guarantee that the network can produce disconnected decision regions. We discuss the implications of this result for the construction of neural networks, in particular the relation to the problem of adversarial manipulation of classifiers.