Empirical Studies on the Properties of Linear Regions in Deep Neural Networks
This provides insights into DNN behavior that could inspire new optimization methods, though it is incremental as it builds on existing linear region analysis.
The paper investigates how different optimization techniques affect the local properties of linear regions in deep neural networks with piecewise linear activations, finding that they produce completely different linear regions despite similar classification accuracies.
A deep neural network (DNN) with piecewise linear activations can partition the input space into numerous small linear regions, where different linear functions are fitted. It is believed that the number of these regions represents the expressivity of the DNN. This paper provides a novel and meticulous perspective to look into DNNs: Instead of just counting the number of the linear regions, we study their local properties, such as the inspheres, the directions of the corresponding hyperplanes, the decision boundaries, and the relevance of the surrounding regions. We empirically observed that different optimization techniques lead to completely different linear regions, even though they result in similar classification accuracies. We hope our study can inspire the design of novel optimization techniques, and help discover and analyze the behaviors of DNNs.