Conceptual capacity and effective complexity of neural networks
This work addresses the challenge of understanding and quantifying the true complexity of neural networks for researchers and practitioners, though it appears incremental as it builds on existing concepts like tangent spaces and PAC learning.
The authors tackled the problem of measuring the effective complexity of neural networks by proposing a new measure based on the diversity of tangent spaces, which correlates with generalization capabilities. Empirical results show that actual capacity can be surprisingly small even in large networks, despite a theoretical maximum equivalent to the number of neurons.
We propose a complexity measure of a neural network mapping function based on the diversity of the set of tangent spaces from different inputs. Treating each tangent space as a linear PAC concept we use an entropy-based measure of the bundle of concepts in order to estimate the conceptual capacity of the network. The theoretical maximal capacity of a ReLU network is equivalent to the number of its neurons. In practice however, due to correlations between neuron activities within the network, the actual capacity can be remarkably small, even for very big networks. Empirical evaluations show that this new measure is correlated with the complexity of the mapping function and thus the generalisation capabilities of the corresponding network. It captures the effective, as oppose to the theoretical, complexity of the network function. We also showcase some uses of the proposed measure for analysis and comparison of trained neural network models.