Fractal and Regular Geometry of Deep Neural Networks
This work provides theoretical insights into the geometry of neural networks, which is incremental for understanding network behavior in machine learning.
The paper investigates how the geometric properties of random deep neural networks vary with depth and activation functions, showing that less regular activations lead to fractal boundary volumes with increasing Hausdorff dimension, while more regular activations result in boundary volumes that converge, remain constant, or diverge based on a spectral parameter.
We study the geometric properties of random neural networks by investigating the boundary volumes of their excursion sets for different activation functions, as the depth increases. More specifically, we show that, for activations which are not very regular (e.g., the Heaviside step function), the boundary volumes exhibit fractal behavior, with their Hausdorff dimension monotonically increasing with the depth. On the other hand, for activations which are more regular (e.g., ReLU, logistic and $\tanh$), as the depth increases, the expected boundary volumes can either converge to zero, remain constant or diverge exponentially, depending on a single spectral parameter which can be easily computed. Our theoretical results are confirmed in some numerical experiments based on Monte Carlo simulations.