Tropical Geometry of Deep Neural Networks
This provides a novel mathematical framework for understanding neural networks, potentially benefiting researchers in machine learning theory.
The paper connects feedforward neural networks with ReLU activation to tropical geometry, showing they are equivalent to tropical rational maps, and deduces that deeper networks are exponentially more expressive than shallow ones.
We establish, for the first time, connections between feedforward neural networks with ReLU activation and tropical geometry --- we show that the family of such neural networks is equivalent to the family of tropical rational maps. Among other things, we deduce that feedforward ReLU neural networks with one hidden layer can be characterized by zonotopes, which serve as building blocks for deeper networks; we relate decision boundaries of such neural networks to tropical hypersurfaces, a major object of study in tropical geometry; and we prove that linear regions of such neural networks correspond to vertices of polytopes associated with tropical rational functions. An insight from our tropical formulation is that a deeper network is exponentially more expressive than a shallow network.