On the Depth of Monotone ReLU Neural Networks and ICNNs
This work addresses theoretical limitations in neural network design for researchers in machine learning theory, providing foundational insights into model expressivity.
The paper tackles the expressivity of monotone ReLU networks and input convex neural networks (ICNNs) in terms of depth, proving that monotone networks cannot compute or approximate the maximum function, and establishing sharp depth lower bounds and separations between these models and standard ReLU networks.
We study two models of ReLU neural networks: monotone networks (ReLU$^+$) and input convex neural networks (ICNN). Our focus is on expressivity, mostly in terms of depth, and we prove the following lower bounds. For the maximum function MAX$_n$ computing the maximum of $n$ real numbers, we show that ReLU$^+$ networks cannot compute MAX$_n$, or even approximate it. We prove a sharp $n$ lower bound on the ICNN depth complexity of MAX$_n$. We also prove depth separations between ReLU networks and ICNNs; for every $k$, there is a depth-2 ReLU network of size $O(k^2)$ that cannot be simulated by a depth-$k$ ICNN. The proofs are based on deep connections between neural networks and polyhedral geometry, and also use isoperimetric properties of triangulations.