Convexity in ReLU Neural Networks: beyond ICNNs?
This work addresses the need for rigorous guarantees in learning-based methods for mathematical imaging, though it is incremental in extending theoretical understanding of convex neural networks.
The paper tackles the problem of characterizing convexity in ReLU neural networks to explore the expressivity of Input Convex Neural Networks (ICNNs), providing necessary and sufficient conditions and showing that for 1-hidden-layer networks, convex functions can be expressed by ICNNs, but this fails with more layers.
Convex functions and their gradients play a critical role in mathematical imaging, from proximal optimization to Optimal Transport. The successes of deep learning has led many to use learning-based methods, where fixed functions or operators are replaced by learned neural networks. Regardless of their empirical superiority, establishing rigorous guarantees for these methods often requires to impose structural constraints on neural architectures, in particular convexity. The most popular way to do so is to use so-called Input Convex Neural Networks (ICNNs). In order to explore the expressivity of ICNNs, we provide necessary and sufficient conditions for a ReLU neural network to be convex. Such characterizations are based on product of weights and activations, and write nicely for any architecture in the path-lifting framework. As particular applications, we study our characterizations in depth for 1 and 2-hidden-layer neural networks: we show that every convex function implemented by a 1-hidden-layer ReLU network can be also expressed by an ICNN with the same architecture; however this property no longer holds with more layers. Finally, we provide a numerical procedure that allows an exact check of convexity for ReLU neural networks with a large number of affine regions.