Input Convex Gradient Networks
This work addresses the challenge of efficiently representing convex gradients for applications like generative flow models, though it appears incremental with limited empirical validation.
The paper tackled the problem of modeling convex gradients, which are expressive vector fields used in optimal transport and generative models, by introducing Input Convex Gradient Networks (ICGNs) and showing that a single-layer ICGN can fit a toy example better than a single-layer Input-Convex Neural Network (ICNN).
The gradients of convex functions are expressive models of non-trivial vector fields. For example, Brenier's theorem yields that the optimal transport map between any two measures on Euclidean space under the squared distance is realized as a convex gradient, which is a key insight used in recent generative flow models. In this paper, we study how to model convex gradients by integrating a Jacobian-vector product parameterized by a neural network, which we call the Input Convex Gradient Network (ICGN). We theoretically study ICGNs and compare them to taking the gradient of an Input-Convex Neural Network (ICNN), empirically demonstrating that a single layer ICGN can fit a toy example better than a single layer ICNN. Lastly, we explore extensions to deeper networks and connections to constructions from Riemannian geometry.