Constructive Universal High-Dimensional Distribution Generation through Deep ReLU Networks
This provides a constructive method for universal distribution generation in machine learning, though it appears incremental as it builds on prior space-filling function research.
The paper tackles the problem of generating high-dimensional target distributions from low-dimensional noise using deep ReLU networks, achieving arbitrarily close approximations with Wasserstein distance driven to zero. It shows the construction incurs no additional error cost compared to using independent random variables.
We present an explicit deep neural network construction that transforms uniformly distributed one-dimensional noise into an arbitrarily close approximation of any two-dimensional Lipschitz-continuous target distribution. The key ingredient of our design is a generalization of the "space-filling" property of sawtooth functions discovered in (Bailey & Telgarsky, 2018). We elicit the importance of depth - in our neural network construction - in driving the Wasserstein distance between the target distribution and the approximation realized by the network to zero. An extension to output distributions of arbitrary dimension is outlined. Finally, we show that the proposed construction does not incur a cost - in terms of error measured in Wasserstein-distance - relative to generating $d$-dimensional target distributions from $d$ independent random variables.