Generalization of the Change of Variables Formula with Applications to Residual Flows
This work addresses a bottleneck in normalizing flow design for machine learning practitioners, offering incremental improvements in model flexibility.
The paper tackles the limitation of requiring smooth transformations in normalizing flows by introducing $\mathcal{L}$-diffeomorphisms, which relax smoothness on zero-measure sets, enabling the use of non-smooth functions like ReLU and improving performance in residual flows.
Normalizing flows leverage the Change of Variables Formula (CVF) to define flexible density models. Yet, the requirement of smooth transformations (diffeomorphisms) in the CVF poses a significant challenge in the construction of these models. To enlarge the design space of flows, we introduce $\mathcal{L}$-diffeomorphisms as generalized transformations which may violate these requirements on zero Lebesgue-measure sets. This relaxation allows e.g. the use of non-smooth activation functions such as ReLU. Finally, we apply the obtained results to planar, radial, and contractive residual flows.