CDFlow: Building Invertible Layers with Circulant and Diagonal Matrices
This work addresses a key bottleneck in normalizing flows for scalable generative modeling, offering practical speedups for researchers and practitioners in machine learning.
The paper tackles the challenge of designing efficient invertible linear layers for normalizing flows by introducing a novel layer based on circulant and diagonal matrices, which reduces parameter complexity from O(n^2) to O(mn) and accelerates matrix inversion from O(n^3) to O(mn log n) and log-determinant computation from O(n^3) to O(mn).
Normalizing flows are deep generative models that enable efficient likelihood estimation and sampling through invertible transformations. A key challenge is to design linear layers that enhance expressiveness while maintaining efficient computation of the Jacobian determinant and inverse. We introduce a novel invertible linear layer based on the product of circulant and diagonal matrices. This decomposition reduces parameter complexity from $\mathcal{O}(n^2)$ to $\mathcal{O}(mn)$ using $m$ diagonal matrices and $m-1$ circulant matrices while still approximating general linear transformations. By leveraging the Fast Fourier Transform, our approach reduces the time complexity of matrix inversion from $\mathcal{O}(n^3)$ to $\mathcal{O}(mn\log n)$ and that of computing the log-determinant from $\mathcal{O}(n^3)$ to $\mathcal{O}(mn)$, where $n$ is the input dimension. We build upon this layer to develop Circulant-Diagonal Flow (CDFlow), which achieves strong density estimation on natural image datasets and effectively models data with inherent periodic structure. Furthermore, CDFlow significantly accelerates key operations in normalizing flows, providing practical benefits for scalable generative modeling.