Equivalence in Deep Neural Networks via Conjugate Matrix Ensembles
This work addresses the challenge of comparing neural architectures for researchers in machine learning and neuroscience, though it appears incremental as it builds on existing ensemble methods from statistical quantum mechanics.
The paper tackled the problem of detecting equivalence among deep learning architectures by developing a numerical method based on Mixed Matrix Ensembles and conjugate circular ensembles, showing that spectral density differences vanish with varying decay rates, which can be applied to Neural Architecture Search and biological network classification.
A numerical approach is developed for detecting the equivalence of deep learning architectures. The method is based on generating Mixed Matrix Ensembles (MMEs) out of deep neural network weight matrices and {\it conjugate circular ensemble} matching the neural architecture topology. Following this, the empirical evidence supports the {\it phenomenon} that difference between spectral densities of neural architectures and corresponding {\it conjugate circular ensemble} are vanishing with different decay rates at the long positive tail part of the spectrum i.e., cumulative Circular Spectral Difference (CSD). This finding can be used in establishing equivalences among different neural architectures via analysis of fluctuations in CSD. We investigated this phenomenon for a wide range of deep learning vision architectures and with circular ensembles originating from statistical quantum mechanics. Practical implications of the proposed method for artificial and natural neural architectures discussed such as the possibility of using the approach in Neural Architecture Search (NAS) and classification of biological neural networks.