Digital implementations of deep feature extractors are intrinsically informative
This work addresses a foundational issue in machine learning by providing theoretical insights into energy propagation, which is incremental but relevant for optimizing deep learning architectures.
The paper tackles the problem of balancing computational complexity and expressiveness in deep feature extractors by proving an upper bound for energy propagation speed across various neural network models, including CNNs, and demonstrates global exponential energy decay for specific cases.
Rapid information (energy) propagation in deep feature extractors is crucial to balance computational complexity versus expressiveness as a representation of the input. We prove an upper bound for the speed of energy propagation in a unified framework that covers different neural network models, both over Euclidean and non-Euclidean domains. Additional structural information about the signal domain can be used to explicitly determine or improve the rate of decay. To illustrate this, we show global exponential energy decay for a range of 1) feature extractors with discrete-domain input signals, and 2) convolutional neural networks (CNNs) via scattering over locally compact abelian (LCA) groups.