Energy Propagation in Scattering Convolution Networks Can Be Arbitrarily Slow
This work addresses the stability of signal processing in scattering networks for researchers in machine learning and signal analysis, revealing fundamental limitations in energy propagation.
The paper tackled the problem of energy decay in deep convolutional neural networks, specifically wavelet scattering transforms, and showed that energy decay can be arbitrarily slow for generic signals, contrasting with previous exponential decay results for Gabor-based transforms.
We analyze energy decay for deep convolutional neural networks employed as feature extractors, including Mallat's wavelet scattering transform. For time-frequency scattering transforms based on Gabor filters, previous work has established that energy decay is exponential for arbitrary square-integrable input signals. In contrast, our main results allow proving that this is false for wavelet scattering in arbitrary dimensions. Specifically, we show that the energy decay of wavelet and wavelet-like scattering transforms acting on generic square-integrable signals can be arbitrarily slow. Importantly, this slow decay behavior holds for dense subsets of $L^2(\mathbb{R}^d)$, indicating that rapid energy decay is generally an unstable property of signals. We complement these findings with positive results that allow us to infer fast (up to exponential) energy decay for generalized Sobolev spaces tailored to the frequency localization of the underlying filter bank. Both negative and positive results highlight that energy decay in scattering networks critically depends on the interplay between the respective frequency localizations of both the signal and the filters used.