Aliasing in Convnets: A Frame-Theoretic Perspective
This addresses stability and generalization issues in CNNs for researchers and practitioners, though it appears incremental as it builds on existing frame-theoretic approaches.
The paper analyzes aliasing in convolutional neural networks with strided layers, developing frame-theoretic methods to characterize Parseval stability and derive practical stability bounds. It introduces two efficient optimization objectives to suppress aliasing and provides closed-form expressions for aliasing effects in random kernels at initialization.
Using a stride in a convolutional layer inherently introduces aliasing, which has implications for numerical stability and statistical generalization. While techniques such as the parametrizations via paraunitary systems have been used to promote orthogonal convolution and thus ensure Parseval stability, a general analysis of aliasing and its effects on the stability has not been done in this context. In this article, we adapt a frame-theoretic approach to describe aliasing in convolutional layers with 1D kernels, leading to practical estimates for stability bounds and characterizations of Parseval stability, that are tailored to take short kernel sizes into account. From this, we derive two computationally very efficient optimization objectives that promote Parseval stability via systematically suppressing aliasing. Finally, for layers with random kernels, we derive closed-form expressions for the expected value and variance of the terms that describe the aliasing effects, revealing fundamental insights into the aliasing behavior at initialization.