Gradient Residual Connections
This addresses the challenge of approximating rapidly varying functions in neural networks, which is relevant for tasks like super-resolution, but the approach appears incremental as it builds on existing residual network architectures.
The paper tackles the problem of neural networks struggling to approximate high-frequency functions by proposing gradient residual connections as a complement to standard identity skip connections. The method improves approximation quality on a synthetic high-frequency regression task and achieves comparable performance to standard residual networks on image classification and segmentation.
Existing work has linked properties of a function's gradient to the difficulty of function approximation. Motivated by these insights, we study how gradient information can be leveraged to improve neural network's ability to approximate high-frequency functions, and we propose a gradient-based residual connection as a complement to the standard identity skip connection used in residual networks. We provide simple theoretical intuition for why gradient information can help distinguish inputs and improve the approximation of functions with rapidly varying behaviour. On a synthetic regression task with a high-frequency sinusoidal ground truth, we show that conventional residual connections struggle to capture high-frequency patterns. In contrast, our gradient residual substantially improves approximation quality. We then introduce a convex combination of the standard and gradient residuals, allowing the network to flexibly control how strongly it relies on gradient information. After validating the design choices of our proposed method through an ablation study, we further validate our approach's utility on the single-image super-resolution task, where the underlying function may be high-frequency. Finally, on standard tasks such as image classification and segmentation, our method achieves performance comparable to standard residual networks, suggesting its broad utility.