Towards the Spectral bias Alleviation by Normalizations in Coordinate Networks
This addresses a fundamental limitation in coordinate networks used for inverse problems in scientific computing, though it appears incremental as it adapts existing normalization methods to a new context.
The paper tackles the spectral bias problem in coordinate networks that limits their ability to learn high-frequency components, showing that applying classical normalization techniques (batch and layer normalization) improves the neural tangent kernel's eigenvalue distribution and achieves significant improvements and new state-of-the-art results across six tasks including image compression and novel view synthesis.
Representing signals using coordinate networks dominates the area of inverse problems recently, and is widely applied in various scientific computing tasks. Still, there exists an issue of spectral bias in coordinate networks, limiting the capacity to learn high-frequency components. This problem is caused by the pathological distribution of the neural tangent kernel's (NTK's) eigenvalues of coordinate networks. We find that, this pathological distribution could be improved using classical normalization techniques (batch normalization and layer normalization), which are commonly used in convolutional neural networks but rarely used in coordinate networks. We prove that normalization techniques greatly reduces the maximum and variance of NTK's eigenvalues while slightly modifies the mean value, considering the max eigenvalue is much larger than the most, this variance change results in a shift of eigenvalues' distribution from a lower one to a higher one, therefore the spectral bias could be alleviated. Furthermore, we propose two new normalization techniques by combining these two techniques in different ways. The efficacy of these normalization techniques is substantiated by the significant improvements and new state-of-the-arts achieved by applying normalization-based coordinate networks to various tasks, including the image compression, computed tomography reconstruction, shape representation, magnetic resonance imaging, novel view synthesis and multi-view stereo reconstruction.