Positional Embedding-Aware Activations
This addresses the spectral bias issue in neural networks for researchers and practitioners in fields like image and audio representation, offering a significant improvement over existing methods.
The authors tackled the problem of spectral bias in neural networks by introducing SPDER, a simple MLP architecture with a novel activation function that learns positional embeddings automatically, resulting in 10x faster training and losses 1,500-50,000x lower than state-of-the-art for image representation.
We present a neural network architecture designed to naturally learn a positional embedding and overcome the spectral bias towards lower frequencies faced by conventional activation functions. Our proposed architecture, SPDER, is a simple MLP that uses an activation function composed of a sinusoidal multiplied by a sublinear function, called the damping function. The sinusoidal enables the network to automatically learn the positional embedding of an input coordinate while the damping passes on the actual coordinate value by preventing it from being projected down to within a finite range of values. Our results indicate that SPDERs speed up training by 10x and converge to losses 1,500-50,000x lower than that of the state-of-the-art for image representation. SPDER is also state-of-the-art in audio representation. The superior representation capability allows SPDER to also excel on multiple downstream tasks such as image super-resolution and video frame interpolation. We provide intuition as to why SPDER significantly improves fitting compared to that of other INR methods while requiring no hyperparameter tuning or preprocessing.