Improving Accuracy and Efficiency of Implicit Neural Representations: Making SIREN a WINNER
This addresses a fundamental limitation in implicit neural representations for researchers and practitioners, though it is incremental as it builds on SIREN with a novel initialization strategy.
The paper tackled the problem of sinusoidal representation networks (SIRENs) struggling to fit signals outside their frequency support, leading to a 'spectral bottleneck' phenomenon. The result was WINNER, a weight initialization method using Gaussian noise scaled by the target signal's spectral centroid, which achieved state-of-the-art audio fitting and significant gains in image and 3D shape fitting over base SIREN.
We identify and address a fundamental limitation of sinusoidal representation networks (SIRENs), a class of implicit neural representations. SIRENs Sitzmann et al. (2020), when not initialized appropriately, can struggle at fitting signals that fall outside their frequency support. In extreme cases, when the network's frequency support misaligns with the target spectrum, a 'spectral bottleneck' phenomenon is observed, where the model yields to a near-zero output and fails to recover even the frequency components that are within its representational capacity. To overcome this, we propose WINNER - Weight Initialization with Noise for Neural Representations. WINNER perturbs uniformly initialized weights of base SIREN with Gaussian noise - whose noise scales are adaptively determined by the spectral centroid of the target signal. Similar to random Fourier embeddings, this mitigates 'spectral bias' but without introducing additional trainable parameters. Our method achieves state-of-the-art audio fitting and significant gains in image and 3D shape fitting tasks over base SIREN. Beyond signal fitting, WINNER suggests new avenues in adaptive, target-aware initialization strategies for optimizing deep neural network training. For code and data visit cfdlabtechnion.github.io/siren_square/.