Learning Positional Embeddings for Coordinate-MLPs
This addresses a specific bottleneck in coordinate-MLPs for researchers in neural representation fields, but it is incremental as it builds on existing embedding methods.
The paper tackles the problem of poor generalization in coordinate-MLPs by proposing a novel method to learn instance-specific positional embeddings using graph-Laplacian regularization, achieving better performance and stability compared to random Fourier features.
We propose a novel method to enhance the performance of coordinate-MLPs by learning instance-specific positional embeddings. End-to-end optimization of positional embedding parameters along with network weights leads to poor generalization performance. Instead, we develop a generic framework to learn the positional embedding based on the classic graph-Laplacian regularization, which can implicitly balance the trade-off between memorization and generalization. This framework is then used to propose a novel positional embedding scheme, where the hyperparameters are learned per coordinate (i.e, instance) to deliver optimal performance. We show that the proposed embedding achieves better performance with higher stability compared to the well-established random Fourier features (RFF). Further, we demonstrate that the proposed embedding scheme yields stable gradients, enabling seamless integration into deep architectures as intermediate layers.