Learning High-Frequency Functions Made Easy with Sinusoidal Positional Encoding
This addresses a bottleneck in machine learning tasks requiring high-frequency feature learning, offering a plug-and-play solution for improved efficiency and performance.
The paper tackles the problem of learning high-frequency functions from low-dimensional inputs by introducing sinusoidal positional encoding (SPE), which eliminates the need for manual hyperparameter tuning and achieves enhanced fidelity and faster training across tasks like 3D view synthesis and text-to-speech generation.
Fourier features based positional encoding (PE) is commonly used in machine learning tasks that involve learning high-frequency features from low-dimensional inputs, such as 3D view synthesis and time series regression with neural tangent kernels. Despite their effectiveness, existing PEs require manual, empirical adjustment of crucial hyperparameters, specifically the Fourier features, tailored to each unique task. Further, PEs face challenges in efficiently learning high-frequency functions, particularly in tasks with limited data. In this paper, we introduce sinusoidal PE (SPE), designed to efficiently learn adaptive frequency features closely aligned with the true underlying function. Our experiments demonstrate that SPE, without hyperparameter tuning, consistently achieves enhanced fidelity and faster training across various tasks, including 3D view synthesis, Text-to-Speech generation, and 1D regression. SPE is implemented as a direct replacement for existing PEs. Its plug-and-play nature lets numerous tasks easily adopt and benefit from SPE.