Understanding NTK Variance in Implicit Neural Representations
This work addresses spectral bias in INRs, a domain-specific issue for researchers in neural representation learning, by providing a unified theoretical framework that explains existing methods, making it incremental.
The paper tackled the problem of slow convergence and poor high-frequency detail recovery in Implicit Neural Representations (INRs) by analyzing how architectural choices affect Neural Tangent Kernel (NTK) conditioning, showing that mechanisms like positional encoding and Hadamard modulation reduce NTK eigenvalue variance, leading to faster convergence and improved reconstruction quality in experiments.
Implicit Neural Representations (INRs) often converge slowly and struggle to recover high-frequency details due to spectral bias. While prior work links this behavior to the Neural Tangent Kernel (NTK), how specific architectural choices affect NTK conditioning remains unclear. We show that many INR mechanisms can be understood through their impact on a small set of pairwise similarity factors and scaling terms that jointly determine NTK eigenvalue variance. For standard coordinate MLPs, limited input-feature interactions induce large eigenvalue dispersion and poor conditioning. We derive closed-form variance decompositions for common INR components and show that positional encoding reshapes input similarity, spherical normalization reduces variance via layerwise scaling, and Hadamard modulation introduces additional similarity factors strictly below one, yielding multiplicative variance reduction. This unified view explains how diverse INR architectures mitigate spectral bias by improving NTK conditioning. Experiments across multiple tasks confirm the predicted variance reductions and demonstrate faster, more stable convergence with improved reconstruction quality.