Transformers as Meta-Learners for Implicit Neural Representations
This addresses a bottleneck in INR optimization for researchers and practitioners in computer vision and graphics, offering a more efficient and precise method, though it builds on prior gradient-based meta-learning work.
The paper tackles the inefficiency and poor generalization of fitting Implicit Neural Representations (INRs) from sparse observations by proposing Transformers as hypernetworks to directly generate full INR weights, achieving improved reconstruction precision in tasks like 2D image regression and 3D view synthesis.
Implicit Neural Representations (INRs) have emerged and shown their benefits over discrete representations in recent years. However, fitting an INR to the given observations usually requires optimization with gradient descent from scratch, which is inefficient and does not generalize well with sparse observations. To address this problem, most of the prior works train a hypernetwork that generates a single vector to modulate the INR weights, where the single vector becomes an information bottleneck that limits the reconstruction precision of the output INR. Recent work shows that the whole set of weights in INR can be precisely inferred without the single-vector bottleneck by gradient-based meta-learning. Motivated by a generalized formulation of gradient-based meta-learning, we propose a formulation that uses Transformers as hypernetworks for INRs, where it can directly build the whole set of INR weights with Transformers specialized as set-to-set mapping. We demonstrate the effectiveness of our method for building INRs in different tasks and domains, including 2D image regression and view synthesis for 3D objects. Our work draws connections between the Transformer hypernetworks and gradient-based meta-learning algorithms and we provide further analysis for understanding the generated INRs.