Meta-Learning Sparse Implicit Neural Representations
This addresses the computational and memory inefficiency problem for researchers and practitioners using implicit neural representations, though it is incremental as it builds on existing meta-learning and sparsity techniques.
The paper tackles the scalability issue of implicit neural representations for large datasets by proposing a meta-learning approach with sparsity constraints, resulting in meta-learned sparse neural representations achieving a much smaller loss than dense models with the same number of parameters.
Implicit neural representations are a promising new avenue of representing general signals by learning a continuous function that, parameterized as a neural network, maps the domain of a signal to its codomain; the mapping from spatial coordinates of an image to its pixel values, for example. Being capable of conveying fine details in a high dimensional signal, unboundedly of its domain, implicit neural representations ensure many advantages over conventional discrete representations. However, the current approach is difficult to scale for a large number of signals or a data set, since learning a neural representation -- which is parameter heavy by itself -- for each signal individually requires a lot of memory and computations. To address this issue, we propose to leverage a meta-learning approach in combination with network compression under a sparsity constraint, such that it renders a well-initialized sparse parameterization that evolves quickly to represent a set of unseen signals in the subsequent training. We empirically demonstrate that meta-learned sparse neural representations achieve a much smaller loss than dense meta-learned models with the same number of parameters, when trained to fit each signal using the same number of optimization steps.