Mapping Networks
This addresses the problem of efficient training and overfitting for researchers and practitioners using large deep learning models, representing a novel method rather than an incremental improvement.
The paper tackles the challenge of escalating parameter counts in deep learning models by introducing Mapping Networks, which replace high-dimensional weight spaces with compact trainable latent vectors, achieving comparable or better performance across vision and sequence tasks with a 500× reduction in trainable parameters (99.5% reduction).
The escalating parameter counts in modern deep learning models pose a fundamental challenge to efficient training and resolution of overfitting. We address this by introducing the \emph{Mapping Networks} which replace the high dimensional weight space by a compact, trainable latent vector based on the hypothesis that the trained parameters of large networks reside on smooth, low-dimensional manifolds. Henceforth, the Mapping Theorem enforced by a dedicated Mapping Loss, shows the existence of a mapping from this latent space to the target weight space both theoretically and in practice. Mapping Networks significantly reduce overfitting and achieve comparable to better performance than target network across complex vision and sequence tasks, including Image Classification, Deepfake Detection etc, with $\mathbf{99.5\%}$, i.e., around $500\times$ reduction in trainable parameters.