Contextual HyperNetworks for Novel Feature Adaptation
This addresses the problem of time and data-efficient adaptation to novel features for practitioners in dynamic domains like recommender systems, though it is incremental as it builds on existing hypernetwork and meta-learning concepts.
The paper tackles the challenge of adapting neural networks to incorporate new output features efficiently, particularly in online learning settings, by proposing Contextual HyperNetworks (CHNs) that generate parameters for new features using existing data and metadata, resulting in improved few-shot learning performance over baselines across recommender systems, e-learning, and healthcare tasks.
While deep learning has obtained state-of-the-art results in many applications, the adaptation of neural network architectures to incorporate new output features remains a challenge, as neural networks are commonly trained to produce a fixed output dimension. This issue is particularly severe in online learning settings, where new output features, such as items in a recommender system, are added continually with few or no associated observations. As such, methods for adapting neural networks to novel features which are both time and data-efficient are desired. To address this, we propose the Contextual HyperNetwork (CHN), an auxiliary model which generates parameters for extending the base model to a new feature, by utilizing both existing data as well as any observations and/or metadata associated with the new feature. At prediction time, the CHN requires only a single forward pass through a neural network, yielding a significant speed-up when compared to re-training and fine-tuning approaches. To assess the performance of CHNs, we use a CHN to augment a partial variational autoencoder (P-VAE), a deep generative model which can impute the values of missing features in sparsely-observed data. We show that this system obtains improved few-shot learning performance for novel features over existing imputation and meta-learning baselines across recommender systems, e-learning, and healthcare tasks.