In-Context In-Context Learning with Transformer Neural Processes
This addresses a limitation in meta-learning models for practitioners who have access to multiple related datasets, but it appears incremental as it builds on existing transformer neural processes.
The paper tackles the problem of neural processes (NPs) being unable to condition on multiple datasets simultaneously, which limits their ability to integrate similar datasets for improved predictions. They introduce the ICICL-TNP model, which enables in-context in-context learning and demonstrates effectiveness in experiments, though no concrete numbers are provided in the abstract.
Neural processes (NPs) are a powerful family of meta-learning models that seek to approximate the posterior predictive map of the ground-truth stochastic process from which each dataset in a meta-dataset is sampled. There are many cases in which practitioners, besides having access to the dataset of interest, may also have access to other datasets that share similarities with it. In this case, integrating these datasets into the NP can improve predictions. We equip NPs with this functionality and describe this paradigm as in-context in-context learning. Standard NP architectures, such as the convolutional conditional NP (ConvCNP) or the family of transformer neural processes (TNPs), are not capable of in-context in-context learning, as they are only able to condition on a single dataset. We address this shortcoming by developing the in-context in-context learning pseudo-token TNP (ICICL-TNP). The ICICL-TNP builds on the family of PT-TNPs, which utilise pseudo-token-based transformer architectures to sidestep the quadratic computational complexity associated with regular transformer architectures. Importantly, the ICICL-TNP is capable of conditioning on both sets of datapoints and sets of datasets, enabling it to perform in-context in-context learning. We demonstrate the importance of in-context in-context learning and the effectiveness of the ICICL-TNP in a number of experiments.