Robust Neural Processes for Noisy Data
This addresses the robustness issue for in-context learning models, which is incremental as it adapts an existing framework to noisy conditions.
The paper tackles the problem of in-context learning models being sensitive to noisy data, finding that attention-based models are particularly prone to in-context overfitting. It proposes a simple training method for Neural Processes that outperforms other models across all noise levels on 1D functions and 2D image datasets.
Models that adapt their predictions based on some given contexts, also known as in-context learning, have become ubiquitous in recent years. We propose to study the behavior of such models when data is contaminated by noise. Towards this goal we use the Neural Processes (NP) framework, as a simple and rigorous way to learn a distribution over functions, where predictions are based on a set of context points. Using this framework, we find that the models that perform best on clean data, are different than the models that perform best on noisy data. Specifically, models that process the context using attention, are more severely affected by noise, leading to in-context overfitting. We propose a simple method to train NP models that makes them more robust to noisy data. Experiments on 1D functions and 2D image datasets demonstrate that our method leads to models that outperform all other NP models for all noise levels.