Adversarial Monte Carlo Meta-Learning of Optimal Prediction Procedures
This work addresses the challenge of developing robust prediction methods in machine learning, though it appears incremental by building on existing meta-learning and game theory concepts.
The paper tackles the problem of meta-learning prediction procedures by framing it as a two-player game between Nature and a Predictor, establishing an optimal strategy with specific invariance properties and introducing a neural network architecture that performs favorably in experiments.
We frame the meta-learning of prediction procedures as a search for an optimal strategy in a two-player game. In this game, Nature selects a prior over distributions that generate labeled data consisting of features and an associated outcome, and the Predictor observes data sampled from a distribution drawn from this prior. The Predictor's objective is to learn a function that maps from a new feature to an estimate of the associated outcome. We establish that, under reasonable conditions, the Predictor has an optimal strategy that is equivariant to shifts and rescalings of the outcome and is invariant to permutations of the observations and to shifts, rescalings, and permutations of the features. We introduce a neural network architecture that satisfies these properties. The proposed strategy performs favorably compared to standard practice in both parametric and nonparametric experiments.