Unsupervised Risk Estimation Using Only Conditional Independence Structure
This addresses the challenge of model evaluation in unsupervised settings with distribution shifts, which is incremental as it builds on conditional independence assumptions rather than introducing a new paradigm.
The paper tackles the problem of estimating a model's test error from unlabeled data under distribution shifts, assuming only that certain conditional independencies are preserved, and shows that this can be done without requiring the optimal predictor to be the same or the distribution to be parametric, with extensions to various losses and structured outputs.
We show how to estimate a model's test error from unlabeled data, on distributions very different from the training distribution, while assuming only that certain conditional independencies are preserved between train and test. We do not need to assume that the optimal predictor is the same between train and test, or that the true distribution lies in any parametric family. We can also efficiently differentiate the error estimate to perform unsupervised discriminative learning. Our technical tool is the method of moments, which allows us to exploit conditional independencies in the absence of a fully-specified model. Our framework encompasses a large family of losses including the log and exponential loss, and extends to structured output settings such as hidden Markov models.