Online Active Model Selection for Pre-trained Classifiers
This addresses the practical need for efficient model selection in online prediction tasks, but it is incremental as it builds on existing active learning and model selection frameworks.
The paper tackles the problem of selecting the best pre-trained classifier from a set of k models using an online stream of unlabeled data, by actively querying labels to minimize the number of queries while ensuring high-probability identification of the best model. The result is an algorithm with theoretical guarantees and demonstrated effectiveness in experiments.
Given $k$ pre-trained classifiers and a stream of unlabeled data examples, how can we actively decide when to query a label so that we can distinguish the best model from the rest while making a small number of queries? Answering this question has a profound impact on a range of practical scenarios. In this work, we design an online selective sampling approach that actively selects informative examples to label and outputs the best model with high probability at any round. Our algorithm can be used for online prediction tasks for both adversarial and stochastic streams. We establish several theoretical guarantees for our algorithm and extensively demonstrate its effectiveness in our experimental studies.