An Efficient Plugin Method for Metric Optimization of Black-Box Models
This addresses the challenge for users who rely on black-box models via APIs and need to optimize specific metrics without retraining, though it is incremental as it builds on existing post-processing techniques.
The paper tackles the problem of adapting black-box model predictions to a target distribution and optimizing confusion matrix metrics without model access, proposing a post-hoc method called Plugin that achieves competitive performance on tabular and language tasks.
Many machine learning algorithms and classifiers are available only via API queries as a ``black-box'' -- that is, the downstream user has no ability to change, re-train, or fine-tune the model on a particular target distribution. Indeed, the downstream user may not even have knowledge of the \emph{original} training distribution or performance metric used to construct and optimize the black-box model. We propose a simple and efficient method, Plugin, which \emph{post-processes} arbitrary multiclass predictions from any black-box classifier in order to simultaneously (1) adapt these predictions to a target distribution; and (2) optimize a particular metric of the confusion matrix. Importantly, Plugin is a completely \textit{post-hoc} method which does not rely on feature information, only requires a small amount of probabilistic predictions along with their corresponding true label, and optimizes metrics by querying. We empirically demonstrate that Plugin is both broadly applicable and has performance competitive with related methods on a variety of tabular and language tasks.