Federated Calibration and Evaluation of Binary Classifiers
This addresses practical obstacles for using supervised classifiers in federated learning settings, enabling calibration and evaluation without compromising privacy, though it is incremental in extending existing methods to new privacy models.
The paper tackles the problem of calibrating and evaluating binary classifiers on distributed private data without centralizing labels, showing how to perform calibration and compute metrics like precision, recall, accuracy, and ROC-AUC under three privacy models, with experiments clarifying trade-offs between privacy, accuracy, and data efficiency.
We address two major obstacles to practical use of supervised classifiers on distributed private data. Whether a classifier was trained by a federation of cooperating clients or trained centrally out of distribution, (1) the output scores must be calibrated, and (2) performance metrics must be evaluated -- all without assembling labels in one place. In particular, we show how to perform calibration and compute precision, recall, accuracy and ROC-AUC in the federated setting under three privacy models (i) secure aggregation, (ii) distributed differential privacy, (iii) local differential privacy. Our theorems and experiments clarify tradeoffs between privacy, accuracy, and data efficiency. They also help decide whether a given application has sufficient data to support federated calibration and evaluation.