Distributed Conformal Prediction via Message Passing
This work addresses the need for reliable calibration in safety-critical domains like healthcare, but it is incremental as it extends existing conformal prediction methods to decentralized scenarios.
The paper tackles the problem of post-hoc calibration for pre-trained models in decentralized settings with limited data and communication, proposing two message-passing approaches (Q-DCP and H-DCP) that achieve reliable inference with distribution-free coverage guarantees, as demonstrated through experiments on trade-offs like hyperparameter tuning and communication overhead.
Post-hoc calibration of pre-trained models is critical for ensuring reliable inference, especially in safety-critical domains such as healthcare. Conformal Prediction (CP) offers a robust post-hoc calibration framework, providing distribution-free statistical coverage guarantees for prediction sets by leveraging held-out datasets. In this work, we address a decentralized setting where each device has limited calibration data and can communicate only with its neighbors over an arbitrary graph topology. We propose two message-passing-based approaches for achieving reliable inference via CP: quantile-based distributed conformal prediction (Q-DCP) and histogram-based distributed conformal prediction (H-DCP). Q-DCP employs distributed quantile regression enhanced with tailored smoothing and regularization terms to accelerate convergence, while H-DCP uses a consensus-based histogram estimation approach. Through extensive experiments, we investigate the trade-offs between hyperparameter tuning requirements, communication overhead, coverage guarantees, and prediction set sizes across different network topologies. The code of our work is released on: https://github.com/HaifengWen/Distributed-Conformal-Prediction.