Confidence Calibration for Systems with Cascaded Predictive Modules
This work addresses the challenge of reliable uncertainty quantification for autonomous systems with multiple modules, offering a practical solution for domains like robotics and navigation.
The paper tackles the problem of uncertainty propagation in cascaded predictive systems by introducing novel conformal prediction solutions that provide calibrated prediction intervals at the system level, demonstrating improved accuracy on both synthetic and real-world datasets like Matterport3D for indoor navigation.
Existing conformal prediction algorithms estimate prediction intervals at target confidence levels to characterize the performance of a regression model on new test samples. However, considering an autonomous system consisting of multiple modules, prediction intervals constructed for individual modules fall short of accommodating uncertainty propagation over different modules and thus cannot provide reliable predictions on system behavior. We address this limitation and present novel solutions based on conformal prediction to provide prediction intervals calibrated for a predictive system consisting of cascaded modules (e.g., an upstream feature extraction module and a downstream regression module). Our key idea is to leverage module-level validation data to characterize the system-level error distribution without direct access to end-to-end validation data. We provide theoretical justification and empirical experimental results to demonstrate the effectiveness of proposed solutions. In comparison to prediction intervals calibrated for individual modules, our solutions generate improved intervals with more accurate performance guarantees for system predictions, which are demonstrated on both synthetic systems and real-world systems performing overlap prediction for indoor navigation using the Matterport3D dataset.