Fast, Distribution-free Predictive Inference for Neural Networks with Coverage Guarantees
This provides a faster, distribution-free method for predictive inference in neural networks, addressing a computational bottleneck for researchers and practitioners in machine learning.
The paper tackles the computational inefficiency of bootstrap methods for predictive inference in neural networks by introducing a differentially private algorithm that trains one model and approximates leave-one-out models, achieving rigorous coverage guarantees with reduced computation, as demonstrated in simulations and real data experiments.
This paper introduces a novel, computationally-efficient algorithm for predictive inference (PI) that requires no distributional assumptions on the data and can be computed faster than existing bootstrap-type methods for neural networks. Specifically, if there are $n$ training samples, bootstrap methods require training a model on each of the $n$ subsamples of size $n-1$; for large models like neural networks, this process can be computationally prohibitive. In contrast, our proposed method trains one neural network on the full dataset with $(ε, δ)$-differential privacy (DP) and then approximates each leave-one-out model efficiently using a linear approximation around the differentially-private neural network estimate. With exchangeable data, we prove that our approach has a rigorous coverage guarantee that depends on the preset privacy parameters and the stability of the neural network, regardless of the data distribution. Simulations and experiments on real data demonstrate that our method satisfies the coverage guarantees with substantially reduced computation compared to bootstrap methods.