Continual Release Moment Estimation with Differential Privacy
This work addresses privacy concerns in continual data analysis for machine learning applications, offering a method that reduces noise compared to naive approaches.
The paper tackles the problem of privately and continually estimating first and second moments of data with reduced noise, achieving improved accuracy while maintaining privacy, as demonstrated in applications like Gaussian density estimation and DP-Adam training on CIFAR-10.
We propose Joint Moment Estimation (JME), a method for continually and privately estimating both the first and second moments of data with reduced noise compared to naive approaches. JME uses the matrix mechanism and a joint sensitivity analysis to allow the second moment estimation with no additional privacy cost, thereby improving accuracy while maintaining privacy. We demonstrate JME's effectiveness in two applications: estimating the running mean and covariance matrix for Gaussian density estimation, and model training with DP-Adam on CIFAR-10.