Linear-Time User-Level DP-SCO via Robust Statistics
This work addresses privacy concerns in large-scale machine learning by providing a more efficient method for user-level DP-SCO, though it appears incremental as it builds on existing DP-SGD approaches with novel robust statistical techniques.
The paper tackles the problem of high noise accumulation and suboptimal utility in user-level differentially private stochastic convex optimization (DP-SCO) by introducing a linear-time algorithm that uses robust statistics like the median and trimmed mean to bound sensitivity and reduce gradient estimation noise, achieving an improved privacy-utility trade-off with optimal theoretical bounds up to logarithmic factors.
User-level differentially private stochastic convex optimization (DP-SCO) has garnered significant attention due to the paramount importance of safeguarding user privacy in modern large-scale machine learning applications. Current methods, such as those based on differentially private stochastic gradient descent (DP-SGD), often struggle with high noise accumulation and suboptimal utility due to the need to privatize every intermediate iterate. In this work, we introduce a novel linear-time algorithm that leverages robust statistics, specifically the median and trimmed mean, to overcome these challenges. Our approach uniquely bounds the sensitivity of all intermediate iterates of SGD with gradient estimation based on robust statistics, thereby significantly reducing the gradient estimation noise for privacy purposes and enhancing the privacy-utility trade-off. By sidestepping the repeated privatization required by previous methods, our algorithm not only achieves an improved theoretical privacy-utility trade-off but also maintains computational efficiency. We complement our algorithm with an information-theoretic lower bound, showing that our upper bound is optimal up to logarithmic factors and the dependence on $ε$. This work sets the stage for more robust and efficient privacy-preserving techniques in machine learning, with implications for future research and application in the field.