Improved Rates for Differentially Private Stochastic Convex Optimization with Heavy-Tailed Data
This work addresses privacy-preserving machine learning for non-Lipschitz, heavy-tailed data, which is relevant for applications with noisy or outlier-prone datasets, though it builds incrementally on prior assumptions about gradient moments.
The paper tackles differentially private stochastic convex optimization with heavy-tailed data, providing improved upper bounds on excess population risk for convex and strongly convex loss functions under concentrated DP, along with nearly-matching lower bounds that reveal separations between pure and concentrated DP.
We study stochastic convex optimization with heavy-tailed data under the constraint of differential privacy (DP). Most prior work on this problem is restricted to the case where the loss function is Lipschitz. Instead, as introduced by Wang, Xiao, Devadas, and Xu \cite{WangXDX20}, we study general convex loss functions with the assumption that the distribution of gradients has bounded $k$-th moments. We provide improved upper bounds on the excess population risk under concentrated DP for convex and strongly convex loss functions. Along the way, we derive new algorithms for private mean estimation of heavy-tailed distributions, under both pure and concentrated DP. Finally, we prove nearly-matching lower bounds for private stochastic convex optimization with strongly convex losses and mean estimation, showing new separations between pure and concentrated DP.