Beyond Ordinary Lipschitz Constraints: Differentially Private Stochastic Optimization with Tsybakov Noise Condition
This work addresses privacy-preserving optimization for scenarios with heavy-tailed noise, offering theoretical guarantees that improve over standard Lipschitz assumptions, though it is incremental in extending existing DP-SCO frameworks.
The paper tackles stochastic convex optimization under differential privacy with Tsybakov noise conditions, where Lipschitz constants may be unbounded, by proposing algorithms that achieve utility bounds independent of the Lipschitz constant and providing matching lower bounds for concentrated differential privacy.
We study Stochastic Convex Optimization in the Differential Privacy model (DP-SCO). Unlike previous studies, here we assume the population risk function satisfies the Tsybakov Noise Condition (TNC) with some parameter $θ>1$, where the Lipschitz constant of the loss could be extremely large or even unbounded, but the $\ell_2$-norm gradient of the loss has bounded $k$-th moment with $k\geq 2$. For the Lipschitz case with $θ\geq 2$, we first propose an $(\varepsilon, δ)$-DP algorithm whose utility bound is $\Tilde{O}\left(\left(\tilde{r}_{2k}(\frac{1}{\sqrt{n}}+(\frac{\sqrt{d}}{n\varepsilon}))^\frac{k-1}{k}\right)^\fracθ{θ-1}\right)$ in high probability, where $n$ is the sample size, $d$ is the model dimension, and $\tilde{r}_{2k}$ is a term that only depends on the $2k$-th moment of the gradient. It is notable that such an upper bound is independent of the Lipschitz constant. We then extend to the case where $θ\geq \barθ> 1$ for some known constant $\barθ$. Moreover, when the privacy budget $\varepsilon$ is small enough, we show an upper bound of $\tilde{O}\left(\left(\tilde{r}_{k}(\frac{1}{\sqrt{n}}+(\frac{\sqrt{d}}{n\varepsilon}))^\frac{k-1}{k}\right)^\fracθ{θ-1}\right)$ even if the loss function is not Lipschitz. For the lower bound, we show that for any $θ\geq 2$, the private minimax rate for $ρ$-zero Concentrated Differential Privacy is lower bounded by $Ω\left(\left(\tilde{r}_{k}(\frac{1}{\sqrt{n}}+(\frac{\sqrt{d}}{n\sqrtρ}))^\frac{k-1}{k}\right)^\fracθ{θ-1}\right)$.