Empirical Risk Minimization in the Non-interactive Local Model of Differential Privacy
This work addresses privacy-preserving machine learning for data analysts by providing more efficient algorithms, though it is incremental as it builds on prior research to refine conditions and methods.
The paper tackles the problem of high sample complexity in Empirical Risk Minimization under non-interactive Local Differential Privacy by identifying conditions on loss functions, such as smoothness or Lipschitz properties, that allow for algorithms with reduced dependency on dimensionality, achieving linear or improved error bounds.
In this paper, we study the Empirical Risk Minimization (ERM) problem in the non-interactive Local Differential Privacy (LDP) model. Previous research on this problem \citep{smith2017interaction} indicates that the sample complexity, to achieve error $α$, needs to be exponentially depending on the dimensionality $p$ for general loss functions. In this paper, we make two attempts to resolve this issue by investigating conditions on the loss functions that allow us to remove such a limit. In our first attempt, we show that if the loss function is $(\infty, T)$-smooth, by using the Bernstein polynomial approximation we can avoid the exponential dependency in the term of $α$. We then propose player-efficient algorithms with $1$-bit communication complexity and $O(1)$ computation cost for each player. The error bound of these algorithms is asymptotically the same as the original one. With some additional assumptions, we also give an algorithm which is more efficient for the server. In our second attempt, we show that for any $1$-Lipschitz generalized linear convex loss function, there is an $(ε, δ)$-LDP algorithm whose sample complexity for achieving error $α$ is only linear in the dimensionality $p$. Our results use a polynomial of inner product approximation technique. Finally, motivated by the idea of using polynomial approximation and based on different types of polynomial approximations, we propose (efficient) non-interactive locally differentially private algorithms for learning the set of k-way marginal queries and the set of smooth queries.