Private Learning and Sanitization: Pure vs. Approximate Differential Privacy
This work addresses privacy-preserving machine learning by quantifying efficiency gains from approximate differential privacy, which is incremental but provides concrete bounds for practitioners.
The paper compares sample complexity for private learning and sanitization under pure versus approximate differential privacy, showing that approximate privacy can significantly reduce sample requirements, with specific results for functions like point functions and threshold functions.
We compare the sample complexity of private learning [Kasiviswanathan et al. 2008] and sanitization~[Blum et al. 2008] under pure $ε$-differential privacy [Dwork et al. TCC 2006] and approximate $(ε,δ)$-differential privacy [Dwork et al. Eurocrypt 2006]. We show that the sample complexity of these tasks under approximate differential privacy can be significantly lower than that under pure differential privacy. We define a family of optimization problems, which we call Quasi-Concave Promise Problems, that generalizes some of our considered tasks. We observe that a quasi-concave promise problem can be privately approximated using a solution to a smaller instance of a quasi-concave promise problem. This allows us to construct an efficient recursive algorithm solving such problems privately. Specifically, we construct private learners for point functions, threshold functions, and axis-aligned rectangles in high dimension. Similarly, we construct sanitizers for point functions and threshold functions. We also examine the sample complexity of label-private learners, a relaxation of private learning where the learner is required to only protect the privacy of the labels in the sample. We show that the VC dimension completely characterizes the sample complexity of such learners, that is, the sample complexity of learning with label privacy is equal (up to constants) to learning without privacy.