Privacy Amplification in Differentially Private Zeroth-Order Optimization with Hidden States
This work addresses a critical gap in privacy analysis for zeroth-order optimization, which is important for applications like fine-tuning LLMs under differential privacy, though it is incremental as it extends existing frameworks to a new method.
The paper tackles the problem of establishing differential privacy guarantees for zeroth-order optimization methods, which are used for fine-tuning large language models under privacy constraints, by proving a convergent DP bound and generalizing the privacy amplification-by-iteration framework to this setting.
Zeroth-order optimization has emerged as a promising approach for fine-tuning large language models on domain-specific data, particularly under differential privacy (DP) and memory constraints. While first-order methods have been extensively studied from a privacy perspective, the privacy analysis and algorithmic design for zeroth-order methods remain significantly underexplored. A critical open question concerns hidden-state DP analysis: although convergent privacy bounds are known for first-order methods, it has remained unclear whether similar guarantees can be established for zeroth-order methods. In this work, we provide an affirmative answer by proving a convergent DP bound for zeroth-order optimization. Our analysis generalizes the celebrated privacy amplification-by-iteration framework to the setting of smooth loss functions in zeroth-order optimization. Furthermore, it induces better DP zeroth-order algorithmic designs that are previously unknown to the literature.