Nonparametric Variational Differential Privacy via Embedding Parameter Clipping
This incremental improvement addresses the privacy-utility trade-off in variational models, making them more robust and practical for privacy-preserving language models.
The paper tackled the issue of poor privacy guarantees and low utility in nonparametric variational differential privacy models by introducing a parameter clipping strategy, which consistently achieved tighter Rényi Divergence bounds for stronger privacy and higher performance on downstream tasks.
The nonparametric variational information bottleneck (NVIB) provides the foundation for nonparametric variational differential privacy (NVDP), a framework for building privacy-preserving language models. However, the learned latent representations can drift into regions with high information content, leading to poor privacy guarantees, but also low utility due to numerical instability during training. In this work, we introduce a principled parameter clipping strategy to directly address this issue. Our method is mathematically derived from the objective of minimizing the Rényi Divergence (RD) upper bound, yielding specific, theoretically grounded constraints on the posterior mean, variance, and mixture weight parameters. We apply our technique to an NVIB based model and empirically compare it against an unconstrained baseline. Our findings demonstrate that the clipped model consistently achieves tighter RD bounds, implying stronger privacy, while simultaneously attaining higher performance on several downstream tasks. This work presents a simple yet effective method for improving the privacy-utility trade-off in variational models, making them more robust and practical.