When Gradient Clipping Becomes a Control Mechanism for Differential Privacy in Deep Learning
This work addresses a critical parameter-tuning challenge for practitioners implementing differential privacy in deep learning, though it represents an incremental improvement over existing adaptive clipping methods.
The paper tackles the problem of setting gradient clipping thresholds in differentially private deep learning, where improper thresholds cause optimization bias or noise-dominated updates. The authors propose a control-driven clipping strategy that uses weight-only spectral diagnostics to adapt thresholds, achieving competitive accuracy with existing methods while reducing computational overhead.
Privacy-preserving training on sensitive data commonly relies on differentially private stochastic optimization with gradient clipping and Gaussian noise. The clipping threshold is a critical control knob: if set too small, systematic over-clipping induces optimization bias; if too large, injected noise dominates updates and degrades accuracy. Existing adaptive clipping methods often depend on per-example gradient norm statistics, adding computational overhead and introducing sensitivity to datasets and architectures. We propose a control-driven clipping strategy that adapts the threshold using a lightweight, weight-only spectral diagnostic computed from model parameters. At periodic probe steps, the method analyzes a designated weight matrix via spectral decomposition and estimates a heavy-tailed spectral indicator associated with training stability. This indicator is smoothed over time and fed into a bounded feedback controller that updates the clipping threshold multiplicatively in the log domain. Because the controller uses only parameters produced during privacy-preserving training, the resulting threshold updates are post-processing and do not increase privacy loss beyond that of the underlying DP optimizer under standard composition accounting.