DP-KFC: Data-Free Preconditioning for Privacy-Preserving Deep Learning
Enables privacy-preserving deep learning in data-scarce domains (e.g., medical) by resolving the geometric mismatch between anisotropic loss landscapes and isotropic DP noise without consuming privacy budget.
DP-KFC constructs KFAC preconditioners for DP-SGD using only synthetic noise, eliminating the need for private or public data. It consistently outperforms DP-SGD and adaptive baselines at ε ≤ 3, matching private-data preconditioners while public-data variants degrade by up to 4.8%.
Differentially private optimization suffers from a fundamental geometric mismatch: deep networks have highly anisotropic loss landscapes, yet DP-SGD injects isotropic noise. Second-order preconditioning can resolve this, but estimating curvature typically requires private data (consuming privacy budget) or public data (introducing distribution shift). We show that the Fisher Information Matrix decouples into architectural sensitivity, recoverable via synthetic noise, and input correlations, approximable from modality-specific frequency statistics. We propose DP-KFC, which constructs KFAC preconditioners by probing networks with structured synthetic noise, requiring neither private nor public data. Empirically, DP-KFC consistently outperforms DP-SGD and adaptive baselines across diverse modalities in strong privacy regimes ($\varepsilon \leq 3$). DP-KFC matches private-data preconditioners while public-data variants degrade by up to $4.8\%$, showing that curvature can be estimated without consuming privacy budget or introducing distribution shift. This enables privacy-preserving learning in specialized domains (e.g., medical applications) where regulatory constraints make data scarce.