LoRA and Privacy: When Random Projections Help (and When They Don't)
This work addresses privacy risks in machine learning fine-tuning, particularly for methods like LoRA, revealing vulnerabilities and offering solutions for practitioners in privacy-sensitive domains, though it is incremental in building on existing differential privacy frameworks.
The paper introduces the Wishart projection mechanism for differential privacy, showing that random projections alone can provide privacy for vector-valued queries without additive noise, but fail for matrix-valued queries, where they are vulnerable to membership inference attacks (AUC > 0.99). It also finds that low-rank fine-tuning like LoRA can be more private than full fine-tuning at the same noise level, with preliminary experiments indicating potential for lower noise and improved accuracy.
We introduce the (Wishart) projection mechanism, a randomized map of the form $S \mapsto M f(S)$ with $M \sim W_d(1/r I_d, r)$ and study its differential privacy properties. For vector-valued queries $f$, we prove non-asymptotic DP guarantees without any additive noise, showing that Wishart randomness alone can suffice. For matrix-valued queries, however, we establish a sharp negative result: in the noise-free setting, the mechanism is not DP, and we demonstrate its vulnerability by implementing a near perfect membership inference attack (AUC $> 0.99$). We then analyze a noisy variant and prove privacy amplification due to randomness and low rank projection, in both large- and small-rank regimes, yielding stronger privacy guarantees than additive noise alone. Finally, we show that LoRA-style updates are an instance of the matrix-valued mechanism, implying that LoRA is not inherently private despite its built-in randomness, but that low-rank fine-tuning can be more private than full fine-tuning at the same noise level. Preliminary experiments suggest that tighter accounting enables lower noise and improved accuracy in practice.