Scaling Laws for Differentially Private Language Models
This addresses the need for privacy-preserving training in large language models using sensitive user data, providing guidance for efficient and private model scaling.
The paper tackles the problem of understanding scaling laws for differentially private language model training, establishing laws that model compute-privacy-utility tradeoffs and optimal configurations.
Scaling laws have emerged as important components of large language model (LLM) training as they can predict performance gains through scale, and provide guidance on important hyper-parameter choices that would otherwise be expensive. LLMs also rely on large, high-quality training datasets, like those sourced from (sometimes sensitive) user data. Training models on this sensitive user data requires careful privacy protections like differential privacy (DP). However, the dynamics of DP training are significantly different, and consequently their scaling laws are not yet fully understood. In this work, we establish scaling laws that accurately model the intricacies of DP LLM training, providing a complete picture of the compute-privacy-utility tradeoffs and the optimal training configurations in many settings.