Improved Bounds for Private and Robust Alignment
This work addresses the challenge of ensuring privacy and robustness in language model alignment, which is crucial for deploying AI systems in sensitive or adversarial environments, though it is primarily theoretical and incremental in nature.
The paper tackles the problem of aligning language models under privacy constraints and adversarial corruption by establishing theoretical upper bounds on suboptimality gaps in offline and online settings. It shows that log loss with an MLE-style algorithm achieves near-optimal rates for privacy-only cases and improves bounds for joint privacy-and-corruption settings, including the first results for online alignment.
In this paper, we study the private and robust alignment of language models from a theoretical perspective by establishing upper bounds on the suboptimality gap in both offline and online settings. We consider preference labels subject to privacy constraints and/or adversarial corruption, and analyze two distinct interplays between them: privacy-first and corruption-first. For the privacy-only setting, we show that log loss with an MLE-style algorithm achieves near-optimal rates, in contrast to conventional wisdom. For the joint privacy-and-corruption setting, we first demonstrate that existing offline algorithms in fact provide stronger guarantees -- simultaneously in terms of corruption level and privacy parameters -- than previously known, which further yields improved bounds in the corruption-only regime. In addition, we also present the first set of results for private and robust online alignment. Our results are enabled by new uniform convergence guarantees for log loss and square loss under privacy and corruption, which we believe have broad applicability across learning theory and statistics.