PluralLLM: Pluralistic Alignment in LLMs via Federated Learning
This addresses the problem of privacy-invasive and computationally expensive alignment for LLM users by offering a scalable and privacy-preserving alternative, though it is incremental as it builds on existing federated learning techniques.
The paper tackles the challenge of aligning Large Language Models with diverse human preferences while preserving privacy and fairness by introducing PluralLLM, a federated learning-based approach that achieves 46% faster convergence and a 4% improvement in alignment scores compared to centralized methods.
Ensuring Large Language Models (LLMs) align with diverse human preferences while preserving privacy and fairness remains a challenge. Existing methods, such as Reinforcement Learning from Human Feedback (RLHF), rely on centralized data collection, making them computationally expensive and privacy-invasive. We introduce PluralLLM a federated learning-based approach that enables multiple user groups to collaboratively train a transformer-based preference predictor without sharing sensitive data, which can also serve as a reward model for aligning LLMs. Our method leverages Federated Averaging (FedAvg) to aggregate preference updates efficiently, achieving 46% faster convergence, a 4% improvement in alignment scores, and nearly the same group fairness measure as in centralized training. Evaluated on a Q/A preference alignment task, PluralLLM demonstrates that federated preference learning offers a scalable and privacy-preserving alternative for aligning LLMs with diverse human values.