SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPC
This work addresses privacy constraints in specialized domains like healthcare and finance, offering an incremental improvement over existing MPC-based methods by extending them to efficient tuning.
The paper tackles the challenge of adapting large language models to privacy-sensitive domains by proposing SecP-Tuning, an efficient privacy-preserving prompt tuning framework using secure multi-party computation, which achieves up to 16 times acceleration and 20 times reduction in communication overhead while outperforming baseline methods in performance scores.
Large Language Models (LLMs) have revolutionized numerous fields, yet their adaptation to specialized tasks in privacy-sensitive domains such as healthcare and finance remains constrained due to the scarcity of accessible training data caused by stringent privacy requirements. Secure Multi-party Computation (MPC)-based privacy-preserving machine learning provides theoretical guarantees for the privacy of model parameters and data. However, its application to LLMs has been predominantly limited to inference, as fine-tuning introduces significant efficiency challenges, particularly in backward propagation, optimizer, and self-attention operations. To address these challenges, we propose SecP-Tuning, the first MPC-based framework designed for efficient, privacy-preserving prompt tuning of LLMs. SecP-Tuning innovatively integrates Forward-only Tuning (FoT) through the ``data owner-server interaction" paradigm, effectively removing the need for privacy-preserving computations in backward propagation and optimization processes. Furthermore, it devises an efficient privacy-preserving Random Feature Attention (RFA), effectively mitigating the computational complexity of softmax-based self-attention and circumventing MPC-incompatible nonlinear operations. Experimental results demonstrate that, compared to full-Parameter Supervised Fine-Tuning (SFT) and gradient-based prompt tuning, SecP-Tuning achieves approximately 12 times and 16 times end-to-end acceleration, as well as 18 times and 20 times reductions in communication overhead, respectively. Moreover, in five few-shot tasks, it achieves an average performance score of 82.45, outperforming SFT's 79.90 and prompt tuning's 73.73. Additionally, the ``black-box/API-style" privacy-preserving tuning paradigm of SecP-Tuning effectively avoids memory leakage risks caused by gradient/parameter transmission.