Exploring the Benefits of Differentially Private Pre-training and Parameter-Efficient Fine-tuning for Table Transformers
This work addresses privacy-preserving machine learning for tabular data, offering incremental improvements in parameter efficiency and accuracy for domain-specific applications.
The paper tackles the problem of balancing privacy, accuracy, and efficiency in transfer learning for tabular data by exploring differentially private pre-training and parameter-efficient fine-tuning methods for Table Transformers, finding that methods like Adapter, LoRA, and Prompt Tuning outperform traditional approaches in downstream task accuracy and parameter efficiency on the ACSIncome dataset.
For machine learning with tabular data, Table Transformer (TabTransformer) is a state-of-the-art neural network model, while Differential Privacy (DP) is an essential component to ensure data privacy. In this paper, we explore the benefits of combining these two aspects together in the scenario of transfer learning -- differentially private pre-training and fine-tuning of TabTransformers with a variety of parameter-efficient fine-tuning (PEFT) methods, including Adapter, LoRA, and Prompt Tuning. Our extensive experiments on the ACSIncome dataset show that these PEFT methods outperform traditional approaches in terms of the accuracy of the downstream task and the number of trainable parameters, thus achieving an improved trade-off among parameter efficiency, privacy, and accuracy. Our code is available at github.com/IBM/DP-TabTransformer.