Towards Practical Few-shot Federated NLP
This addresses the challenge of federated few-shot learning for NLP, enabling practical applications with privacy constraints and data scarcity, though it is incremental in combining existing techniques.
The paper tackles the problem of fine-tuning transformer-based pre-trained models for NLP tasks with limited labeled data distributed across heterogeneous devices, proposing AUG-FedPrompt, a prompt-based federated learning system that uses unlabeled data for augmentation, achieving performance comparable to full-set fine-tuning with scarce labeled data.
Transformer-based pre-trained models have emerged as the predominant solution for natural language processing (NLP). Fine-tuning such pre-trained models for downstream tasks often requires a considerable amount of labeled private data. In practice, private data is often distributed across heterogeneous mobile devices and may be prohibited from being uploaded. Moreover, well-curated labeled data is often scarce, presenting an additional challenge. To address these challenges, we first introduce a data generator for federated few-shot learning tasks, which encompasses the quantity and skewness of scarce labeled data in a realistic setting. Subsequently, we propose AUG-FedPrompt, a prompt-based federated learning system that exploits abundant unlabeled data for data augmentation. Our experiments indicate that AUG-FedPrompt can perform on par with full-set fine-tuning with a limited amount of labeled data. However, such competitive performance comes at a significant system cost.