Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER
This addresses the need for adaptable and efficient NLP models for mobile applications, representing an incremental advance in pre-finetuning methods.
The paper tackled the problem of deploying efficient NLP models on mobile platforms by proposing a multi-task pre-finetuning framework with task-primary LoRA modules to avoid conflicting optimization signals, achieving average improvements of +0.8% for NER and +8.8% for text classification across 21 downstream tasks.
Deploying natural language processing (NLP) models on mobile platforms requires models that can adapt across diverse applications while remaining efficient in memory and computation. We investigate pre-finetuning strategies to enhance the adaptability of lightweight BERT-like encoders for two fundamental NLP task families: named entity recognition (NER) and text classification. While pre-finetuning improves downstream performance for each task family individually, we find that naïve multi-task pre-finetuning introduces conflicting optimization signals that degrade overall performance. To address this, we propose a simple yet effective multi-task pre-finetuning framework based on task-primary LoRA modules, which enables a single shared encoder backbone with modular adapters. Our approach achieves performance comparable to individual pre-finetuning while meeting practical deployment constraint. Experiments on 21 downstream tasks show average improvements of +0.8% for NER and +8.8% for text classification, demonstrating the effectiveness of our method for versatile mobile NLP applications.