L3 Ensembles: Lifelong Learning Approach for Ensemble of Foundational Language Models
This addresses efficient adaptation for NLP tasks in resource-limited settings, though it appears incremental as it builds on existing lifelong learning and ensemble methods.
The paper tackles the problem of fine-tuning foundational language models for NLP tasks on resource-constrained devices by proposing a lifelong learning ensemble approach, resulting in accuracy improvements of 4% to 36% over fine-tuned models and up to 15.4% over state-of-the-art models on benchmarks.
Fine-tuning pre-trained foundational language models (FLM) for specific tasks is often impractical, especially for resource-constrained devices. This necessitates the development of a Lifelong Learning (L3) framework that continuously adapts to a stream of Natural Language Processing (NLP) tasks efficiently. We propose an approach that focuses on extracting meaningful representations from unseen data, constructing a structured knowledge base, and improving task performance incrementally. We conducted experiments on various NLP tasks to validate its effectiveness, including benchmarks like GLUE and SuperGLUE. We measured good performance across the accuracy, training efficiency, and knowledge transfer metrics. Initial experimental results show that the proposed L3 ensemble method increases the model accuracy by 4% ~ 36% compared to the fine-tuned FLM. Furthermore, L3 model outperforms naive fine-tuning approaches while maintaining competitive or superior performance (up to 15.4% increase in accuracy) compared to the state-of-the-art language model (T5) for the given task, STS benchmark.