Data-Efficiency with a Single GPU: An Exploration of Transfer Methods for Small Language Models
This work addresses the challenge of making small language models more effective with limited data, which is important for researchers and practitioners with constrained computational resources, though it is incremental as it builds on known methods for large models.
The study tackled the problem of improving data efficiency for small language models (under 500M parameters) by exploring transfer methods like multi-task learning (MTL) and instruction tuning, finding that general purpose MTL gave a 31% relative improvement and in-domain MTL added 37.6%, while instruction tuning only provided a 2% gain.
Multi-task learning (MTL), instruction tuning, and prompting have recently been shown to improve the generalizability of large language models to new tasks. However, the benefits of such methods are less well-documented in smaller language models, with some studies finding contradictory results. In this work, we explore and isolate the effects of (i) model size, (ii) general purpose MTL, (iii) in-domain MTL, (iv) instruction tuning, and (v) few-shot fine-tuning for models with fewer than 500 million parameters. Our experiments in the zero-shot setting demonstrate that models gain 31% relative improvement, on average, from general purpose MTL, with an additional 37.6% relative gain from in-domain MTL. Contradictory to prior works on large models, we find that instruction tuning provides a modest 2% performance improvement for small models.