Pretrained Generative Language Models as General Learning Frameworks for Sequence-Based Tasks
This provides a general learning framework for sequence-based tasks, addressing computational and skill barriers, though it is incremental as it builds on existing pretrained models.
The authors tackled the challenge of applying pretrained generative language models to sequence-based tasks by instruction fine-tuning small models (125M to 1.3B parameters) with 10,000 to 1,000,000 examples, achieving near state-of-the-art results on cheminformatics tasks.
We propose that small pretrained foundational generative language models with millions of parameters can be utilized as a general learning framework for sequence-based tasks. Our proposal overcomes the computational resource, skill set, and timeline challenges associated with training neural networks and language models from scratch. Further, our approach focuses on creating small and highly specialized models that can accurately execute a challenging task of which the base model is incapable of performing. We demonstrate that 125M, 350M, and 1.3B parameter pretrained foundational language models can be instruction fine-tuned with 10,000-to-1,000,000 instruction examples to achieve near state-of-the-art results on challenging cheminformatics tasks. We also demonstrate the role of successive language model fine-tuning epochs on improved outcomes, as well as the importance of both data formatting and pretrained foundational language model selection for instruction fine-tuning success.