SCITUNE: Aligning Large Language Models with Scientific Multimodal Instructions
This work addresses the challenge of aligning LLMs with scientific goals for researchers and practitioners, representing an incremental improvement over existing methods.
The authors tackled the problem of aligning large language models with scientific multimodal instructions by introducing SciTune, a tuning framework that improved model performance on the ScienceQA benchmark, surpassing human performance on average and in many sub-categories.
Instruction finetuning is a popular paradigm to align large language models (LLM) with human intent. Despite its popularity, this idea is less explored in improving the LLMs to align existing foundation models with scientific disciplines, concepts and goals. In this work, we present SciTune as a tuning framework to improve the ability of LLMs to follow scientific multimodal instructions. To test our methodology, we use a human-generated scientific instruction tuning dataset and train a large multimodal model LLaMA-SciTune that connects a vision encoder and LLM for science-focused visual and language understanding. In comparison to the models that are finetuned with machine generated data only, LLaMA-SciTune surpasses human performance on average and in many sub-categories on the ScienceQA benchmark.