Learning to Instruct for Visual Instruction Tuning
This addresses performance degradation in multimodal AI systems by reducing over-reliance on language priors, though it is an incremental improvement over existing visual instruction tuning methods.
The paper tackles the problem of overfitting and shortcut learning in visual instruction tuning for multimodal LLMs by incorporating the loss function into both instruction and response sequences, achieving up to 9% improvement on multimodal benchmarks and 18% better captioning performance with no extra data or computational cost.
We propose L2T, an advancement of visual instruction tuning (VIT). While VIT equips Multimodal LLMs (MLLMs) with promising multimodal capabilities, the current design choices for VIT often result in overfitting and shortcut learning, potentially degrading performance. This gap arises from an overemphasis on instruction-following abilities, while neglecting the proactive understanding of visual information. Inspired by this, L2T adopts a simple yet effective approach by incorporating the loss function into both the instruction and response sequences. It seamlessly expands the training data, and regularizes the MLLMs from overly relying on language priors. Based on this merit, L2T achieves a significant relative improvement of up to 9% on comprehensive multimodal benchmarks, requiring no additional training data and incurring negligible computational overhead. Surprisingly, L2T attains exceptional fundamental visual capabilities, yielding up to an 18% improvement in captioning performance, while simultaneously alleviating hallucination in MLLMs. Github code: https://github.com/Feng-Hong/L2T.