Aligning MAGMA by Few-Shot Learning and Finetuning
This work addresses alignment issues in vision-language models for AI safety, but it appears incremental as it builds on existing MAGMA without introducing new methods.
The paper tackled the problem of aligning the MAGMA vision-language model with human values by evaluating its performance in three scenarios: out-of-the-box, after few-shot learning, and after finetuning on aligned examples, but no concrete results or numbers are provided.
The goal of vision-language modeling is to allow models to tie language understanding with visual inputs. The aim of this paper is to evaluate and align the Visual Language Model (VLM) called Multimodal Augmentation of Generative Models through Adapter-based finetuning (MAGMA) with human values. MAGMA is a VLM that is capable of image captioning and visual question-answering. We will evaluate its alignment in three different scenarios. To begin, we assess MAGMA's out-of-the-box alignment through the checkpoint provided by Hugging Face. Then, we measure if few-shot learning manages to improve the results. Finally, we finetune the model on aligned examples and evaluate its behavior.