Personalized Large Vision-Language Models
This addresses the need for more interactive and referentially friendly AI assistants in vision-language tasks, though it is incremental as it builds on existing LVLM frameworks.
The paper tackles the problem of personalizing large vision-language models (LVLMs) to handle interactive dialogues using referential concepts, enabling more customizable conversations and continuous addition of new concepts without extra costs, with results showing effectiveness and superiority through comprehensive analyses.
The personalization model has gained significant attention in image generation yet remains underexplored for large vision-language models (LVLMs). Beyond generic ones, with personalization, LVLMs handle interactive dialogues using referential concepts (e.g., ``Mike and Susan are talking.'') instead of the generic form (e.g., ``a boy and a girl are talking.''), making the conversation more customizable and referentially friendly. In addition, PLVM is equipped to continuously add new concepts during a dialogue without incurring additional costs, which significantly enhances the practicality. PLVM proposes Aligner, a pre-trained visual encoder to align referential concepts with the queried images. During the dialogues, it extracts features of reference images with these corresponding concepts and recognizes them in the queried image, enabling personalization. We note that the computational cost and parameter count of the Aligner are negligible within the entire framework. With comprehensive qualitative and quantitative analyses, we reveal the effectiveness and superiority of PLVM.