VIP5: Towards Multimodal Foundation Models for Recommendation
This work addresses the challenge of integrating multiple AI modalities for recommender systems, representing an incremental advancement by adapting existing foundation model techniques to a new domain.
The paper tackles the problem of disparate modeling across computer vision, natural language processing, and recommender systems by proposing VIP5, a multimodal foundation model that unifies visual, textual, and personalization modalities under the P5 recommendation paradigm, resulting in improved recommendation performance and increased efficiency in training time and memory usage.
Computer Vision (CV), Natural Language Processing (NLP), and Recommender Systems (RecSys) are three prominent AI applications that have traditionally developed independently, resulting in disparate modeling and engineering methodologies. This has impeded the ability for these fields to directly benefit from each other's advancements. With the recent development of foundation models, large language models have emerged as a potential general-purpose interface for unifying different modalities and problem formulations. In light of this, we propose the development of a multimodal foundation model (MFM) considering visual, textual, and personalization modalities under the P5 recommendation paradigm, thus named VIP5 (Visual P5), to unify various modalities and recommendation tasks. This will enable the processing of multiple modalities in a shared architecture for improved recommendations. To achieve this, we introduce multimodal personalized prompts to accommodate multiple modalities under a shared format. Additionally, we propose a parameter-efficient training method for foundation models, which involves freezing the P5 backbone and fine-tuning lightweight adapters, resulting in improved recommendation performance and increased efficiency in terms of training time and memory usage. Code and data of VIP5 are available at https://github.com/jeykigung/VIP5.