Online-PVLM: Advancing Personalized VLMs with Online Concept Learning
This work addresses the scalability and efficiency limitations of personalized VLMs for large-scale applications, enabling real-time user-specific concept adaptation.
The paper tackles the problem of real-time adaptation in personalized visual language models by proposing Online-PVLM, a framework that uses hyperbolic representations for online concept learning without retraining, achieving state-of-the-art performance as demonstrated in extensive experiments.
Personalized Visual Language Models (VLMs) are gaining increasing attention for their formidable ability in user-specific concepts aligned interactions (e.g., identifying a user's bike). Existing methods typically require the learning of separate embeddings for each new concept, which fails to support real-time adaptation during testing. This limitation becomes particularly pronounced in large-scale scenarios, where efficient retrieval of concept embeddings is not achievable. To alleviate this gap, we propose Online-PVLM, a framework for online concept learning by leveraging hyperbolic representations. Our approach makes a train-free paradigm for concept embeddings generation at test time, making the use of personalized VLMs both scalable and efficient. In addition, we develop OP-Eval, a comprehensive and large-scale benchmark comprising 1,292 concepts and over 30K high-quality instances with diverse question types, designed to rigorously assess online concept learning in realistic scenarios. Extensive experiments demonstrate the state-of-the-art performance of our proposed framework. Our source code and dataset will be made available.