A General Framework for Multimodal LLM-Based Multimedia Understanding in Large-Scale Recommendation Systems
It addresses the challenge of using MM-LLMs in latency-constrained industrial recommendation systems, but the gains are marginal.
The paper proposes a framework integrating Multimodal LLMs into large-scale recommendation systems, achieving a 0.35% offline AUC increase and 0.02% online metric improvement.
Conventional recommendation systems frequently fail to fully exploit the high-dimensional semantic signals inherent in multimedia content, thereby limiting the fidelity of user preference modeling. While Multimodal Large Language Models (MM-LLMs) offer robust mechanisms for interpreting such complex data, their integration into latency-constrained, industrial-scale architectures remains a significant challenge. To address this, we propose a generalized framework for MM-LLM-driven multimedia understanding. Our methodology employs a tripartite architecture encompassing content interpretation, representation extraction, and systematic pipeline integration, instantiated via a LLaMA2-based model that generates descriptive captions subsequently ingested as tokenized categorical features. Empirical evaluation demonstrates the efficacy of this approach, yielding a $0.35\%$ increase in offline AUC and a $0.02\%$ improvement in online metrics at scale, substantiating the practical viability of leveraging MM-LLMs to enhance large-scale recommendation performance.