Learning Instance-Level Representation for Large-Scale Multi-Modal Pretraining in E-commerce
It addresses the need for scalable multi-modal foundation models in E-commerce, offering a domain-specific improvement over existing frameworks.
The paper tackles the problem of sub-optimal performance when applying general vision-language pretraining to E-commerce due to differences between natural and product images, resulting in ECLIP, which surpasses existing methods by a large margin on downstream tasks without fine-tuning.
This paper aims to establish a generic multi-modal foundation model that has the scalable capability to massive downstream applications in E-commerce. Recently, large-scale vision-language pretraining approaches have achieved remarkable advances in the general domain. However, due to the significant differences between natural and product images, directly applying these frameworks for modeling image-level representations to E-commerce will be inevitably sub-optimal. To this end, we propose an instance-centric multi-modal pretraining paradigm called ECLIP in this work. In detail, we craft a decoder architecture that introduces a set of learnable instance queries to explicitly aggregate instance-level semantics. Moreover, to enable the model to focus on the desired product instance without reliance on expensive manual annotations, two specially configured pretext tasks are further proposed. Pretrained on the 100 million E-commerce-related data, ECLIP successfully extracts more generic, semantic-rich, and robust representations. Extensive experimental results show that, without further fine-tuning, ECLIP surpasses existing methods by a large margin on a broad range of downstream tasks, demonstrating the strong transferability to real-world E-commerce applications.