ITEm: Unsupervised Image-Text Embedding Learning for eCommerce
This work addresses the challenge of multimodal embedding learning for eCommerce applications, offering an incremental improvement over existing methods.
The paper tackles the problem of learning product embeddings from multiple modalities in eCommerce, where some modalities dominate others, by proposing an unsupervised image-text embedding model (ITEm) that extends BERT to better attend to both image and text, resulting in substantial gains on tasks like product search and category prediction compared to strong baselines.
Product embedding serves as a cornerstone for a wide range of applications in eCommerce. The product embedding learned from multiple modalities shows significant improvement over that from a single modality, since different modalities provide complementary information. However, some modalities are more informatively dominant than others. How to teach a model to learn embedding from different modalities without neglecting information from the less dominant modality is challenging. We present an image-text embedding model (ITEm), an unsupervised learning method that is designed to better attend to image and text modalities. We extend BERT by (1) learning an embedding from text and image without knowing the regions of interest; (2) training a global representation to predict masked words and to construct masked image patches without their individual representations. We evaluate the pre-trained ITEm on two tasks: the search for extremely similar products and the prediction of product categories, showing substantial gains compared to strong baseline models.