UserBERT: Modeling Long- and Short-Term User Preferences via Self-Supervision
This addresses the problem of limited labeled data for user behavior understanding in e-commerce, offering a novel self-supervised approach that is incremental in adapting BERT to a new domain.
The paper tackles the challenge of learning from unlabeled e-commerce user behavior data by extending BERT to model user actions as sequences, using self-supervision to pre-train representations that capture long- and short-term preferences. The result shows significant improvements in three real-world tasks, outperforming task-specific and multi-task learning methods.
E-commerce platforms generate vast amounts of customer behavior data, such as clicks and purchases, from millions of unique users every day. However, effectively using this data for behavior understanding tasks is challenging because there are usually not enough labels to learn from all users in a supervised manner. This paper extends the BERT model to e-commerce user data for pre-training representations in a self-supervised manner. By viewing user actions in sequences as analogous to words in sentences, we extend the existing BERT model to user behavior data. Further, our model adopts a unified structure to simultaneously learn from long-term and short-term user behavior, as well as user attributes. We propose methods for the tokenization of different types of user behavior sequences, the generation of input representation vectors, and a novel pretext task to enable the pre-trained model to learn from its own input, eliminating the need for labeled training data. Extensive experiments demonstrate that the learned representations result in significant improvements when transferred to three different real-world tasks, particularly compared to task-specific modeling and multi-task representation learning