Pre-trained Language Model and Knowledge Distillation for Lightweight Sequential Recommendation
This work addresses the problem of real-time recommendation for users by providing a more efficient method, though it is incremental as it builds on existing knowledge distillation and pre-trained model techniques.
The paper tackles the challenge of applying large pre-trained language models for real-time sequential recommendation by proposing a two-stage algorithm that fine-tunes a pre-trained model and distills it into a lightweight model, resulting in enhanced recommendation accuracy and timely services as shown in experiments on multiple datasets.
Sequential recommendation models user interests based on historical behaviors to provide personalized recommendation. Previous sequential recommendation algorithms primarily employ neural networks to extract features of user interests, achieving good performance. However, due to the recommendation system datasets sparsity, these algorithms often employ small-scale network frameworks, resulting in weaker generalization capability. Recently, a series of sequential recommendation algorithms based on large pre-trained language models have been proposed. Nonetheless, given the real-time demands of recommendation systems, the challenge remains in applying pre-trained language models for rapid recommendations in real scenarios. To address this, we propose a sequential recommendation algorithm based on a pre-trained language model and knowledge distillation. The key of proposed algorithm is to transfer pre-trained knowledge across domains and achieve lightweight inference by knowledge distillation. The algorithm operates in two stages: in the first stage, we fine-tune the pre-trained language model on the recommendation dataset to transfer the pre-trained knowledge to the recommendation task; in the second stage, we distill the trained language model to transfer the learned knowledge to a lightweight model. Extensive experiments on multiple public recommendation datasets show that the proposed algorithm enhances recommendation accuracy and provide timely recommendation services.