Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation
This work addresses efficiency and domain adaptation challenges in news recommendation systems, offering a practical solution for low-latency online services, though it is incremental as it builds on existing PLM-based methods.
The paper tackles the domain shift and computational inefficiency of using pre-trained language models (PLMs) for news recommendation by proposing Tiny-NewsRec, which uses domain-specific post-training and knowledge distillation, achieving improved performance and efficiency validated on two real-world datasets.
News recommendation is a widely adopted technique to provide personalized news feeds for the user. Recently, pre-trained language models (PLMs) have demonstrated the great capability of natural language understanding and benefited news recommendation via improving news modeling. However, most existing works simply finetune the PLM with the news recommendation task, which may suffer from the known domain shift problem between the pre-training corpus and downstream news texts. Moreover, PLMs usually contain a large volume of parameters and have high computational overhead, which imposes a great burden on low-latency online services. In this paper, we propose Tiny-NewsRec, which can improve both the effectiveness and the efficiency of PLM-based news recommendation. We first design a self-supervised domain-specific post-training method to better adapt the general PLM to the news domain with a contrastive matching task between news titles and news bodies. We further propose a two-stage knowledge distillation method to improve the efficiency of the large PLM-based news recommendation model while maintaining its performance. Multiple teacher models originated from different time steps of our post-training procedure are used to transfer comprehensive knowledge to the student in both its post-training and finetuning stage. Extensive experiments on two real-world datasets validate the effectiveness and efficiency of our method.