Optimizing Encoder-Only Transformers for Session-Based Recommendation Systems
This addresses the challenge of single-session recommendation for users without historical data, offering a novel method that is incremental in improving transformer-based approaches.
The paper tackled the problem of predicting the next item in session-based recommendation by introducing Sequential Masked Modeling for encoder-only transformers, resulting in consistent outperformance of state-of-the-art models on datasets like Yoochoose 1/64, Diginetica, and Tmall.
Session-based recommendation is the task of predicting the next item a user will interact with, often without access to historical user data. In this work, we introduce Sequential Masked Modeling, a novel approach for encoder-only transformer architectures to tackle the challenges of single-session recommendation. Our method combines data augmentation through window sliding with a unique penultimate token masking strategy to capture sequential dependencies more effectively. By enhancing how transformers handle session data, Sequential Masked Modeling significantly improves next-item prediction performance. We evaluate our approach on three widely-used datasets, Yoochoose 1/64, Diginetica, and Tmall, comparing it to state-of-the-art single-session, cross-session, and multi-relation approaches. The results demonstrate that our Transformer-SMM models consistently outperform all models that rely on the same amount of information, while even rivaling methods that have access to more extensive user history. This study highlights the potential of encoder-only transformers in session-based recommendation and opens the door for further improvements.