Sliding Spectrum Decomposition for Diversified Recommendation
This addresses diversity in content feeds for social media users, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackled the diversity problem in content feed recommendations by proposing sliding spectrum decomposition (SSD) to capture user perception of diversity in long item sequences, and it was successfully deployed in Xiaohongshu App's Explore Feed, serving tens of millions of users daily with demonstrated effectiveness and efficiency.
Content feed, a type of product that recommends a sequence of items for users to browse and engage with, has gained tremendous popularity among social media platforms. In this paper, we propose to study the diversity problem in such a scenario from an item sequence perspective using time series analysis techniques. We derive a method called sliding spectrum decomposition (SSD) that captures users' perception of diversity in browsing a long item sequence. We also share our experiences in designing and implementing a suitable item embedding method for accurate similarity measurement under long tail effect. Combined together, they are now fully implemented and deployed in Xiaohongshu App's production recommender system that serves the main Explore Feed product for tens of millions of users every day. We demonstrate the effectiveness and efficiency of the method through theoretical analysis, offline experiments and online A/B tests.