Algorithms and Architecture for Real-time Recommendations at News UK
This addresses the need for real-time recommendation updates in news publishing, though it is incremental as it builds on existing collaborative filtering methods.
The paper tackles the problem of generating real-time recommendations for news items as they are published, by introducing a new incremental collaborative filtering algorithm and scalable architecture, demonstrating its effectiveness on clickstream data from The Times and deploying it in production at News UK.
Recommendation systems are recognised as being hugely important in industry, and the area is now well understood. At News UK, there is a requirement to be able to quickly generate recommendations for users on news items as they are published. However, little has been published about systems that can generate recommendations in response to changes in recommendable items and user behaviour in a very short space of time. In this paper we describe a new algorithm for updating collaborative filtering models incrementally, and demonstrate its effectiveness on clickstream data from The Times. We also describe the architecture that allows recommendations to be generated on the fly, and how we have made each component scalable. The system is currently being used in production at News UK.