Evaluating the performance-deviation of itemKNN in RecBole and LensKit
This work addresses performance discrepancies in recommender system libraries for researchers and practitioners, but it is incremental as it focuses on tuning an existing method for consistency.
This study compared the performance of itemKNN algorithms in RecBole and LensKit libraries on four datasets, finding that RecBole initially outperformed LensKit on metrics like nDCG (e.g., 18% higher on ML-100K), but after adjusting LensKit's similarity matrix to match RecBole's method, both libraries achieved nearly identical results (e.g., nDCG of 0.2586 on ML-1M).
This study examines the performance of item-based k-Nearest Neighbors (ItemKNN) algorithms in the RecBole and LensKit recommender system libraries. Using four data sets (Anime, Modcloth, ML-100K, and ML-1M), we assess each library's efficiency, accuracy, and scalability, focusing primarily on normalized discounted cumulative gain (nDCG). Our results show that RecBole outperforms LensKit on two of three metrics on the ML-100K data set: it achieved an 18% higher nDCG, 14% higher precision, and 35% lower recall. To ensure a fair comparison, we adjusted LensKit's nDCG calculation to match RecBole's method. This alignment made the performance more comparable, with LensKit achieving an nDCG of 0.2540 and RecBole 0.2674. Differences in similarity matrix calculations were identified as the main cause of performance deviations. After modifying LensKit to retain only the top K similar items, both libraries showed nearly identical nDCG values across all data sets. For instance, both achieved an nDCG of 0.2586 on the ML-1M data set with the same random seed. Initially, LensKit's original implementation only surpassed RecBole in the ModCloth dataset.