Expanding Click and Buy rates: Exploration of evaluation metrics that measure the impact of personalized recommendation engines on e-commerce platforms
This work addresses the need for more accurate evaluation metrics for recommendation systems in e-commerce, though it is incremental as it builds on existing metrics.
The paper tackles the problem of evaluating personalized recommendation models on e-commerce platforms by introducing two new metrics, CTR NoRepeat and Click & Buy rate, to address overestimation in clicks and validate purchase relevance, showing through significance tests and correlation analysis that these variants provide necessary improvements over standard metrics.
To identify the most appropriate recommendation model for an e-commerce business, a live evaluation should be performed on the shopping website to measure the influence of personalization in real-time. The aim of this paper is to introduce and justify two new metrics -- CTR NoRepeat and Click & Buy rate -- which stem from the standard metrics, Click-through(CTR) and Buy-through rate(BTR), respectively. The former variation tackles the issue of overestimation of clicks in the original CTR while the latter accounts for noting purchases of products that have been previously clicked, in order to validate that the buy included in the metric is a result of customer interactions. A significance test for independence of two means is conducted for multiple datasets, between each of the new metrics and its respective parent to determine the novelty and necessity of the variants. The Pearson-correlation coefficient is calculated to assess the strength of the linear relationships and conclude on the predictability factor amongst the aforementioned factors to investigate unknown connections between customer clicks and buys. Additionally, other metrics such as hits per customer, buyers per customer, clicks per customer etc. are introduced that help explain indicators of customer behavior on the e-commerce website in reference.