Imitate TheWorld: A Search Engine Simulation Platform
This work addresses the gap between offline model evaluation and online business impact in e-commerce search, offering a simulation platform for more accurate performance prediction.
The paper tackles the problem of ranking models in e-commerce search that achieve high offline metrics but fail to increase real-world revenue, by building a simulated search engine called AESim that uses adversarial learning and GAIL to generate virtual users and behavior patterns from real data. The result shows that AESim better reflects online performance than classic metrics, serving as a surrogate for evaluating models without online deployment.
Recent E-commerce applications benefit from the growth of deep learning techniques. However, we notice that many works attempt to maximize business objectives by closely matching offline labels which follow the supervised learning paradigm. This results in models obtain high offline performance in terms of Area Under Curve (AUC) and Normalized Discounted Cumulative Gain (NDCG), but cannot consistently increase the revenue metrics such as purchases amount of users. Towards the issues, we build a simulated search engine AESim that can properly give feedback by a well-trained discriminator for generated pages, as a dynamic dataset. Different from previous simulation platforms which lose connection with the real world, ours depends on the real data in AliExpress Search: we use adversarial learning to generate virtual users and use Generative Adversarial Imitation Learning (GAIL) to capture behavior patterns of users. Our experiments also show AESim can better reflect the online performance of ranking models than classic ranking metrics, implying AESim can play a surrogate of AliExpress Search and evaluate models without going online.