Practical User Feedback-driven Internal Search Using Online Learning to Rank
This addresses the need for more effective and adaptable search in organizational knowledge bases, though it is incremental as it builds on existing learning-to-rank methods with practical deployment enhancements.
The paper tackles the problem of improving internal knowledge base search by developing Spoke, a system that uses conversational user feedback and online learning-to-rank to customize relevance scoring for each organization, resulting in up to 41% better performance in offline F1 comparisons than baselines.
We present a system, Spoke, for creating and searching internal knowledge base (KB) articles for organizations. Spoke is available as a SaaS (Software-as-a-Service) product deployed across hundreds of organizations with a diverse set of domains. Spoke continually improves search quality using conversational user feedback which allows it to provide better search experience than standard information retrieval systems without encoding any explicit domain knowledge. We achieve this by using a real-time online learning-to-rank (L2R) algorithm that automatically customizes relevance scoring for each organization deploying Spoke by using a query similarity kernel. The focus of this paper is on incorporating practical considerations into our relevance scoring function and algorithm that make Spoke easy to deploy and suitable for handling events that naturally happen over the life-cycle of any KB deployment. We show that Spoke outperforms competitive baselines by up to 41% in offline F1 comparisons.