Beyond Lexical: A Semantic Retrieval Framework for Textual SearchEngine
This work addresses the problem of improving search engine accuracy for complex queries, particularly for tail queries, which is incremental as it builds on existing BERT-based methods.
The paper tackles the challenge of retrieving relevant documents for verbose or tail queries in search engines by introducing a vector space search framework using a deep semantic matching model based on BERT architecture, resulting in considerable improvements in retrieval performance and search quality as demonstrated by offline and online metrics.
Search engine has become a fundamental component in various web and mobile applications. Retrieving relevant documents from the massive datasets is challenging for a search engine system, especially when faced with verbose or tail queries. In this paper, we explore a vector space search framework for document retrieval. Specifically, we trained a deep semantic matching model so that each query and document can be encoded as a low dimensional embedding. Our model was trained based on BERT architecture. We deployed a fast k-nearest-neighbor index service for online serving. Both offline and online metrics demonstrate that our method improved retrieval performance and search quality considerably, particularly for tail