What Makes a Top-Performing Precision Medicine Search Engine? Tracing Main System Features in a Systematic Way
This work addresses the explanatory gap for researchers and practitioners in information retrieval by systematically identifying key features in precision medicine search, though it is incremental as it builds on existing TREC challenges and methods.
The study tackled the problem of understanding which system features most impact performance in precision medicine search engines by conducting an ablation analysis on a BM25-based system using TREC-PM data, finding that features like query expansion and BM25 parameters significantly affect effectiveness as measured by infNDCG.
From 2017 to 2019 the Text REtrieval Conference (TREC) held a challenge task on precision medicine using documents from medical publications (PubMed) and clinical trials. Despite lots of performance measurements carried out in these evaluation campaigns, the scientific community is still pretty unsure about the impact individual system features and their weights have on the overall system performance. In order to overcome this explanatory gap, we first determined optimal feature configurations using the Sequential Model-based Algorithm Configuration (SMAC) program and applied its output to a BM25-based search engine. We then ran an ablation study to systematically assess the individual contributions of relevant system features: BM25 parameters, query type and weighting schema, query expansion, stop word filtering, and keyword boosting. For evaluation, we employed the gold standard data from the three TREC-PM installments to evaluate the effectiveness of different features using the commonly shared infNDCG metric.