Consistency and Variation in Kernel Neural Ranking Model
This addresses reproducibility and deployment issues for neural IR models, offering an incremental improvement through ensemble methods.
The paper investigates the consistency of the K-NRM neural ranking model, finding low variance in overall performance but high variation in individual query rankings due to different latent matching patterns in embeddings, and shows that ensemble rankers built on these patterns improve effectiveness and generalization.
This paper studies the consistency of the kernel-based neural ranking model K-NRM, a recent state-of-the-art neural IR model, which is important for reproducible research and deployment in the industry. We find that K-NRM has low variance on relevance-based metrics across experimental trials. In spite of this low variance in overall performance, different trials produce different document rankings for individual queries. The main source of variance in our experiments was found to be different latent matching patterns captured by K-NRM. In the IR-customized word embeddings learned by K-NRM, the query-document word pairs follow two different matching patterns that are equally effective, but align word pairs differently in the embedding space. The different latent matching patterns enable a simple yet effective approach to construct ensemble rankers, which improve K-NRM's effectiveness and generalization abilities.