Pre-trained Graphformer-based Ranking at Web-scale Search (Extended Abstract)
This work addresses the problem of distributional shifts in heterogeneous models for web-scale search ranking, presenting an incremental improvement in model integration.
The paper tackles the challenge of integrating Transformer and Graph Neural Network models for learning to rank at web scale by introducing the MPGraf model, which uses a modular pre-training strategy to combine their regression and link prediction capabilities, with extensive offline and online experiments conducted for evaluation.
Both Transformer and Graph Neural Networks (GNNs) have been employed in the domain of learning to rank (LTR). However, these approaches adhere to two distinct yet complementary problem formulations: ranking score regression based on query-webpage pairs, and link prediction within query-webpage bipartite graphs, respectively. While it is possible to pre-train GNNs or Transformers on source datasets and subsequently fine-tune them on sparsely annotated LTR datasets, the distributional shifts between the pair-based and bipartite graph domains present significant challenges in integrating these heterogeneous models into a unified LTR framework at web scale. To address this, we introduce the novel MPGraf model, which leverages a modular and capsule-based pre-training strategy, aiming to cohesively integrate the regression capabilities of Transformers with the link prediction strengths of GNNs. We conduct extensive offline and online experiments to rigorously evaluate the performance of MPGraf.