MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale
This work addresses the problem of improving zero-shot transfer performance for text matching tasks, benefiting researchers and practitioners in natural language processing, though it is incremental as it builds on existing self-supervised and multi-task learning methods.
The paper tackled zero-shot transfer of text matching models by self-supervised training on 140 source domains from community question answering forums, showing that most models outperformed IR baselines and that broad domain selection is crucial, with the best model achieving new state-of-the-art results on nine benchmarks after fine-tuning.
We study the zero-shot transfer capabilities of text matching models on a massive scale, by self-supervised training on 140 source domains from community question answering forums in English. We investigate the model performances on nine benchmarks of answer selection and question similarity tasks, and show that all 140 models transfer surprisingly well, where the large majority of models substantially outperforms common IR baselines. We also demonstrate that considering a broad selection of source domains is crucial for obtaining the best zero-shot transfer performances, which contrasts the standard procedure that merely relies on the largest and most similar domains. In addition, we extensively study how to best combine multiple source domains. We propose to incorporate self-supervised with supervised multi-task learning on all available source domains. Our best zero-shot transfer model considerably outperforms in-domain BERT and the previous state of the art on six benchmarks. Fine-tuning of our model with in-domain data results in additional large gains and achieves the new state of the art on all nine benchmarks.