Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding
This work addresses the challenge of multi-task learning for scientific literature understanding, which could accelerate scientific discovery, but it is incremental as it builds on existing pre-trained language models and contrastive learning techniques.
The authors tackled the problem of jointly utilizing pre-training data across multiple heterogeneous scientific literature understanding tasks by proposing SciMult, a multi-task contrastive learning framework, and achieved state-of-the-art performance on benchmark datasets.
Scientific literature understanding tasks have gained significant attention due to their potential to accelerate scientific discovery. Pre-trained language models (LMs) have shown effectiveness in these tasks, especially when tuned via contrastive learning. However, jointly utilizing pre-training data across multiple heterogeneous tasks (e.g., extreme multi-label paper classification, citation prediction, and literature search) remains largely unexplored. To bridge this gap, we propose a multi-task contrastive learning framework, SciMult, with a focus on facilitating common knowledge sharing across different scientific literature understanding tasks while preventing task-specific skills from interfering with each other. To be specific, we explore two techniques -- task-aware specialization and instruction tuning. The former adopts a Mixture-of-Experts Transformer architecture with task-aware sub-layers; the latter prepends task-specific instructions to the input text so as to produce task-aware outputs. Extensive experiments on a comprehensive collection of benchmark datasets verify the effectiveness of our task-aware specialization strategy, where we outperform state-of-the-art scientific pre-trained LMs. Code, datasets, and pre-trained models can be found at https://scimult.github.io/.