How to Pre-Train Your Model? Comparison of Different Pre-Training Models for Biomedical Question Answering
This work addresses overfitting issues for researchers in biomedical question answering, but it is incremental as it compares existing pre-training methods rather than introducing new ones.
The paper tackled the problem of overfitting in deep learning models on small-scale biomedical datasets by comparing pre-training strategies, finding that open domain question answering models outperform reading comprehension models when fine-tuned on the BIOASQ dataset.
Using deep learning models on small scale datasets would result in overfitting. To overcome this problem, the process of pre-training a model and fine-tuning it to the small scale dataset has been used extensively in domains such as image processing. Similarly for question answering, pre-training and fine-tuning can be done in several ways. Commonly reading comprehension models are used for pre-training, but we show that other types of pre-training can work better. We compare two pre-training models based on reading comprehension and open domain question answering models and determine the performance when fine-tuned and tested over BIOASQ question answering dataset. We find open domain question answering model to be a better fit for this task rather than reading comprehension model.