Is ChatGPT a Biomedical Expert? -- Exploring the Zero-Shot Performance of Current GPT Models in Biomedical Tasks
This work assesses the practical utility of commercial LLMs for biomedical experts, showing incremental gains in specific tasks but limitations in retrieval.
The study evaluated GPT-3.5-Turbo and GPT-4 on biomedical tasks from the 2023 BioASQ challenge, finding that in answer generation, they achieved competitive performance with leading systems using zero-shot learning, while in retrieval, they fell short despite improvements from query expansion.
We assessed the performance of commercial Large Language Models (LLMs) GPT-3.5-Turbo and GPT-4 on tasks from the 2023 BioASQ challenge. In Task 11b Phase B, which is focused on answer generation, both models demonstrated competitive abilities with leading systems. Remarkably, they achieved this with simple zero-shot learning, grounded with relevant snippets. Even without relevant snippets, their performance was decent, though not on par with the best systems. Interestingly, the older and cheaper GPT-3.5-Turbo system was able to compete with GPT-4 in the grounded Q&A setting on factoid and list answers. In Task 11b Phase A, focusing on retrieval, query expansion through zero-shot learning improved performance, but the models fell short compared to other systems. The code needed to rerun these experiments is available through GitHub.