Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting
This work provides a benchmarking framework and tools to support LLM applications in bioinformatics, addressing the lack of effective evaluation across diverse tasks, but it is incremental as it builds on existing benchmarking efforts.
The authors introduced Bio-benchmark, a comprehensive prompting-based framework with 30 bioinformatics tasks, and evaluated six LLMs like GPT-4o and Llama-3.1-70b using 0-shot and few-shot CoT settings, showing that their BioFinder tool increased answer extraction accuracy by around 30% compared to existing methods.
Large language models (LLMs) have become important tools in solving biological problems, offering improvements in accuracy and adaptability over conventional methods. Several benchmarks have been proposed to evaluate the performance of these LLMs. However, current benchmarks can hardly evaluate the performance of these models across diverse tasks effectively. In this paper, we introduce a comprehensive prompting-based benchmarking framework, termed Bio-benchmark, which includes 30 key bioinformatics tasks covering areas such as proteins, RNA, drugs, electronic health records, and traditional Chinese medicine. Using this benchmark, we evaluate six mainstream LLMs, including GPT-4o and Llama-3.1-70b, etc., using 0-shot and few-shot Chain-of-Thought (CoT) settings without fine-tuning to reveal their intrinsic capabilities. To improve the efficiency of our evaluations, we demonstrate BioFinder, a new tool for extracting answers from LLM responses, which increases extraction accuracy by round 30% compared to existing methods. Our benchmark results show the biological tasks suitable for current LLMs and identify specific areas requiring enhancement. Furthermore, we propose targeted prompt engineering strategies for optimizing LLM performance in these contexts. Based on these findings, we provide recommendations for the development of more robust LLMs tailored for various biological applications. This work offers a comprehensive evaluation framework and robust tools to support the application of LLMs in bioinformatics.