EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models
This provides a domain-specific evaluation method for selecting and fine-tuning language models in environmental science, but it is incremental as it applies existing benchmarking techniques to a new domain.
The authors tackled the problem of evaluating large language models' knowledge in environmental science by proposing EnviroExam, a benchmark based on university curricula with 936 questions across 42 courses, and found that 61.3% of models passed 5-shot tests and 48.39% passed 0-shot tests.
In the field of environmental science, it is crucial to have robust evaluation metrics for large language models to ensure their efficacy and accuracy. We propose EnviroExam, a comprehensive evaluation method designed to assess the knowledge of large language models in the field of environmental science. EnviroExam is based on the curricula of top international universities, covering undergraduate, master's, and doctoral courses, and includes 936 questions across 42 core courses. By conducting 0-shot and 5-shot tests on 31 open-source large language models, EnviroExam reveals the performance differences among these models in the domain of environmental science and provides detailed evaluation standards. The results show that 61.3% of the models passed the 5-shot tests, while 48.39% passed the 0-shot tests. By introducing the coefficient of variation as an indicator, we evaluate the performance of mainstream open-source large language models in environmental science from multiple perspectives, providing effective criteria for selecting and fine-tuning language models in this field. Future research will involve constructing more domain-specific test sets using specialized environmental science textbooks to further enhance the accuracy and specificity of the evaluation.