Testing LLM performance on the Physics GRE: some observations
This work addresses the need to benchmark LLMs for STEM education applications, though it appears incremental as it focuses on a single model and test.
The paper evaluated Bard, a large language model, on the Physics GRE examination to assess its performance and limitations, but did not provide specific numerical results.
With the recent developments in large language models (LLMs) and their widespread availability through open source models and/or low-cost APIs, several exciting products and applications are emerging, many of which are in the field of STEM educational technology for K-12 and university students. There is a need to evaluate these powerful language models on several benchmarks, in order to understand their risks and limitations. In this short paper, we summarize and analyze the performance of Bard, a popular LLM-based conversational service made available by Google, on the standardized Physics GRE examination.