Are Frontier Large Language Models Suitable for Q&A in Science Centres?
This addresses the practical problem of using LLMs for educational engagement in science centers, though it is incremental as it applies existing models to a new domain.
This paper investigated whether frontier Large Language Models (LLMs) are suitable for Q&A in science centers, evaluating GPT-4, Claude 3.5 Sonnet, and Gemini 1.5 on questions from the National Space Centre. The results showed Claude outperformed others in clarity and engagement for young audiences, but all models exhibited a trade-off where higher novelty reduced factual reliability.
This paper investigates the suitability of frontier Large Language Models (LLMs) for Q&A interactions in science centres, with the aim of boosting visitor engagement while maintaining factual accuracy. Using a dataset of questions collected from the National Space Centre in Leicester (UK), we evaluated responses generated by three leading models: OpenAI's GPT-4, Claude 3.5 Sonnet, and Google Gemini 1.5. Each model was prompted for both standard and creative responses tailored to an 8-year-old audience, and these responses were assessed by space science experts based on accuracy, engagement, clarity, novelty, and deviation from expected answers. The results revealed a trade-off between creativity and accuracy, with Claude outperforming GPT and Gemini in both maintaining clarity and engaging young audiences, even when asked to generate more creative responses. Nonetheless, experts observed that higher novelty was generally associated with reduced factual reliability across all models. This study highlights the potential of LLMs in educational settings, emphasizing the need for careful prompt engineering to balance engagement with scientific rigor.