Evaluating Large Language Models for automatic analysis of teacher simulations
This addresses the need for automatic evaluation in teacher education simulations, though it is incremental as it compares existing LLMs on a specific application.
The researchers tackled the problem of automatically analyzing open-ended responses in digital teacher simulations by evaluating Large Language Models (LLMs) like DeBERTaV3 and Llama 3 with various techniques. They found that Llama 3 performed better and more stably than DeBERTaV3, especially when identifying new characteristics, with performance varying significantly depending on the characteristic.
Digital Simulations (DS) provide safe environments where users interact with an agent through conversational prompts, providing engaging learning experiences that can be used to train teacher candidates in realistic classroom scenarios. These simulations usually include open-ended questions, allowing teacher candidates to express their thoughts but complicating an automatic response analysis. To address this issue, we have evaluated Large Language Models (LLMs) to identify characteristics (user behaviors) in the responses of DS for teacher education. We evaluated the performance of DeBERTaV3 and Llama 3, combined with zero-shot, few-shot, and fine-tuning. Our experiments discovered a significant variation in the LLMs' performance depending on the characteristic to identify. Additionally, we noted that DeBERTaV3 significantly reduced its performance when it had to identify new characteristics. In contrast, Llama 3 performed better than DeBERTaV3 in detecting new characteristics and showing more stable performance. Therefore, in DS where teacher educators need to introduce new characteristics because they change depending on the simulation or the educational objectives, it is more recommended to use Llama 3. These results can guide other researchers in introducing LLMs to provide the highly demanded automatic evaluations in DS.