The Limits of Goal-Setting Theory in LLM-Driven Assessment
This highlights a problem for users and developers relying on LLMs for consistent assessments, showing incremental insights into model limitations.
The paper tested whether more specific prompts improve consistency in ChatGPT's evaluation of student submissions, but found that increased prompt specificity did not reduce performance variance, challenging the assumption that LLMs behave like human evaluators.
Many users interact with AI tools like ChatGPT using a mental model that treats the system as human-like, which we call Model H. According to goal-setting theory, increased specificity in goals should reduce performance variance. If Model H holds, then prompting a chatbot with more detailed instructions should lead to more consistent evaluation behavior. This paper tests that assumption through a controlled experiment in which ChatGPT evaluated 29 student submissions using four prompts with increasing specificity. We measured consistency using intra-rater reliability (Cohen's Kappa) across repeated runs. Contrary to expectations, performance did not improve consistently with increased prompt specificity, and performance variance remained largely unchanged. These findings challenge the assumption that LLMs behave like human evaluators and highlight the need for greater robustness and improved input integration in future model development.