Beyond Testers' Biases: Guiding Model Testing with Knowledge Bases using LLMs
This addresses the challenge of systematic requirements elicitation for model testing, particularly for practitioners in NLP and related domains, though it is incremental in automating part of the testing process.
The paper tackles the problem of identifying what to test in model testing by proposing Weaver, an interactive tool that uses large language models to generate knowledge bases and recommend concepts, helping testers elicit requirements beyond their biases. In a user study, testers identified more diverse concepts and found over 200 failing test cases for stance detection with zero-shot ChatGPT.
Current model testing work has mostly focused on creating test cases. Identifying what to test is a step that is largely ignored and poorly supported. We propose Weaver, an interactive tool that supports requirements elicitation for guiding model testing. Weaver uses large language models to generate knowledge bases and recommends concepts from them interactively, allowing testers to elicit requirements for further testing. Weaver provides rich external knowledge to testers and encourages testers to systematically explore diverse concepts beyond their own biases. In a user study, we show that both NLP experts and non-experts identified more, as well as more diverse concepts worth testing when using Weaver. Collectively, they found more than 200 failing test cases for stance detection with zero-shot ChatGPT. Our case studies further show that Weaver can help practitioners test models in real-world settings, where developers define more nuanced application scenarios (e.g., code understanding and transcript summarization) using LLMs.