Oracle-Checker Scheme for Evaluating a Generative Large Language Model
This work addresses the challenge of reliable evaluation for generative LLMs, which is crucial for researchers and developers, but it appears incremental as it adapts existing checking concepts to new contexts.
The authors tackled the problem of evaluating generative large language models by proposing an oracle-checker scheme, which uses property testing and program checking methods, and demonstrated its application in entity extraction and paraphrase decision tasks.
This work presents a novel approach called oracle-checker scheme for evaluating the answer given by a generative large language model (LLM). Two types of checkers are presented. The first type of checker follows the idea of property testing. The second type of checker follows the idea of program checking. Their applications are demonstrated in two separate contexts, entity extraction and paraphrase decision, respectively.