Self-Evaluation of Large Language Model based on Glass-box Features
This addresses the need for model-aware evaluation methods in open-source LLMs, offering a novel approach that could enhance evaluation efficiency and accuracy.
The study tackled the problem of evaluating Large Language Models by proposing a self-evaluation method using glass-box features, specifically finding that softmax distribution reliably indicates output quality, with experimental validation on public benchmarks.
The proliferation of open-source Large Language Models (LLMs) underscores the pressing need for evaluation methods. Existing works primarily rely on external evaluators, focusing on training and prompting strategies. However, a crucial aspect, model-aware glass-box features, is overlooked. In this study, we explore the utility of glass-box features under the scenario of self-evaluation, namely applying an LLM to evaluate its own output. We investigate various glass-box feature groups and discovered that the softmax distribution serves as a reliable quality indicator for self-evaluation. Experimental results on public benchmarks validate the feasibility of self-evaluation of LLMs using glass-box features.