Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluation
This addresses the need for scalable and cost-effective evaluation of LLMs in dynamic settings, though it appears incremental as it builds on existing interaction-based methods.
The paper tackles the problem of evaluating LLMs in dynamic real-world scenarios by proposing a deep interaction-based framework that uses interactions between LLMs in designed tasks, demonstrating effectiveness through experiments on four tasks.
Large Language Models (LLMs) have made progress in various real-world tasks, which stimulates requirements for the evaluation of LLMs. Existing LLM evaluation methods are mainly supervised signal-based which depends on static datasets and cannot evaluate the ability of LLMs in dynamic real-world scenarios where deep interaction widely exists. Other LLM evaluation methods are human-based which are costly and time-consuming and are incapable of large-scale evaluation of LLMs. To address the issues above, we propose a novel Deep Interaction-based LLM-evaluation framework. In our proposed framework, LLMs' performances in real-world domains can be evaluated from their deep interaction with other LLMs in elaborately designed evaluation tasks. Furthermore, our proposed framework is a general evaluation method that can be applied to a host of real-world tasks such as machine translation and code generation. We demonstrate the effectiveness of our proposed method through extensive experiments on four elaborately designed evaluation tasks.