AntEval: Evaluation of Social Interaction Competencies in LLM-Driven Agents
This addresses the problem of evaluating social interaction competencies in AI agents for researchers and developers, though it is incremental as it builds on existing LLM capabilities.
The paper tackles the lack of robust evaluation methods for LLM-driven agents in complex social interactions by introducing AntEval, a framework with novel metrics like IEP and IEG, showing significant potential for improving agents' natural interaction abilities.
Large Language Models (LLMs) have demonstrated their ability to replicate human behaviors across a wide range of scenarios. However, their capability in handling complex, multi-character social interactions has yet to be fully explored, primarily due to the absence of robust, quantitative evaluation methods. This gap has slowed the development of agents proficient in more nuanced interactions beyond simple exchanges, for example, small talk. To address this challenge, we introduce the Multi-Agent Interaction Evaluation Framework (AntEval), encompassing a novel interaction framework and evaluation methods. The interaction framework aims to foster an complex interaction environment that bolsters information exchange and intention expression within social interactions. Furthermore, we introduce evaluation methods, including two metrics: Information Exchanging Precision (IEP) and Interaction Expressiveness Gap (IEG), designed for the quantitative and objective assessment of agents' interaction competencies. Our findings highlight the utility of these evaluative methods and show significant potential for improving LLMs' ability to construct agents that interact in a more natural manner with human-like intricacy.