Characteristic AI Agents via Large Language Models
This work addresses a gap in academic research for role-driven chatbots using LLMs, providing a benchmark for the community, but it is incremental as it builds on existing commercial products and focuses on evaluation rather than novel methods.
The authors tackled the problem of evaluating Large Language Models (LLMs) in constructing Characteristic AI Agents by simulating real-life individuals, creating a benchmark called Character100 with a dataset, techniques, and evaluation metrics, and found experimental results that highlight potential directions for improvement.
The advancement of Large Language Models (LLMs) has led to significant enhancements in the performance of chatbot systems. Many researchers have dedicated their efforts to the development of bringing characteristics to chatbots. While there have been commercial products for developing role-driven chatbots using LLMs, it is worth noting that academic research in this area remains relatively scarce. Our research focuses on investigating the performance of LLMs in constructing Characteristic AI Agents by simulating real-life individuals across different settings. Current investigations have primarily focused on act on roles with simple profiles. In response to this research gap, we create a benchmark for the characteristic AI agents task, including dataset, techniques, and evaluation metrics. A dataset called ``Character100'' is built for this benchmark, comprising the most-visited people on Wikipedia for language models to role-play. With the constructed dataset, we conduct comprehensive assessment of LLMs across various settings. In addition, we devise a set of automatic metrics for quantitative performance evaluation. The experimental results underscore the potential directions for further improvement in the capabilities of LLMs in constructing characteristic AI agents. The benchmark is available at https://github.com/nuaa-nlp/Character100.