Individual Turing Test: A Case Study of LLM-based Simulation Using Longitudinal Personal Data
This addresses the challenge of personalized AI simulation for applications like digital assistants or legacy systems, but it is incremental as it builds on existing LLM techniques without a breakthrough.
This paper tackled the problem of using LLMs to simulate a specific individual by evaluating methods like fine-tuning and retrieval-augmented generation on a ten-year messaging archive, finding that current methods fail to pass an Individual Turing Test with acquaintances but perform better with strangers, with fine-tuning excelling in language style and retrieval-based methods in personal opinions.
Large Language Models (LLMs) have demonstrated remarkable human-like capabilities, yet their ability to replicate a specific individual remains under-explored. This paper presents a case study to investigate LLM-based individual simulation with a volunteer-contributed archive of private messaging history spanning over ten years. Based on the messaging data, we propose the "Individual Turing Test" to evaluate whether acquaintances of the volunteer can correctly identify which response in a multi-candidate pool most plausibly comes from the volunteer. We investigate prevalent LLM-based individual simulation approaches including: fine-tuning, retrieval-augmented generation (RAG), memory-based approach, and hybrid methods that integrate fine-tuning and RAG or memory. Empirical results show that current LLM-based simulation methods do not pass the Individual Turing Test, but they perform substantially better when the same test is conducted on strangers to the target individual. Additionally, while fine-tuning improves the simulation in daily chats representing the language style of the individual, retrieval-augmented and memory-based approaches demonstrate stronger performance on questions involving personal opinions and preferences. These findings reveal a fundamental trade-off between parametric and non-parametric approaches to individual simulation with LLMs when given a longitudinal context.