Your Next Token Prediction: A Multilingual Benchmark for Personalized Response Generation
This addresses the problem of personalizing language models for real-world communication tasks like email or social messaging, though it is incremental as it builds on existing LLM capabilities with a new benchmark.
The paper tackles the challenge of generating personalized responses that reflect individual communication styles by introducing the 'Your Next Token Prediction (YNTP)' task, using a multilingual benchmark of 100 dialogue sessions across English, Japanese, and Chinese to evaluate methods and establish a foundation for user-aligned language modeling.
Large language models (LLMs) excel at general next-token prediction but still struggle to generate responses that reflect how individuals truly communicate, such as replying to emails or social messages in their own style. However, real SNS or email histories are difficult to collect due to privacy concerns. To address this, we propose the task of "Your Next Token Prediction (YNTP)", which models a user's precise word choices through controlled human-agent conversations. We build a multilingual benchmark of 100 dialogue sessions across English, Japanese, and Chinese, where users interact for five days with psychologically grounded NPCs based on MBTI dimensions. This setup captures natural, daily-life communication patterns and enables analysis of users' internal models. We evaluate prompt-based and fine-tuning-based personalization methods, establishing the first benchmark for YNTP and a foundation for user-aligned language modeling. The dataset is available at: https://github.com/AnonymousHub4Submissions/your-next-token-prediction-dataset-100