Evaluation of Multilingual LLMs Personalized Text Generation Capabilities Targeting Groups and Social-Media Platforms
This work addresses the problem of assessing misuse and benefits of personalized text generation in multiple languages for researchers and policymakers, but it is incremental as it extends prior English-only studies to more languages.
The study evaluated multilingual large language models' ability to generate personalized text across 10 languages, analyzing 17,280 texts from 16 models to find that personalization quality varies by demographic groups and social-media platforms, with platform-targeted personalization reducing detectability more, especially in English.
Capabilities of large language models to generate multilingual coherent text have continuously enhanced in recent years, which opens concerns about their potential misuse. Previous research has shown that they can be misused for generation of personalized disinformation in multiple languages. It has also been observed that personalization negatively affects detectability of machine-generated texts; however, this has been studied in the English language only. In this work, we examine this phenomenon across 10 languages, while we focus not only on potential misuse of personalization capabilities, but also on potential benefits they offer. Overall, we cover 1080 combinations of various personalization aspects in the prompts, for which the texts are generated by 16 distinct language models (17,280 texts in total). Our results indicate that there are differences in personalization quality of the generated texts when targeting demographic groups and when targeting social-media platforms across languages. Personalization towards platforms affects detectability of the generated texts in a higher scale, especially in English, where the personalization quality is the highest.