Autonomy Matters: A Study on Personalization-Privacy Dilemma in LLM Agents
This addresses the trade-off between personalization and privacy for users of LLM agents, offering insights into autonomy moderation, but it is incremental as it builds on existing dilemma research.
The study investigated the personalization-privacy dilemma in LLM agents, finding that personalization without privacy preferences increases privacy concerns and reduces trust and willingness to use, with intermediate autonomy moderating these effects compared to no or full autonomy.
Large Language Model (LLM) agents require personal information for personalization in order to better act on users' behalf in daily tasks, but this raises privacy concerns and a personalization-privacy dilemma. Agent's autonomy introduces both risks and opportunities, yet its effects remain unclear. To better understand this, we conducted a 3$\times$3 between-subjects experiment ($N=450$) to study how agent's autonomy level and personalization influence users' privacy concerns, trust and willingness to use, as well as the underlying psychological processes. We find that personalization without considering users' privacy preferences increases privacy concerns and decreases trust and willingness to use. Autonomy moderates these effects: Intermediate autonomy flattens the impact of personalization compared to No- and Full autonomy conditions. Our results suggest that rather than aiming for perfect model alignment in output generation, balancing autonomy of agent's action and user control offers a promising path to mitigate the personalization-privacy dilemma.