Adopt $\neq$ Adapt: Longitudinal Analyses of LLM Conversations in the Wild
For researchers studying user-LLM interactions, this work highlights the importance of longitudinal analysis and cautions against generalizing from datasets like WildChat.
This paper analyzes the conversational trajectories of ~12,000 Bing Copilot users and compares them with WildChat-4.8M, finding that individual user habits are overwhelmingly sticky and that WildChat is skewed towards power users, not representing typical user-AI interactions.
Although a growing body of research has begun to describe user--LLM interactions, the picture it paints is largely static; little is known about how individual users change their behavior over time. To address this gap, we analyze the conversational trajectories of $\sim$12,000 randomly sampled Microsoft Bing Copilot users and compare these with data from WildChat-4.8M. While the Copilot data contains significant population-level trends, we find that trends in individual user trajectories are much weaker; user habits prove to be overwhelmingly sticky. We also find stark differences between users of different activity levels: more active users have more successful conversations and use the LLM for more complex and professionally oriented tasks. Some user trends also appear in WildChat-4.8M, but we find evidence that this dataset is significantly skewed towards highly proficient "power" users. Ultimately, our results suggest that existing user behavior is difficult to change and demonstrate the extent of user heterogeneity. Our comparison between datasets highlights that WildChat does not represent typical user-AI interactions, an important caveat for downstream uses of the data.