Rachel Xu

Ian Beacock, Rachel Xu, Laura Murray et al.

As conversational AI systems become more deeply integrated into daily life, the implications for human agency are increasingly urgent to understand. AI's potential to amplify capability sits alongside risks of individual and collective disempowerment, yet empirical, ecologically-valid evidence about cumulative usage is scarce. We analyze deep ethnographic data from a study of daily AI chatbot users (n = 51) in the United States, Germany, and Singapore to illuminate conversational AI usage in situated context as a sociotechnical practice. We show that people consistently link sustained AI usage to perceived gains in individual agency. Crucially, these perceived gains often outweigh concerns about accuracy, reliability, and consistency to shape usage patterns. Our findings challenge prevailing assumptions about how and why humans use AI systems over time, suggesting that traditional trust-based models are not sufficient for explaining human behavior with conversational AI. Finally, we expose a critical tension: immediate psychological boosts to perceived agency may not necessarily translate into material effects, structural empowerment, or long-term capacity. Our results help establish a new foundation for novel behavioral frameworks, measurement tools, and AI benchmarks to ensure conversational AI strengthens human agency in substantial, sustained ways.

Rachel Xu, Nhu Le, Rebekah Park et al.

We conducted in-person ethnography in India and the US to investigate how young people (18-24) trusted online content, just as generative AI (genAI) became mainstream. We found that when online, how participants determined what content to trust was shaped by emotional states, which we term "information modes." Our participants reflexively shifted between modes to maintain "emotional equilibrium," and eschewed engaging literacy skills in the more passive modes in which they spent the most time. We found participants imported trust heuristics from established online contexts into emerging ones (i.e., genAI). This led them to use ill-fitting trust heuristics, and exposed them to the risk of trusting false and misleading information. While many had reservations about AI, prioritizing efficiency, they used genAI and habitual heuristics to quickly achieve goals at the expense of accuracy. We conclude that literacy interventions designed to match users' distinct information modes will be most effective.

Nahema Marchal, Rachel Xu, Rasmi Elasmar et al.

Generative, multimodal artificial intelligence (GenAI) offers transformative potential across industries, but its misuse poses significant risks. Prior research has shed light on the potential of advanced AI systems to be exploited for malicious purposes. However, we still lack a concrete understanding of how GenAI models are specifically exploited or abused in practice, including the tactics employed to inflict harm. In this paper, we present a taxonomy of GenAI misuse tactics, informed by existing academic literature and a qualitative analysis of approximately 200 observed incidents of misuse reported between January 2023 and March 2024. Through this analysis, we illuminate key and novel patterns in misuse during this time period, including potential motivations, strategies, and how attackers leverage and abuse system capabilities across modalities (e.g. image, text, audio, video) in the wild.