Beliefs and Misconceptions around Integrated Conversational AI
For UX researchers and designers of integrated AI, this paper provides insights into user trust and behavior, though the findings are incremental.
The study investigates how users develop understanding and trust in integrated conversational AI, finding that participants rely on existing perceptions of LLMs and internet search, and that citations increase trustworthiness without prompting verification.
LLM-driven conversational AI is beginning to disappear into the background, shifting from something used directly towards something increasingly integrated into existing workflows. In the process, markers of origin and training are smoothed away as LLMs become commodified in the eyes of users. We explore how people approach using a web browser with conversational AI built in, focusing on how they develop their understanding and determine whether to trust its outputs. We conducted a study where 20 participants used the Copilot AI features in Microsoft Edge to conduct information retrieval and planning tasks. Participants relied on a combination of existing perceptions of LLMs and internet search, tracing the effect of beliefs about how Copilot generated answers on prompting strategies. The inclusion of citations increased the trustworthiness of answers without participants feeling the need to be check them, with participants often reaching for the same information sources as the CAI when fact-checking.