Dice in the Black Box: User Experiences with an Inscrutable Algorithm
This work addresses the problem of over-trust in AI systems for users, highlighting risks in human-AI interaction, but it is incremental as it builds on existing research about algorithm trust.
The study investigated user trust in black box algorithms by deploying a random-response algorithm framed as intelligent for assessing emotional writing, finding that users placed inordinate trust in it. The researchers qualitatively analyzed trust pathways and recommended corrective approaches.
We demonstrate that users may be prone to place an inordinate amount of trust in black box algorithms that are framed as intelligent. We deploy an algorithm that purportedly assesses the positivity and negativity of a users' writing emotional writing. In actuality, the algorithm responds in a random fashion. We qualitatively examine the paths to trust that users followed while testing the system. In light of the ease with which users may trust systems exhibiting "intelligent behavior" we recommend corrective approaches.