On the Suitability of LLM-Driven Agents for Dark Pattern Audits
This addresses the problem of scalable auditing for dark patterns in data rights portals, which is crucial for protecting user rights, but the approach is incremental as it builds on existing LLM capabilities for web interaction.
The study investigated whether LLM-driven agents can reliably detect manipulative interface designs (dark patterns) on data broker websites, finding that the agent could consistently navigate and classify dark patterns across 456 sites but also identified limitations in its reliability and failure conditions.
As LLM-driven agents begin to autonomously navigate the web, their ability to interpret and respond to manipulative interface design becomes critical. A fundamental question that emerges is: can such agents reliably recognize patterns of friction, misdirection, and coercion in interface design (i.e., dark patterns)? We study this question in a setting where the workflows are consequential: website portals associated with the submission of CCPA-related data rights requests. These portals operationalize statutory rights, but they are implemented as interactive interfaces whose design can be structured to facilitate, burden, or subtly discourage the exercise of those rights. We design and deploy an LLM-driven auditing agent capable of end-to-end traversal of rights-request workflows, structured evidence gathering, and classification of potential dark patterns. Across a set of 456 data broker websites, we evaluate: (1) the ability of the agent to consistently locate and complete request flows, (2) the reliability and reproducibility of its dark pattern classifications, and (3) the conditions under which it fails or produces poor judgments. Our findings characterize both the feasibility and the limitations of using LLM-driven agents for scalable dark pattern auditing.