Accounting for AI and Users Shaping One Another: The Role of Mathematical Models
This addresses the problem of AI systems not accounting for user interactions in societal domains, but it is a position paper with incremental theoretical contributions.
The paper argues for developing formal mathematical models to account for how AI systems and users mutually shape each other, proposing these models for implementation, monitoring, anticipation, and control of societal impacts, using content recommender systems as a case study.
As AI systems enter into a growing number of societal domains, these systems increasingly shape and are shaped by user preferences, opinions, and behaviors. However, the design of AI systems rarely accounts for how AI and users shape one another. In this position paper, we argue for the development of formal interaction models which mathematically specify how AI and users shape one another. Formal interaction models can be leveraged to (1) specify interactions for implementation, (2) monitor interactions through empirical analysis, (3) anticipate societal impacts via counterfactual analysis, and (4) control societal impacts via interventions. The design space of formal interaction models is vast, and model design requires careful consideration of factors such as style, granularity, mathematical complexity, and measurability. Using content recommender systems as a case study, we critically examine the nascent literature of formal interaction models with respect to these use-cases and design axes. More broadly, we call for the community to leverage formal interaction models when designing, evaluating, or auditing any AI system which interacts with users.