Provocation: Who benefits from "inclusion" in Generative AI?
This work critiques inclusion practices in AI for marginalized groups, highlighting a gap in current approaches without proposing a new technical solution, making it incremental in terms of methodology.
The paper argues that current participatory evaluation structures in generative AI development fail to explicitly address the benefits and harms for socially marginalized groups, potentially overlooking the need for systemic change. It uses a speculative case study to identify barriers that must be overcome to realize benefits and mitigate harms for these communities.
The demands for accurate and representative generative AI systems means there is an increased demand on participatory evaluation structures. While these participatory structures are paramount to to ensure non-dominant values, knowledge and material culture are also reflected in AI models and the media they generate, we argue that dominant structures of community participation in AI development and evaluation are not explicit enough about the benefits and harms that members of socially marginalized groups may experience as a result of their participation. Without explicit interrogation of these benefits by AI developers, as a community we may remain blind to the immensity of systemic change that is needed as well. To support this provocation, we present a speculative case study, developed from our own collective experiences as AI researchers. We use this speculative context to itemize the barriers that need to be overcome in order for the proposed benefits to marginalized communities to be realized, and harms mitigated.