Mapping the Stochastic Penal Colony
For researchers and critics of content moderation, this paper offers a new theoretical lens and methodology to analyze punitive moderation practices, though it is primarily conceptual and qualitative.
This paper reframes content moderation as a 'stochastic penal colony' using Foucault's penal theory, develops a novel auto-ethnographic and procedural justice methodology, and applies it to three case studies (pre-Musk Twitter, OpenAI's DALL-E 2, Pinterest). It finds that all three platforms use the threat of account suspension to banish users to this punitive space.
With peak content moderation seemingly behind us, this paper revisits its punitive side. But instead of focusing on who is being (disproportionately) moderated, it focuses on the punishment itself and explores the question of how content moderation treats users posting violative content unjustly, while the organizations doing the moderation act in a self-serving manner. First, this paper reworks Foucault's model of the penal system for the algorithmic age, restoring the penal colony as a figuratively liminal practice between punishment as performance and punishment as discipline, i.e., the stochastic penal colony. Second, it develops a novel methodology that combines auto-ethnography for collecting experiences and artifacts with procedural justice for analyzing them. Third, it applies this conceptual and methodological framing to three case studies, one on pre-Musk Twitter's gallingly performative moderation, one on OpenAI's exhaustively controlling moderation for DALL-E 2, and one on Pinterest's underhandedly manipulative moderation. While substantially different, all three feature the pervasive threat of account suspension, which banishes users to the stochastic penal colony.