Catching Dark Signals in Algorithms: Unveiling Audiovisual and Thematic Markers of Unsafe Content Recommended for Children and Teenagers
This work addresses the problem of algorithmic exposure to harmful content for children and teenagers on short-form video platforms, offering a framework to categorize online harms.
The study analyzed 4,492 short videos recommended to children and teenagers on Instagram Reels, TikTok, and YouTube Shorts, finding that unsafe content exhibits darker visual features and contains both explicit harmful content and implicit harm from anxiety-inducing ordinary content. The authors propose a framework of online harm (explicit, implicit, unintended) to understand risks to younger audiences.
The prevalence of short form video platforms, combined with the ineffectiveness of age verification mechanisms, raises concerns about the potential harms facing children and teenagers in an algorithm-moderated online environment. We conducted multimodal feature analysis and thematic topic modeling of 4,492 short videos recommended to children and teenagers on Instagram Reels, TikTok, and YouTube Shorts, collected as a part of an algorithm auditing experiment. This feature-level and content-level analysis revealed that unsafe (i.e., problematic, mentally distressing) short videos (a) possess darker visual features and (b) contain explicitly harmful content and implicit harm from anxiety-inducing ordinary content. We introduce a useful framework of online harm (i.e., explicit, implicit, unintended), providing a unique lens for understanding the dynamic, multifaceted online risks facing children and teenagers. The findings highlight the importance of protecting younger audiences in critical developmental stages from both explicit and implicit risks on social media, calling for nuanced content moderation, age verification, and platform regulation.