System Safety Engineering for Social and Ethical ML Risks: A Case Study
This work addresses social and ethical risks in high-consequence ML systems for policymakers and engineers, but it is incremental as it adapts an existing safety engineering method to a new domain.
The paper tackles the disjointed and ad-hoc approaches to mitigating social and ethical risks in ML-driven systems by applying Systems Theoretic Process Analysis (STPA) to Prescription Drug Monitoring Programs, resulting in the identification of such risks and development of design-level controls.
Governments, industry, and academia have undertaken efforts to identify and mitigate harms in ML-driven systems, with a particular focus on social and ethical risks of ML components in complex sociotechnical systems. However, existing approaches are largely disjointed, ad-hoc and of unknown effectiveness. Systems safety engineering is a well established discipline with a track record of identifying and managing risks in many complex sociotechnical domains. We adopt the natural hypothesis that tools from this domain could serve to enhance risk analyses of ML in its context of use. To test this hypothesis, we apply a "best of breed" systems safety analysis, Systems Theoretic Process Analysis (STPA), to a specific high-consequence system with an important ML-driven component, namely the Prescription Drug Monitoring Programs (PDMPs) operated by many US States, several of which rely on an ML-derived risk score. We focus in particular on how this analysis can extend to identifying social and ethical risks and developing concrete design-level controls to mitigate them.