The Bidirectional Relationship Between XAI and Regulation: Operationalizing XAI for the AI Act
This work addresses the problem of regulatory compliance for high-risk AI systems, particularly for human-centered XAI practitioners, by providing an interdisciplinary analysis that bridges legal and technical domains, though it is incremental in applying existing XAI concepts to new regulatory contexts.
The paper tackles the challenge of aligning explainable AI (XAI) methods with the EU AI Act's regulatory requirements, analyzing a real-world clinical decision support system to map XAI stakeholder roles to legal responsibilities and identify gaps where additional measures are needed.
The EU AI Act makes explainability urgent for high-risk AI systems, yet most XAI research focuses on technical metrics rather than regulatory compliance. Understanding how legal requirements reshape XAI method design is challenging: the AI Act regulates organizational relationships (providers, deployers) using legal terminology, specifies obligations without concrete technical requirements, and underrepresents end-users--the very stakeholders whose needs human-centered XAI addresses. As regulations emerge globally, human-centered XAI practitioners face both a challenge and an opportunity: regulations pull XAI research toward real-world deployment, while practitioners can actively shape how explainability enables compliance. This establishes a bidirectional relationship. Our contribution is threefold. First, we provide the first interdisciplinary analysis of XAI's role in the AI Act--conducted by a team comprising AI Act legal experts, ML engineers, and requirements engineers--on a real-world clinical decision support system. Second, we systematically align XAI stakeholder roles with AI Act legal responsibilities, revealing where explainability methods address regulatory requirements versus where additional measures are necessary. Third, we identify three key opportunities for human-centered XAI practitioners: actively defining their roles in regulatory implementation; making the user-to-affected-party relationship explicit where regulations address only provider-deployer obligations; and enabling compliance while building multi-level trust--from regulators to affected parties.