Measuring Bias in AI Models: An Statistical Approach Introducing N-Sigma
This work addresses bias detection for regulatory compliance in AI, particularly under the European Commission's risk-based framework, but it is incremental as it applies an existing statistical method to a new domain.
The paper tackles the problem of measuring bias in AI models, specifically in face recognition technologies, by proposing a novel statistical approach based on the N-Sigma method, which is adapted from general science to machine learning for the first time.
The new regulatory framework proposal on Artificial Intelligence (AI) published by the European Commission establishes a new risk-based legal approach. The proposal highlights the need to develop adequate risk assessments for the different uses of AI. This risk assessment should address, among others, the detection and mitigation of bias in AI. In this work we analyze statistical approaches to measure biases in automatic decision-making systems. We focus our experiments in face recognition technologies. We propose a novel way to measure the biases in machine learning models using a statistical approach based on the N-Sigma method. N-Sigma is a popular statistical approach used to validate hypotheses in general science such as physics and social areas and its application to machine learning is yet unexplored. In this work we study how to apply this methodology to develop new risk assessment frameworks based on bias analysis and we discuss the main advantages and drawbacks with respect to other popular statistical tests.