FairX: A comprehensive benchmarking tool for model analysis using fairness, utility, and explainability
This tool addresses the need for researchers and practitioners to analyze models holistically for fairness, utility, and explainability, though it is incremental as it builds on existing benchmarking approaches.
The authors tackled the lack of comprehensive benchmarking tools for fairness, utility, and explainability in models by developing FairX, an open-source Python tool that supports training and evaluation of bias-mitigation models, including fair generative models, across tabular and image datasets.
We present FairX, an open-source Python-based benchmarking tool designed for the comprehensive analysis of models under the umbrella of fairness, utility, and eXplainability (XAI). FairX enables users to train benchmarking bias-mitigation models and evaluate their fairness using a wide array of fairness metrics, data utility metrics, and generate explanations for model predictions, all within a unified framework. Existing benchmarking tools do not have the way to evaluate synthetic data generated from fair generative models, also they do not have the support for training fair generative models either. In FairX, we add fair generative models in the collection of our fair-model library (pre-processing, in-processing, post-processing) and evaluation metrics for evaluating the quality of synthetic fair data. This version of FairX supports both tabular and image datasets. It also allows users to provide their own custom datasets. The open-source FairX benchmarking package is publicly available at \url{https://github.com/fahim-sikder/FairX}.