Exploring Fusion Techniques in Multimodal AI-Based Recruitment: Insights from FairCVdb
This work addresses fairness issues in recruitment AI for job applicants, but it is incremental as it applies known fusion methods to a specific dataset without introducing new paradigms.
The study tackled fairness and bias in multimodal AI-based recruitment systems by comparing early-fusion and late-fusion techniques on the FairCVdb dataset, finding that early-fusion achieved the lowest mean absolute errors and closely matched ground truth for demographics, while late-fusion resulted in higher errors and generalized scores.
Despite the large body of work on fairness-aware learning for individual modalities like tabular data, images, and text, less work has been done on multimodal data, which fuses various modalities for a comprehensive analysis. In this work, we investigate the fairness and bias implications of multimodal fusion techniques in the context of multimodal AI-based recruitment systems using the FairCVdb dataset. Our results show that early-fusion closely matches the ground truth for both demographics, achieving the lowest MAEs by integrating each modality's unique characteristics. In contrast, late-fusion leads to highly generalized mean scores and higher MAEs. Our findings emphasise the significant potential of early-fusion for accurate and fair applications, even in the presence of demographic biases, compared to late-fusion. Future research could explore alternative fusion strategies and incorporate modality-related fairness constraints to improve fairness. For code and additional insights, visit: https://github.com/Swati17293/Multimodal-AI-Based-Recruitment-FairCVdb