Mitigating Bias in Visual Transformers via Targeted Alignment
This addresses fairness issues in computer vision for users of transformer models, but it is incremental as it builds on prior bias mitigation approaches.
The study tackled bias in visual transformers by identifying that bias is encoded in the query matrix and proposing TADeT, a targeted alignment strategy, which improved fairness on CelebA dataset tasks without performance loss.
As transformer architectures become increasingly prevalent in computer vision, it is critical to understand their fairness implications. We perform the first study of the fairness of transformers applied to computer vision and benchmark several bias mitigation approaches from prior work. We visualize the feature space of the transformer self-attention modules and discover that a significant portion of the bias is encoded in the query matrix. With this knowledge, we propose TADeT, a targeted alignment strategy for debiasing transformers that aims to discover and remove bias primarily from query matrix features. We measure performance using Balanced Accuracy and Standard Accuracy, and fairness using Equalized Odds and Balanced Accuracy Difference. TADeT consistently leads to improved fairness over prior work on multiple attribute prediction tasks on the CelebA dataset, without compromising performance.