Consistency and Accuracy of CelebA Attribute Values
This work addresses data quality issues in a widely used facial attribute dataset, which is crucial for researchers and practitioners in computer vision and machine learning, though it is incremental as it focuses on analysis and correction rather than new methods.
The study analyzed the consistency and accuracy of attribute values in the CelebA dataset, finding that only 12 of 40 attributes had high consistency, with error rates as high as 40% for some attributes, and correcting the 'mouth slightly open' attribute enabled a model to achieve higher accuracy than previously reported.
We report the first systematic analysis of the experimental foundations of facial attribute classification. Two annotators independently assigning attribute values shows that only 12 of 40 common attributes are assigned values with >= 95% consistency, and three (high cheekbones, pointed nose, oval face) have essentially random consistency. Of 5,068 duplicate face appearances in CelebA, attributes have contradicting values on from 10 to 860 of the 5,068 duplicates. Manual audit of a subset of CelebA estimates error rates as high as 40% for (no beard=false), even though the labeling consistency experiment indicates that no beard could be assigned with >= 95% consistency. Selecting the mouth slightly open (MSO) for deeper analysis, we estimate the error rate for (MSO=true) at about 20% and (MSO=false) at about 2%. A corrected version of the MSO attribute values enables learning a model that achieves higher accuracy than previously reported for MSO. Corrected values for CelebA MSO are available at https://github.com/HaiyuWu/CelebAMSO.