An Empirical Study of Uncertainty in Polygon Annotation and the Impact of Quality Assurance
This addresses annotation quality issues for practitioners in computer vision, but is incremental as it builds on existing datasets and methods.
The study quantified the inherent uncertainty in polygon annotations for instance segmentation, finding that annotation reliability depends on reviewing procedures and scene/shape complexity.
Polygons are a common annotation format used for quickly annotating objects in instance segmentation tasks. However, many real-world annotation projects request near pixel-perfect labels. While strict pixel guidelines may appear to be the solution to a successful project, practitioners often fail to assess the feasibility of the work requested, and overlook common factors that may challenge the notion of quality. This paper aims to examine and quantify the inherent uncertainty for polygon annotations and the role that quality assurance plays in minimizing its effect. To this end, we conduct an analysis on multi-rater polygon annotations for several objects from the MS-COCO dataset. The results demonstrate that the reliability of a polygon annotation is dependent on a reviewing procedure, as well as the scene and shape complexity.