PAGAN: Video Affect Annotation Made Easy
This addresses the need for scalable and accessible affect annotation tools for researchers, though it is incremental as it builds on existing annotation methods.
The paper tackles the problem of gathering reliable affect annotations for videos by introducing PAGAN, an accessible online crowdsourcing platform, and finds that processing annotations relatively and using unbounded labelling yields higher inter-rater agreement.
How could we gather affect annotations in a rapid, unobtrusive, and accessible fashion? How could we still make sure that these annotations are reliable enough for data-hungry affect modelling methods? This paper addresses these questions by introducing PAGAN, an accessible, general-purpose, online platform for crowdsourcing affect labels in videos. The design of PAGAN overcomes the accessibility limitations of existing annotation tools, which often require advanced technical skills or even the on-site involvement of the researcher. Such limitations often yield affective corpora that are restricted in size, scope and use, as the applicability of modern data-demanding machine learning methods is rather limited. The description of PAGAN is accompanied by an exploratory study which compares the reliability of three continuous annotation tools currently supported by the platform. Our key results reveal higher inter-rater agreement when annotation traces are processed in a relative manner and collected via unbounded labelling.