A Guide for Manual Annotation of Scientific Imagery: How to Prepare for Large Projects
This addresses the problem of scarce practical guidelines for managing annotation projects, which is incremental as it synthesizes existing knowledge into a guide.
The paper tackles the challenge of managing complex and costly manual annotation projects for scientific imagery by providing a domain-agnostic preparation guide based on extensive experience, aiming to reduce costs and encourage further research.
Despite the high demand for manually annotated image data, managing complex and costly annotation projects remains under-discussed. This is partly due to the fact that leading such projects requires dealing with a set of diverse and interconnected challenges which often fall outside the expertise of specific domain experts, leaving practical guidelines scarce. These challenges range widely from data collection to resource allocation and recruitment, from mitigation of biases to effective training of the annotators. This paper provides a domain-agnostic preparation guide for annotation projects, with a focus on scientific imagery. Drawing from the authors' extensive experience in managing a large manual annotation project, it addresses fundamental concepts including success measures, annotation subjects, project goals, data availability, and essential team roles. Additionally, it discusses various human biases and recommends tools and technologies to improve annotation quality and efficiency. The goal is to encourage further research and frameworks for creating a comprehensive knowledge base to reduce the costs of manual annotation projects across various fields.