Analyzing the Granularity and Cost of Annotation in Clinical Sequence Labeling
This work addresses the cost-effectiveness of dataset annotation for researchers and practitioners in clinical sequence labeling, indicating an incremental finding that detailed annotations may not be necessary.
The study analyzed the relationship between annotation granularity and machine learning performance in clinical sequence labeling, finding that adding detailed manual annotations by a nurse did not significantly improve performance compared to using textual language features alone, suggesting low return on investment for fine-grained annotation.
Well-annotated datasets, as shown in recent top studies, are becoming more important for researchers than ever before in supervised machine learning (ML). However, the dataset annotation process and its related human labor costs remain overlooked. In this work, we analyze the relationship between the annotation granularity and ML performance in sequence labeling, using clinical records from nursing shift-change handover. We first study a model derived from textual language features alone, without additional information based on nursing knowledge. We find that this sequence tagger performs well in most categories under this granularity. Then, we further include the additional manual annotations by a nurse, and find the sequence tagging performance remaining nearly the same. Finally, we give a guideline and reference to the community arguing it is not necessary and even not recommended to annotate in detailed granularity because of a low Return on Investment. Therefore we recommend emphasizing other features, like textual knowledge, for researchers and practitioners as a cost-effective source for increasing the sequence labeling performance.