The Extended Dawid-Skene Model: Fusing Information from Multiple Data Schemas
This addresses a practical data wrangling issue for researchers and practitioners dealing with heterogeneous annotation schemas, offering an incremental extension to the Dawid-Skene model.
The paper tackles the problem of learning from multiple noisy annotations when annotators use different label schemas, proposing the Inter-Schema AdapteR (ISAR) to translate labels without re-annotation. It achieves significant gains on a mouse behavioral dataset, improving out-of-sample log-likelihood from -3.40 to -2.39 and F1-score from 0.785 to 0.864.
While label fusion from multiple noisy annotations is a well understood concept in data wrangling (tackled for example by the Dawid-Skene (DS) model), we consider the extended problem of carrying out learning when the labels themselves are not consistently annotated with the same schema. We show that even if annotators use disparate, albeit related, label-sets, we can still draw inferences for the underlying full label-set. We propose the Inter-Schema AdapteR (ISAR) to translate the fully-specified label-set to the one used by each annotator, enabling learning under such heterogeneous schemas, without the need to re-annotate the data. We apply our method to a mouse behavioural dataset, achieving significant gains (compared with DS) in out-of-sample log-likelihood (-3.40 to -2.39) and F1-score (0.785 to 0.864).