Latent Distribution Assumption for Unbiased and Consistent Consensus Modelling
This addresses label ambiguity in noisy label aggregation, particularly for difficult tasks, and is incremental as it modifies an existing assumption.
The paper tackles the problem of aggregating noisy labels by introducing a latent distribution assumption, which replaces the traditional single true label per object with a distribution generating subjective labels, and shows that this approach is more suitable for ambiguous tasks.
We study the problem of aggregation noisy labels. Usually, it is solved by proposing a stochastic model for the process of generating noisy labels and then estimating the model parameters using the observed noisy labels. A traditional assumption underlying previously introduced generative models is that each object has one latent true label. In contrast, we introduce a novel latent distribution assumption, implying that a unique true label for an object might not exist, but rather each object might have a specific distribution generating a latent subjective label each time the object is observed. Our experiments showed that the novel assumption is more suitable for difficult tasks, when there is an ambiguity in choosing a "true" label for certain objects.