Andreas Bjerre-Nielsen

Terne Sasha Thorn Jakobsen, Andreas Bjerre-Nielsen, Robert Böhm

Crowdsourced annotations of data play a substantial role in the development of Artificial Intelligence (AI). It is broadly recognised that annotations of text data can contain annotator bias, where systematic disagreement in annotations can be traced back to differences in the annotators' backgrounds. Being unaware of such annotator bias can lead to representational bias against minority group perspectives and therefore several methods have been proposed for recognising bias or preserving perspectives. These methods typically require either a substantial number of annotators or annotations per data instance. In this study, we propose a simple method for handling bias in annotations without requirements on the number of annotators or instances. Instead, we ask annotators about their beliefs of other annotators' judgements of an instance, under the hypothesis that these beliefs may provide more representative and less biased labels than judgements. The method was examined in two controlled, survey-based experiments involving Democrats and Republicans (n=1,590) asked to judge statements as arguments and then report beliefs about others' judgements. The results indicate that bias, defined as systematic differences between the two groups of annotators, is consistently reduced when asking for beliefs instead of judgements. Our proposed method therefore has the potential to reduce the risk of annotator bias, thereby improving the generalisability of AI systems and preventing harm to unrepresented socio-demographic groups, and we highlight the need for further studies of this potential in other tasks and downstream applications.

Andreas Bjerre-Nielsen

People choose friendships with people similar to themselves, i.e. they sort by resemblence. Economic studies have shown when sorting is optimal and constitute an equilibrium, however, this presumes lack of beneficial spillovers. We investigate formation of economic and social networks where agents may form or cut ties. We combine a setup with link formation where agents have types that determine the value of a connection. We provide conditions for sorting in friendships, i.e. that agents tend to partner only with those with those sufficiently similar to themselves. Conditions are provided with and without beneficial spillovers from indirect connections. We show that sorting may be suboptimal, yet a socially stable outcome, despite otherwise obeying the conditions for sorting in Becker (1973). We analyze policy tools to mitigate suboptimal sorting. Another feature is that agents with higher value are more central in networks under certain conditions; a side effect is sorting by degree centrality under certain conditions. Finally we illustrate the limits to patterns of sorting and centrality.