Towards More Accountable Search Engines: Online Evaluation of Representation Bias
This addresses the risk of biased search results harming users and society, though it is incremental as it builds on existing bias evaluation efforts.
The paper tackles the problem of search engines lacking accountability for biased information by proposing a method to measure representation bias, specifically applying it to evaluate gender bias in Google's Knowledge Graph Carousel for occupations.
Information availability affects people's behavior and perception of the world. Notably, people rely on search engines to satisfy their need for information. Search engines deliver results relevant to user requests usually without being or making themselves accountable for the information they deliver, which may harm people's lives and, in turn, society. This potential risk urges the development of evaluation mechanisms of bias in order to empower the user in judging the results of search engines. In this paper, we give a possible solution to measuring representation bias with respect to societal features for search engines and apply it to evaluating the gender representation bias for Google's Knowledge Graph Carousel for listing occupations.