Entity-Based Evaluation of Political Bias in Automatic Summarization
This work addresses the issue of political bias in NLP systems for researchers and practitioners, though it is incremental as it builds on existing bias studies.
The paper tackled the problem of political bias in automatic summarization by evaluating how summarization models portray politicians like Donald Trump and Joe Biden, finding consistent differences such as reduced emphasis on Trump's presence and more individualistic representation in summaries.
Growing literature has shown that NLP systems may encode social biases; however, the political bias of summarization models remains relatively unknown. In this work, we use an entity replacement method to investigate the portrayal of politicians in automatically generated summaries of news articles. We develop an entity-based computational framework to assess the sensitivities of several extractive and abstractive summarizers to the politicians Donald Trump and Joe Biden. We find consistent differences in these summaries upon entity replacement, such as reduced emphasis of Trump's presence in the context of the same article and a more individualistic representation of Trump with respect to the collective US government (i.e., administration). These summary dissimilarities are most prominent when the entity is heavily featured in the source article. Our characterization provides a foundation for future studies of bias in summarization and for normative discussions on the ideal qualities of automatic summaries.