Towards Leveraging News Media to Support Impact Assessment of AI Technologies
This addresses the need for more inclusive impact assessments of AI technologies for policymakers and the public, though it is incremental as it builds on existing LLM fine-tuning methods.
The study tackled the problem of expert-driven impact assessments overlooking diverse public effects of AI by fine-tuning LLMs on news articles from 266 domains across 30 countries, resulting in high-quality negative impact generation across coherence, structure, relevance, and plausibility, and capturing a wider range of impact categories than GPT-4.
Expert-driven frameworks for impact assessments (IAs) may inadvertently overlook the effects of AI technologies on the public's social behavior, policy, and the cultural and geographical contexts shaping the perception of AI and the impacts around its use. This research explores the potentials of fine-tuning LLMs on negative impacts of AI reported in a diverse sample of articles from 266 news domains spanning 30 countries around the world to incorporate more diversity into IAs. Our findings highlight (1) the potential of fine-tuned open-source LLMs in supporting IA of AI technologies by generating high-quality negative impacts across four qualitative dimensions: coherence, structure, relevance, and plausibility, and (2) the efficacy of small open-source LLM (Mistral-7B) fine-tuned on impacts from news media in capturing a wider range of categories of impacts that GPT-4 had gaps in covering.