A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
This work addresses the need for scalable harm measurement in AI systems, particularly for developers and researchers focusing on responsible AI, though it builds incrementally on existing expertise.
The authors tackled the problem of measuring responsible AI harms in generative AI applications by developing an automated framework that leverages state-of-the-art LLMs like GPT-4, and they applied it in case studies to assess violations of RAI principles.
We present a framework for the automated measurement of responsible AI (RAI) metrics for large language models (LLMs) and associated products and services. Our framework for automatically measuring harms from LLMs builds on existing technical and sociotechnical expertise and leverages the capabilities of state-of-the-art LLMs, such as GPT-4. We use this framework to run through several case studies investigating how different LLMs may violate a range of RAI-related principles. The framework may be employed alongside domain-specific sociotechnical expertise to create measurements for new harm areas in the future. By implementing this framework, we aim to enable more advanced harm measurement efforts and further the responsible use of LLMs.