PRISM: A Methodology for Auditing Biases in Large Language Models
This addresses the problem of detecting hidden biases in LLMs for AI safety and ethics researchers, though it is incremental as it builds on prior work with a new prompting approach.
The paper tackles the challenge of auditing biases in Large Language Models (LLMs) by introducing PRISM, a methodology that uses task-based inquiry prompting to indirectly elicit model positions, and applies it to assess political leanings across 21 LLMs, finding they generally espouse economically left and socially liberal views.
Auditing Large Language Models (LLMs) to discover their biases and preferences is an emerging challenge in creating Responsible Artificial Intelligence (AI). While various methods have been proposed to elicit the preferences of such models, countermeasures have been taken by LLM trainers, such that LLMs hide, obfuscate or point blank refuse to disclosure their positions on certain subjects. This paper presents PRISM, a flexible, inquiry-based methodology for auditing LLMs - that seeks to illicit such positions indirectly through task-based inquiry prompting rather than direct inquiry of said preferences. To demonstrate the utility of the methodology, we applied PRISM on the Political Compass Test, where we assessed the political leanings of twenty-one LLMs from seven providers. We show LLMs, by default, espouse positions that are economically left and socially liberal (consistent with prior work). We also show the space of positions that these models are willing to espouse - where some models are more constrained and less compliant than others - while others are more neutral and objective. In sum, PRISM can more reliably probe and audit LLMs to understand their preferences, biases and constraints.