White-Box Sensitivity Auditing with Steering Vectors
This work addresses the problem of more rigorous algorithmic audits for LLMs, particularly for regulators and operators concerned with bias, by introducing a white-box method that overcomes the abstract nature of socially relevant properties.
The authors tackled the limitations of black-box evaluations in auditing large language models (LLMs) for properties like bias by proposing a white-box sensitivity auditing framework that uses activation steering to assess model internals. They demonstrated its application in four simulated high-stakes decision tasks, revealing substantial dependence on protected attributes where standard methods suggested little or no bias.
Algorithmic audits are essential tools for examining systems for properties required by regulators or desired by operators. Current audits of large language models (LLMs) primarily rely on black-box evaluations that assess model behavior only through input-output testing. These methods are limited to tests constructed in the input space, often generated by heuristics. In addition, many socially relevant model properties (e.g., gender bias) are abstract and difficult to measure through text-based inputs alone. To address these limitations, we propose a white-box sensitivity auditing framework for LLMs that leverages activation steering to conduct more rigorous assessments through model internals. Our auditing method conducts internal sensitivity tests by manipulating key concepts relevant to the model's intended function for the task. We demonstrate its application to bias audits in four simulated high-stakes LLM decision tasks. Our method consistently reveals substantial dependence on protected attributes in model predictions, even in settings where standard black-box evaluations suggest little or no bias. Our code is openly available at https://github.com/hannahxchen/llm-steering-audit