Mitigating fairwashing using Two-Source Audits
This addresses the need for reliable audits in manipulation-prone setups for regulators and auditors, though it is incremental as it builds on prior work with partial knowledge assumptions.
The paper tackles the problem of fairwashing, where online platforms manipulate audits of their decision-making algorithms, by proposing Two-Source Audits that use an API and a second data source to detect discrepancies. The method is validated empirically, achieving a Pareto-optimal balance between audit and detection objectives.
Recent legislation requires online platforms to provide dedicated APIs to assess the compliance of their decision-making algorithms with the law. Research has nevertheless shown that the auditors of such platforms are prone to manipulation (a practice referred to as \textit{fairwashing}). To address this salient problem, recent work has considered audits under the assumption of partial knowledge of the platform's internal mechanisms. In this paper, we propose a more pragmatic approach with the \textit{Two-Source Audit} setup: while still leveraging the API, we advocate for the adjunction of a second source of data to both perform the audit of a platform and the detection of fairwashing attempts. Our method is based on identifying discrepancies between the two data sources, using data proxies at use in the fairness literature. We formally demonstrate the conditions for success in this fairwashing mitigation task. We then validate our method empirically, demonstrating that Two-Source Audits can achieve a Pareto-optimal balance between the two objectives. We believe this paper sets the stage for reliable audits in manipulation-prone setups, under mild assumptions.