Benchmarking Deception Probes via Black-to-White Performance Boosts
This work addresses the challenge of ensuring AI safety by assessing deception detection methods, though it is incremental as it benchmarks existing probes.
The paper tackled the problem of evaluating how effectively deception probes detect AI assistant deception by comparing white-box and black-box monitoring, finding weak but encouraging performance boosts.
AI assistants will occasionally respond deceptively to user queries. Recently, linear classifiers (called "deception probes") have been trained to distinguish the internal activations of a language model during deceptive versus honest responses. However, it's unclear how effective these probes are at detecting deception in practice, nor whether such probes are resistant to simple counter strategies from a deceptive assistant who wishes to evade detection. In this paper, we compare white-box monitoring (where the monitor has access to token-level probe activations) to black-box monitoring (without such access). We benchmark deception probes by the extent to which the white box monitor outperforms the black-box monitor, i.e. the black-to-white performance boost. We find weak but encouraging black-to-white performance boosts from existing deception probes.