Seamless Deception: Larger Language Models Are Better Knowledge Concealers
This exposes a key limitation in black-box auditing of language models, highlighting the need for robust detection methods to address deception in AI systems.
The study tackled the problem of detecting when language models conceal harmful knowledge, finding that classifiers can detect concealment in smaller models but fail on larger models exceeding 70 billion parameters, achieving no better than random performance.
Language Models (LMs) may acquire harmful knowledge, and yet feign ignorance of these topics when under audit. Inspired by the recent discovery of deception-related behaviour patterns in LMs, we aim to train classifiers that detect when a LM is actively concealing knowledge. Initial findings on smaller models show that classifiers can detect concealment more reliably than human evaluators, with gradient-based concealment proving easier to identify than prompt-based methods. However, contrary to prior work, we find that the classifiers do not reliably generalize to unseen model architectures and topics of hidden knowledge. Most concerningly, the identifiable traces associated with concealment become fainter as the models increase in scale, with the classifiers achieving no better than random performance on any model exceeding 70 billion parameters. Our results expose a key limitation in black-box-only auditing of LMs and highlight the need to develop robust methods to detect models that are actively hiding the knowledge they contain.