Mechanistic Anomaly Detection for "Quirky" Language Models
This addresses supervision challenges for LLMs in low-stakes applications, but it is incremental as it highlights limitations and the need for further advances.
The paper tackled the problem of supervising large language models (LLMs) by using Mechanistic Anomaly Detection (MAD) to identify anomalous training signals based on internal model features, finding that detectors achieved high discrimination on some tasks but were not consistently effective across all models and tasks.
As LLMs grow in capability, the task of supervising LLMs becomes more challenging. Supervision failures can occur if LLMs are sensitive to factors that supervisors are unaware of. We investigate Mechanistic Anomaly Detection (MAD) as a technique to augment supervision of capable models; we use internal model features to identify anomalous training signals so they can be investigated or discarded. We train detectors to flag points from the test environment that differ substantially from the training environment, and experiment with a large variety of detector features and scoring rules to detect anomalies in a set of ``quirky'' language models. We find that detectors can achieve high discrimination on some tasks, but no detector is effective across all models and tasks. MAD techniques may be effective in low-stakes applications, but advances in both detection and evaluation are likely needed if they are to be used in high stakes settings.