Adversarial Circuit Evaluation
This work addresses the reliability of interpretability methods for neural networks, particularly for safety-critical applications, though it is incremental as it evaluates existing circuits rather than proposing new ones.
The researchers evaluated three published neural network circuits (IOI, greater-than, and docstring) by testing them adversarially on inputs where circuit behavior diverges most from the full model, measuring KL divergence via resample ablation. They found that circuits for IOI and docstring tasks failed to match full model behavior even on benign inputs, highlighting the need for more robust circuits in safety-critical applications.
Circuits are supposed to accurately describe how a neural network performs a specific task, but do they really? We evaluate three circuits found in the literature (IOI, greater-than, and docstring) in an adversarial manner, considering inputs where the circuit's behavior maximally diverges from the full model. Concretely, we measure the KL divergence between the full model's output and the circuit's output, calculated through resample ablation, and we analyze the worst-performing inputs. Our results show that the circuits for the IOI and docstring tasks fail to behave similarly to the full model even on completely benign inputs from the original task, indicating that more robust circuits are needed for safety-critical applications.