Easing Optimization Paths: a Circuit Perspective
This work addresses optimization challenges for training large-scale AI models, though it appears incremental as it builds on existing interpretability methods.
The paper tackles the problem of high compute costs and harmful behaviors in large AI systems by applying a circuit perspective from mechanistic interpretability to design a curriculum for efficient learning in a controlled setting.
Gradient descent is the method of choice for training large artificial intelligence systems. As these systems become larger, a better understanding of the mechanisms behind gradient training would allow us to alleviate compute costs and help steer these systems away from harmful behaviors. To that end, we suggest utilizing the circuit perspective brought forward by mechanistic interpretability. After laying out our intuition, we illustrate how it enables us to design a curriculum for efficient learning in a controlled setting. The code is available at \url{https://github.com/facebookresearch/pal}.