Probe-Based Interventions for Modifying Agent Behavior
This work addresses the need for people to influence neural agents' actions post-training, which is an incremental advancement building on prior explainability techniques.
The paper tackles the problem of modifying the behavior of pre-trained neural networks without retraining, formalizing it as a human-assisted decision-making challenge, and demonstrates that their method improves human-agent team performance across various neural networks, including image classifiers and multi-agent reinforcement learning agents.
Neural nets are powerful function approximators, but the behavior of a given neural net, once trained, cannot be easily modified. We wish, however, for people to be able to influence neural agents' actions despite the agents never training with humans, which we formalize as a human-assisted decision-making problem. Inspired by prior art initially developed for model explainability, we develop a method for updating representations in pre-trained neural nets according to externally-specified properties. In experiments, we show how our method may be used to improve human-agent team performance for a variety of neural networks from image classifiers to agents in multi-agent reinforcement learning settings.