Obtaining Faithful Interpretations from Compositional Neural Networks
This addresses the reliability of interpretability in compositional AI models, which is crucial for trust in applications like language and vision, though it is incremental as it builds on existing NMN frameworks.
The authors tackled the problem that neural module networks (NMNs) may not faithfully explain model reasoning, as intermediate outputs often deviate from expected behavior on NLVR2 and DROP datasets. They improved faithfulness with auxiliary supervision and architectural changes, achieving minimal accuracy loss.
Neural module networks (NMNs) are a popular approach for modeling compositionality: they achieve high accuracy when applied to problems in language and vision, while reflecting the compositional structure of the problem in the network architecture. However, prior work implicitly assumed that the structure of the network modules, describing the abstract reasoning process, provides a faithful explanation of the model's reasoning; that is, that all modules perform their intended behaviour. In this work, we propose and conduct a systematic evaluation of the intermediate outputs of NMNs on NLVR2 and DROP, two datasets which require composing multiple reasoning steps. We find that the intermediate outputs differ from the expected output, illustrating that the network structure does not provide a faithful explanation of model behaviour. To remedy that, we train the model with auxiliary supervision and propose particular choices for module architecture that yield much better faithfulness, at a minimal cost to accuracy.