ERIC: Extracting Relations Inferred from Convolutions
This work addresses the interpretability problem for CNN users by providing a method to extract logical rules, though it is incremental as it builds on existing kernel analysis techniques.
The researchers tackled the problem of interpreting convolutional neural networks by approximating kernel behavior across multiple layers with logic programs, achieving extracted program accuracies that correlate with the original model's performance, though with some information loss when chaining approximations or quantizing lower layers.
Our main contribution is to show that the behaviour of kernels across multiple layers of a convolutional neural network can be approximated using a logic program. The extracted logic programs yield accuracies that correlate with those of the original model, though with some information loss in particular as approximations of multiple layers are chained together or as lower layers are quantised. We also show that an extracted program can be used as a framework for further understanding the behaviour of CNNs. Specifically, it can be used to identify key kernels worthy of deeper inspection and also identify relationships with other kernels in the form of the logical rules. Finally, we make a preliminary, qualitative assessment of rules we extract from the last convolutional layer and show that kernels identified are symbolic in that they react strongly to sets of similar images that effectively divide output classes into sub-classes with distinct characteristics.