"I know it when I see it". Visualization and Intuitive Interpretability
This addresses interpretability issues for researchers and practitioners in machine learning, but it is incremental as it critiques existing approaches without new empirical results.
The paper argues that visualization, while enabling intuitive interpretability, also impedes it due to technical pre-interpretation and human bias from positive concepts, suggesting avoidance of singular representations for internal model states.
Most research on the interpretability of machine learning systems focuses on the development of a more rigorous notion of interpretability. I suggest that a better understanding of the deficiencies of the intuitive notion of interpretability is needed as well. I show that visualization enables but also impedes intuitive interpretability, as it presupposes two levels of technical pre-interpretation: dimensionality reduction and regularization. Furthermore, I argue that the use of positive concepts to emulate the distributed semantic structure of machine learning models introduces a significant human bias into the model. As a consequence, I suggest that, if intuitive interpretability is needed, singular representations of internal model states should be avoided.