Quantifying the amount of visual information used by neural caption generators
This work addresses the need for explainability in AI, specifically for image captioning systems, but is incremental as it focuses on analysis rather than new methods.
The paper tackled the problem of understanding how much visual information neural image caption generators actually use, finding that their sensitivity to visual input varies depending on word type and caption position.
This paper addresses the sensitivity of neural image caption generators to their visual input. A sensitivity analysis and omission analysis based on image foils is reported, showing that the extent to which image captioning architectures retain and are sensitive to visual information varies depending on the type of word being generated and the position in the caption as a whole. We motivate this work in the context of broader goals in the field to achieve more explainability in AI.