Seeing in Words: Learning to Classify through Language Bottlenecks
This addresses the need for explainability in AI for computer vision users, though it appears incremental as it builds on existing vision models.
The paper tackles the problem of uninterpretable features in neural networks for computer vision by training a model that uses text as feature representations, achieving effective classification on ImageNet images.
Neural networks for computer vision extract uninterpretable features despite achieving high accuracy on benchmarks. In contrast, humans can explain their predictions using succinct and intuitive descriptions. To incorporate explainability into neural networks, we train a vision model whose feature representations are text. We show that such a model can effectively classify ImageNet images, and we discuss the challenges we encountered when training it.