Seeing in Words: Learning to Classify through Language Bottlenecks
Khalid Saifullah Yuxin Wen Jonas Geiping Micah Goldblum Tom Goldstein
Neural networks for computer vision extract uninterpretable features despite
achieving high accuracy on benchmarks. In contrast, humans can explain their
predictions using succinct and intuitive descriptions. To incorporate
explainability into neural networks, we train a vision model whose feature
representations are text. We show that such a model can effectively classify
ImageNet images, and we discuss the challenges we encountered when training it.