Spuriosity Rankings for Free: A Simple Framework for Last Layer
  Retraining Based on Object Detection
Mohammad Azizmalayeri Reza Abbasi Amir Hosein Haji Mohammad rezaie Reihaneh Zohrabi Mahdi Amiri Mohammad Taghi Manzuri Mohammad Hossein Rohban
Deep neural networks have exhibited remarkable performance in various
domains. However, the reliance of these models on spurious features has raised
concerns about their reliability. A promising solution to this problem is
last-layer retraining, which involves retraining the linear classifier head on
a small subset of data without spurious cues. Nevertheless, selecting this
subset requires human supervision, which reduces its scalability. Moreover,
spurious cues may still exist in the selected subset. As a solution to this
problem, we propose a novel ranking framework that leverages an open vocabulary
object detection technique to identify images without spurious cues. More
specifically, we use the object detector as a measure to score the presence of
the target object in the images. Next, the images are sorted based on this
score, and the last-layer of the model is retrained on a subset of the data
with the highest scores. Our experiments on the ImageNet-1k dataset demonstrate
the effectiveness of this ranking framework in sorting images based on
spuriousness and using them for last-layer retraining.