Identifying and Mitigating Model Failures through Few-shot CLIP-aided
  Diffusion Generation
Atoosa Chegini Soheil Feizi
Deep learning models can encounter unexpected failures, especially when
dealing with challenging sub-populations. One common reason for these failures
is the occurrence of objects in backgrounds that are rarely seen during
training. To gain a better understanding of these failure modes,
human-interpretable descriptions are crucial for further analysis and
improvement which is expensive. In this study, we propose an end-to-end
framework that utilizes the capabilities of large language models (ChatGPT) and
vision-language deep models (CLIP) to generate text descriptions of failure
modes associated with spurious correlations (e.g. rarely seen backgrounds)
without human-in-the-loop intervention. These descriptions can be used to
generate synthetic data using generative models, such as diffusion models. The
model can now use this generated data to learn from its weaknesses and enhance
its performance on backgrounds that are uncommon for each class of data. Our
approach serves as a broad solution, promising progress in comprehending model
failure modes and strengthening deep learning models across a wide range of
failure scenarios (e.g. bacckgrounds, colors) automatically in a few-shot
manner. Our experiments have shown remarkable \textbf{improvements in accuracy
($\sim \textbf{21%}$)} on hard sub-populations (particularly for wrong
background association) across $40$ different models, such as ResNets,
EfficientNets, DenseNets, Vision Transformer (ViT), SwAVs, MoCos, DINOs, and
CLIPs on various datasets such as ImageNet-1000, CIFAR-10, and CIFAR-100.