Addressing Weak Decision Boundaries in Image Classification by
  Leveraging Web Search and Generative Models
Preetam Prabhu Srikar Dammu Yunhe Feng Chirag Shah
Machine learning (ML) technologies are known to be riddled with ethical and
operational problems, however, we are witnessing an increasing thrust by
businesses to deploy them in sensitive applications. One major issue among many
is that ML models do not perform equally well for underrepresented groups. This
puts vulnerable populations in an even disadvantaged and unfavorable position.
We propose an approach that leverages the power of web search and generative
models to alleviate some of the shortcomings of discriminative models. We
demonstrate our method on an image classification problem using ImageNet's
People Subtree subset, and show that it is effective in enhancing robustness
and mitigating bias in certain classes that represent vulnerable populations
(e.g., female doctor of color). Our new method is able to (1) identify weak
decision boundaries for such classes; (2) construct search queries for Google
as well as text for generating images through DALL-E 2 and Stable Diffusion;
and (3) show how these newly captured training samples could alleviate
population bias issue. While still improving the model's overall performance
considerably, we achieve a significant reduction (77.30\%) in the model's
gender accuracy disparity. In addition to these improvements, we observed a
notable enhancement in the classifier's decision boundary, as it is
characterized by fewer weakspots and an increased separation between classes.
Although we showcase our method on vulnerable populations in this study, the
proposed technique is extendable to a wide range of problems and domains.