Realistic Model Selection for Weakly Supervised Object Localization
Shakeeb Murtaza Soufiane Belharbi Marco Pedersoli Eric Granger
Weakly Supervised Object Localization (WSOL) allows for training deep
learning models for classification and localization, using only global
class-level labels. The lack of bounding box (bbox) supervision during training
represents a considerable challenge for hyper-parameter search and model
selection. Earlier WSOL works implicitly observed localization performance over
a test set which leads to biased performance evaluation. More recently, a
better WSOL protocol has been proposed, where a validation set with bbox
annotations is held out for model selection. Although it does not rely on the
test set, this protocol is unrealistic since bboxes are not available in
real-world applications, and when available, it is better to use them directly
to fit model weights. Our initial empirical analysis shows that the
localization performance of a model declines significantly when using only
image-class labels for model selection (compared to using bounding-box
annotations). This suggests that adding bounding-box labels is preferable for
selecting the best model for localization. In this paper, we introduce a new
WSOL validation protocol that provides a localization signal without the need
for manual bbox annotations. In particular, we leverage noisy pseudo boxes from
an off-the-shelf ROI proposal generator such as Selective-Search, CLIP, and RPN
pretrained models for model selection. Our experimental results with several
WSOL methods on ILSVRC and CUB-200-2011 datasets show that our noisy boxes
allow selecting models with performance close to those selected using ground
truth boxes, and better than models selected using only image-class labels.