Optimization of Rank Losses for Image Retrieval
Elias Ramzi Nicolas Audebert Clément Rambour André Araujo Xavier Bitot Nicolas Thome
In image retrieval, standard evaluation metrics rely on score ranking, \eg
average precision (AP), recall at k (R@k), normalized discounted cumulative
gain (NDCG). In this work we introduce a general framework for robust and
decomposable rank losses optimization. It addresses two major challenges for
end-to-end training of deep neural networks with rank losses:
non-differentiability and non-decomposability. Firstly we propose a general
surrogate for ranking operator, SupRank, that is amenable to stochastic
gradient descent. It provides an upperbound for rank losses and ensures robust
training. Secondly, we use a simple yet effective loss function to reduce the
decomposability gap between the averaged batch approximation of ranking losses
and their values on the whole training set. We apply our framework to two
standard metrics for image retrieval: AP and R@k. Additionally we apply our
framework to hierarchical image retrieval. We introduce an extension of AP, the
hierarchical average precision $\mathcal{H}$-AP, and optimize it as well as the
NDCG. Finally we create the first hierarchical landmarks retrieval dataset. We
use a semi-automatic pipeline to create hierarchical labels, extending the
large scale Google Landmarks v2 dataset. The hierarchical dataset is publicly
available at https://github.com/cvdfoundation/google-landmark. Code will be
released at https://github.com/elias-ramzi/SupRank.