GenKL: An Iterative Framework for Resolving Label Ambiguity and Label
  Non-conformity in Web Images Via a New Generalized KL Divergence
Xia Huang Kai Fong Ernest Chong
Web image datasets curated online inherently contain ambiguous
in-distribution (ID) instances and out-of-distribution (OOD) instances, which
we collectively call non-conforming (NC) instances. In many recent approaches
for mitigating the negative effects of NC instances, the core implicit
assumption is that the NC instances can be found via entropy maximization. For
"entropy" to be well-defined, we are interpreting the output prediction vector
of an instance as the parameter vector of a multinomial random variable, with
respect to some trained model with a softmax output layer. Hence, entropy
maximization is based on the idealized assumption that NC instances have
predictions that are "almost" uniformly distributed. However, in real-world web
image datasets, there are numerous NC instances whose predictions are far from
being uniformly distributed. To tackle the limitation of entropy maximization,
we propose $(\alpha, \beta)$-generalized KL divergence,
$\mathcal{D}_{\text{KL}}^{\alpha, \beta}(p\|q)$, which can be used to identify
significantly more NC instances. Theoretical properties of
$\mathcal{D}_{\text{KL}}^{\alpha, \beta}(p\|q)$ are proven, and we also show
empirically that a simple use of $\mathcal{D}_{\text{KL}}^{\alpha,
\beta}(p\|q)$ outperforms all baselines on the NC instance identification task.
Building upon $(\alpha,\beta)$-generalized KL divergence, we also introduce a
new iterative training framework, GenKL, that identifies and relabels NC
instances. When evaluated on three web image datasets, Clothing1M,
Food101/Food101N, and mini WebVision 1.0, we achieved new state-of-the-art
classification accuracies: $81.34\%$, $85.73\%$ and $78.99\%$/$92.54\%$
(top-1/top-5), respectively.