Counterfactual Reasoning for Multi-Label Image Classification via
  Patching-Based Training
Ming-Kun Xie Jia-Hao Xiao Pei Peng Gang Niu Masashi Sugiyama Sheng-Jun Huang
The key to multi-label image classification (MLC) is to improve model
performance by leveraging label correlations. Unfortunately, it has been shown
that overemphasizing co-occurrence relationships can cause the overfitting
issue of the model, ultimately leading to performance degradation. In this
paper, we provide a causal inference framework to show that the correlative
features caused by the target object and its co-occurring objects can be
regarded as a mediator, which has both positive and negative impacts on model
predictions. On the positive side, the mediator enhances the recognition
performance of the model by capturing co-occurrence relationships; on the
negative side, it has the harmful causal effect that causes the model to make
an incorrect prediction for the target object, even when only co-occurring
objects are present in an image. To address this problem, we propose a
counterfactual reasoning method to measure the total direct effect, achieved by
enhancing the direct effect caused only by the target object. Due to the
unknown location of the target object, we propose patching-based training and
inference to accomplish this goal, which divides an image into multiple patches
and identifies the pivot patch that contains the target object. Experimental
results on multiple benchmark datasets with diverse configurations validate
that the proposed method can achieve state-of-the-art performance.