VEglue: Testing Visual Entailment Systems via Object-Aligned Joint
  Erasing
Zhiyuan Chang Mingyang Li Junjie Wang Cheng Li Qing Wang
Visual entailment (VE) is a multimodal reasoning task consisting of
image-sentence pairs whereby a promise is defined by an image, and a hypothesis
is described by a sentence. The goal is to predict whether the image
semantically entails the sentence. VE systems have been widely adopted in many
downstream tasks. Metamorphic testing is the commonest technique for AI
algorithms, but it poses a significant challenge for VE testing. They either
only consider perturbations on single modality which would result in
ineffective tests due to the destruction of the relationship of image-text
pair, or just conduct shallow perturbations on the inputs which can hardly
detect the decision error made by VE systems. Motivated by the fact that
objects in the image are the fundamental element for reasoning, we propose
VEglue, an object-aligned joint erasing approach for VE systems testing. It
first aligns the object regions in the premise and object descriptions in the
hypothesis to identify linked and un-linked objects. Then, based on the
alignment information, three Metamorphic Relations are designed to jointly
erase the objects of the two modalities. We evaluate VEglue on four widely-used
VE systems involving two public datasets. Results show that VEglue could detect
11,609 issues on average, which is 194%-2,846% more than the baselines. In
addition, VEglue could reach 52.5% Issue Finding Rate (IFR) on average, and
significantly outperform the baselines by 17.1%-38.2%. Furthermore, we leverage
the tests generated by VEglue to retrain the VE systems, which largely improves
model performance (50.8% increase in accuracy) on newly generated tests without
sacrificing the accuracy on the original test set.