TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal
  Backdoored Models
Indranil Sur Karan Sikka Matthew Walmer Kaushik Koneripalli Anirban Roy Xiao Lin Ajay Divakaran Susmit Jha
We present a Multimodal Backdoor Defense technique TIJO (Trigger Inversion
using Joint Optimization). Recent work arXiv:2112.07668 has demonstrated
successful backdoor attacks on multimodal models for the Visual Question
Answering task. Their dual-key backdoor trigger is split across two modalities
(image and text), such that the backdoor is activated if and only if the
trigger is present in both modalities. We propose TIJO that defends against
dual-key attacks through a joint optimization that reverse-engineers the
trigger in both the image and text modalities. This joint optimization is
challenging in multimodal models due to the disconnected nature of the visual
pipeline which consists of an offline feature extractor, whose output is then
fused with the text using a fusion module. The key insight enabling the joint
optimization in TIJO is that the trigger inversion needs to be carried out in
the object detection box feature space as opposed to the pixel space. We
demonstrate the effectiveness of our method on the TrojVQA benchmark, where
TIJO improves upon the state-of-the-art unimodal methods from an AUC of 0.6 to
0.92 on multimodal dual-key backdoors. Furthermore, our method also improves
upon the unimodal baselines on unimodal backdoors. We present ablation studies
and qualitative results to provide insights into our algorithm such as the
critical importance of overlaying the inverted feature triggers on all visual
features during trigger inversion. The prototype implementation of TIJO is
available at https://github.com/SRI-CSL/TIJO.