Validating Multimedia Content Moderation Software via Semantic Fusion
Wenxuan Wang Jingyuan Huang Chang Chen Jiazhen Gu Jianping Zhang Weibin Wu Pinjia He Michael Lyu
The exponential growth of social media platforms, such as Facebook and
TikTok, has revolutionized communication and content publication in human
society. Users on these platforms can publish multimedia content that delivers
information via the combination of text, audio, images, and video. Meanwhile,
the multimedia content release facility has been increasingly exploited to
propagate toxic content, such as hate speech, malicious advertisements, and
pornography. To this end, content moderation software has been widely deployed
on these platforms to detect and blocks toxic content. However, due to the
complexity of content moderation models and the difficulty of understanding
information across multiple modalities, existing content moderation software
can fail to detect toxic content, which often leads to extremely negative
impacts.
  We introduce Semantic Fusion, a general, effective methodology for validating
multimedia content moderation software. Our key idea is to fuse two or more
existing single-modal inputs (e.g., a textual sentence and an image) into a new
input that combines the semantics of its ancestors in a novel manner and has
toxic nature by construction. This fused input is then used for validating
multimedia content moderation software. We realized Semantic Fusion as DUO, a
practical content moderation software testing tool. In our evaluation, we
employ DUO to test five commercial content moderation software and two
state-of-the-art models against three kinds of toxic content. The results show
that DUO achieves up to 100% error finding rate (EFR) when testing moderation
software. In addition, we leverage the test cases generated by DUO to retrain
the two models we explored, which largely improves model robustness while
maintaining the accuracy on the original test set.