An Image is Worth a Thousand Toxic Words: A Metamorphic Testing
  Framework for Content Moderation Software
Wenxuan Wang Jingyuan Huang Jen-tse Huang Chang Chen Jiazhen Gu Pinjia He Michael R. Lyu
The exponential growth of social media platforms has brought about a
revolution in communication and content dissemination in human society.
Nevertheless, these platforms are being increasingly misused to spread toxic
content, including hate speech, malicious advertising, and pornography, leading
to severe negative consequences such as harm to teenagers' mental health.
Despite tremendous efforts in developing and deploying textual and image
content moderation methods, malicious users can evade moderation by embedding
texts into images, such as screenshots of the text, usually with some
interference. We find that modern content moderation software's performance
against such malicious inputs remains underexplored. In this work, we propose
OASIS, a metamorphic testing framework for content moderation software. OASIS
employs 21 transform rules summarized from our pilot study on 5,000 real-world
toxic contents collected from 4 popular social media applications, including
Twitter, Instagram, Sina Weibo, and Baidu Tieba. Given toxic textual contents,
OASIS can generate image test cases, which preserve the toxicity yet are likely
to bypass moderation. In the evaluation, we employ OASIS to test five
commercial textual content moderation software from famous companies (i.e.,
Google Cloud, Microsoft Azure, Baidu Cloud, Alibaba Cloud and Tencent Cloud),
as well as a state-of-the-art moderation research model. The results show that
OASIS achieves up to 100% error finding rates. Moreover, through retraining the
models with the test cases generated by OASIS, the robustness of the moderation
model can be improved without performance degradation.