Moderating Illicit Online Image Promotion for Unsafe User-Generated
  Content Games Using Large Vision-Language Models
Keyan Guo Ayush Utkarsh Wenbo Ding Isabelle Ondracek Ziming Zhao Guo Freeman Nishant Vishwamitra Hongxin Hu
Online user-generated content games (UGCGs) are increasingly popular among
children and adolescents for social interaction and more creative online
entertainment. However, they pose a heightened risk of exposure to explicit
content, raising growing concerns for the online safety of children and
adolescents. Despite these concerns, few studies have addressed the issue of
illicit image-based promotions of unsafe UGCGs on social media, which can
inadvertently attract young users. This challenge arises from the difficulty of
obtaining comprehensive training data for UGCG images and the unique nature of
these images, which differ from traditional unsafe content. In this work, we
take the first step towards studying the threat of illicit promotions of unsafe
UGCGs. We collect a real-world dataset comprising 2,924 images that display
diverse sexually explicit and violent content used to promote UGCGs by their
game creators. Our in-depth studies reveal a new understanding of this problem
and the urgent need for automatically flagging illicit UGCG promotions. We
additionally create a cutting-edge system, UGCG-Guard, designed to aid social
media platforms in effectively identifying images used for illicit UGCG
promotions. This system leverages recently introduced large vision-language
models (VLMs) and employs a novel conditional prompting strategy for zero-shot
domain adaptation, along with chain-of-thought (CoT) reasoning for contextual
identification. UGCG-Guard achieves outstanding results, with an accuracy rate
of 94% in detecting these images used for the illicit promotion of such games
in real-world scenarios.