ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in
  Real-World User-AI Conversation
Zi Lin Zihan Wang Yongqi Tong Yangkun Wang Yuxin Guo Yujia Wang Jingbo Shang
Despite remarkable advances that large language models have achieved in
chatbots, maintaining a non-toxic user-AI interactive environment has become
increasingly critical nowadays. However, previous efforts in toxicity detection
have been mostly based on benchmarks derived from social media content, leaving
the unique challenges inherent to real-world user-AI interactions
insufficiently explored. In this work, we introduce ToxicChat, a novel
benchmark based on real user queries from an open-source chatbot. This
benchmark contains the rich, nuanced phenomena that can be tricky for current
toxicity detection models to identify, revealing a significant domain
difference compared to social media content. Our systematic evaluation of
models trained on existing toxicity datasets has shown their shortcomings when
applied to this unique domain of ToxicChat. Our work illuminates the
potentially overlooked challenges of toxicity detection in real-world user-AI
conversations. In the future, ToxicChat can be a valuable resource to drive
further advancements toward building a safe and healthy environment for user-AI
interactions.