Safety and Fairness for Content Moderation in Generative Models
Susan Hao Piyush Kumar Sarah Laszlo Shivani Poddar Bhaktipriya Radharapu Renee Shelby
With significant advances in generative AI, new technologies are rapidly
being deployed with generative components. Generative models are typically
trained on large datasets, resulting in model behaviors that can mimic the
worst of the content in the training data. Responsible deployment of generative
technologies requires content moderation strategies, such as safety input and
output filters. Here, we provide a theoretical framework for conceptualizing
responsible content moderation of text-to-image generative technologies,
including a demonstration of how to empirically measure the constructs we
enumerate. We define and distinguish the concepts of safety, fairness, and
metric equity, and enumerate example harms that can come in each domain. We
then provide a demonstration of how the defined harms can be quantified. We
conclude with a summary of how the style of harms quantification we demonstrate
enables data-driven content moderation decisions.