Using Foundation Models to Detect Policy Violations with Minimal
  Supervision
Sid Mittal Vineet Gupta Frederick Liu Mukund Sundararajan
Foundation models, i.e. large neural networks pre-trained on large text
corpora, have revolutionized NLP. They can be instructed directly (e.g.
(arXiv:2005.14165)) - this is called hard prompting - and they can be tuned
using very little data (e.g. (arXiv:2104.08691)) - this technique is called
soft prompting. We seek to leverage their capabilities to detect policy
violations. Our contributions are: We identify a hard prompt that adapts
chain-of-thought prompting to policy violation tasks. This prompt produces
policy violation classifications, along with extractive explanations that
justify the classification. We compose the hard-prompts with soft prompt tuning
to produce a classifier that attains high accuracy with very little
supervision; the same classifier also produces explanations. Though the
supervision only acts on the classifications, we find that the modified
explanations remain consistent with the (tuned) model's response. Along the
way, we identify several unintuitive aspects of foundation models. For
instance, adding an example from a specific class can actually reduce
predictions of that class, and separately, the effects of tokenization on
scoring etc. Based on our technical results, we identify a simple workflow for
product teams to quickly develop effective policy violation detectors.