Automating Governing Knowledge Commons and Contextual Integrity (GKC-CI)
  Privacy Policy Annotations with Large Language Models
Jake Chanenson Madison Pickering Noah Apthorpe
Identifying contextual integrity (CI) and governing knowledge commons (GKC)
parameters in privacy policy texts can facilitate normative privacy analysis.
However, GKC-CI annotation has heretofore required manual or crowdsourced
effort. This paper demonstrates that high-accuracy GKC-CI parameter annotation
of privacy policies can be performed automatically using large language models.
We fine-tune 18 open-source and proprietary models on 21,588 GKC-CI annotations
from 16 ground truth privacy policies. Our best-performing model (fine-tuned
GPT-3.5 Turbo with prompt engineering) has an accuracy of 86%, exceeding the
performance of prior crowdsourcing approaches despite the complexity of privacy
policy texts and the nuance of the GKC-CI annotation task. We apply our
best-performing model to privacy policies from 164 popular online services,
demonstrating the effectiveness of scaling GKC-CI annotation for data
exploration. We make all annotated policies as well as the training data and
scripts needed to fine-tune our best-performing model publicly available for
future research.