A Statistical Turing Test for Generative Models
Hayden Helm Carey E. Priebe Weiwei Yang
The emergence of human-like abilities of AI systems for content generation in
domains such as text, audio, and vision has prompted the development of
classifiers to determine whether content originated from a human or a machine.
Implicit in these efforts is an assumption that the generation properties of a
human are different from that of the machine. In this work, we provide a
framework in the language of statistical pattern recognition that quantifies
the difference between the distributions of human and machine-generated content
conditioned on an evaluation context. We describe current methods in the
context of the framework and demonstrate how to use the framework to evaluate
the progression of generative models towards human-like capabilities, among
many axes of analysis.