A Monte Carlo Language Model Pipeline for Zero-Shot Sociopolitical Event
  Extraction
Erica Cai Brendan O'Connor
We consider dyadic zero-shot event extraction (EE) to identify actions
between pairs of actors. The \emph{zero-shot} setting allows social scientists
or other non-computational researchers to extract any customized,
user-specified set of events without training, resulting in a \emph{dyadic}
event database, allowing insight into sociopolitical relational dynamics among
actors and the higher level organizations or countries they represent.
Unfortunately, we find that current zero-shot EE methods perform poorly for the
task, with issues including word sense ambiguity, modality mismatch, and
efficiency. Straightforward application of large language model prompting
typically performs even worse. We address these challenges with a new
fine-grained, multi-stage generative question-answer method, using a Monte
Carlo approach to exploit and overcome the randomness of generative outputs. It
performs 90\% fewer queries than a previous approach, with strong performance
on the widely-used Automatic Content Extraction dataset. Finally, we extend our
method to extract affiliations of actor arguments and demonstrate our method
and findings on a dyadic international relations case study.