Into the crossfire: evaluating the use of a language model to
  crowdsource gun violence reports
Adriano Belisario Scott Hale Luc Rocher
Gun violence is a pressing and growing human rights issue that affects nearly
every dimension of the social fabric, from healthcare and education to
psychology and the economy. Reliable data on firearm events is paramount to
developing more effective public policy and emergency responses. However, the
lack of comprehensive databases and the risks of in-person surveys prevent
human rights organizations from collecting needed data in most countries. Here,
we partner with a Brazilian human rights organization to conduct a systematic
evaluation of language models to assist with monitoring real-world firearm
events from social media data. We propose a fine-tuned BERT-based model trained
on Twitter (now X) texts to distinguish gun violence reports from ordinary
Portuguese texts. Our model achieves a high AUC score of 0.97. We then
incorporate our model into a web application and test it in a live
intervention. We study and interview Brazilian analysts who continuously
fact-check social media texts to identify new gun violence events. Qualitative
assessments show that our solution helped all analysts use their time more
efficiently and expanded their search capacities. Quantitative assessments show
that the use of our model was associated with more analysts' interactions with
online users reporting gun violence. Taken together, our findings suggest that
modern Natural Language Processing techniques can help support the work of
human rights organizations.