Measuring the Effect of Influential Messages on Varying Personas
Chenkai Sun Jinning Li Hou Pong Chan ChengXiang Zhai Heng Ji
Predicting how a user responds to news events enables important applications
such as allowing intelligent agents or content producers to estimate the effect
on different communities and revise unreleased messages to prevent unexpected
bad outcomes such as social conflict and moral injury. We present a new task,
Response Forecasting on Personas for News Media, to estimate the response a
persona (characterizing an individual or a group) might have upon seeing a news
message. Compared to the previous efforts which only predict generic comments
to news, the proposed task not only introduces personalization in the modeling
but also predicts the sentiment polarity and intensity of each response. This
enables more accurate and comprehensive inference on the mental state of the
persona. Meanwhile, the generated sentiment dimensions make the evaluation and
application more reliable. We create the first benchmark dataset, which
consists of 13,357 responses to 3,847 news headlines from Twitter. We further
evaluate the SOTA neural language models with our dataset. The empirical
results suggest that the included persona attributes are helpful for the
performance of all response dimensions. Our analysis shows that the
best-performing models are capable of predicting responses that are consistent
with the personas, and as a byproduct, the task formulation also enables many
interesting applications in the analysis of social network groups and their
opinions, such as the discovery of extreme opinion groups.