Automatically Evaluating Opinion Prevalence in Opinion Summarization
Christopher Malon
When faced with a large number of product reviews, it is not clear that a
human can remember all of them and weight opinions representatively to write a
good reference summary. We propose an automatic metric to test the prevalence
of the opinions that a summary expresses, based on counting the number of
reviews that are consistent with each statement in the summary, while
discrediting trivial or redundant statements. To formulate this opinion
prevalence metric, we consider several existing methods to score the factual
consistency of a summary statement with respect to each individual source
review. On a corpus of Amazon product reviews, we gather multiple human
judgments of the opinion consistency, to determine which automatic metric best
expresses consistency in product reviews. Using the resulting opinion
prevalence metric, we show that a human authored summary has only slightly
better opinion prevalence than randomly selected extracts from the source
reviews, and previous extractive and abstractive unsupervised opinion
summarization methods perform worse than humans. We demonstrate room for
improvement with a greedy construction of extractive summaries with twice the
opinion prevalence achieved by humans. Finally, we show that preprocessing
source reviews by simplification can raise the opinion prevalence achieved by
existing abstractive opinion summarization systems to the level of human
performance.