SOUL: Towards Sentiment and Opinion Understanding of Language
Yue Deng Wenxuan Zhang Sinno Jialin Pan Lidong Bing
Sentiment analysis is a well-established natural language processing task,
with sentiment polarity classification being one of its most popular and
representative tasks. However, despite the success of pre-trained language
models in this area, they often fall short of capturing the broader
complexities of sentiment analysis. To address this issue, we propose a new
task called Sentiment and Opinion Understanding of Language (SOUL). SOUL aims
to evaluate sentiment understanding through two subtasks: Review Comprehension
(RC) and Justification Generation (JG). RC seeks to validate statements that
focus on subjective information based on a review text, while JG requires
models to provide explanations for their sentiment predictions. To enable
comprehensive evaluation, we annotate a new dataset comprising 15,028
statements from 3,638 reviews. Experimental results indicate that SOUL is a
challenging task for both small and large language models, with a performance
gap of up to 27% when compared to human performance. Furthermore, evaluations
conducted with both human experts and GPT-4 highlight the limitations of the
small language model in generating reasoning-based justifications. These
findings underscore the challenging nature of the SOUL task for existing
models, emphasizing the need for further advancements in sentiment analysis to
address its complexities. The new dataset and code are available at
https://github.com/DAMO-NLP-SG/SOUL.