An Information-Theoretic Approach to Analyze NLP Classification Tasks
Luran Wang Mark Gales Vatsal Raina
Understanding the importance of the inputs on the output is useful across
many tasks. This work provides an information-theoretic framework to analyse
the influence of inputs for text classification tasks. Natural language
processing (NLP) tasks take either a single element input or multiple element
inputs to predict an output variable, where an element is a block of text. Each
text element has two components: an associated semantic meaning and a
linguistic realization. Multiple-choice reading comprehension (MCRC) and
sentiment classification (SC) are selected to showcase the framework. For MCRC,
it is found that the context influence on the output compared to the question
influence reduces on more challenging datasets. In particular, more challenging
contexts allow a greater variation in complexity of questions. Hence, test
creators need to carefully consider the choice of the context when designing
multiple-choice questions for assessment. For SC, it is found the semantic
meaning of the input text dominates (above 80\% for all datasets considered)
compared to its linguistic realisation when determining the sentiment. The
framework is made available at:
https://github.com/WangLuran/nlp-element-influence