A Study on How Attention Scores in the BERT Model are Aware of Lexical
  Categories in Syntactic and Semantic Tasks on the GLUE Benchmark
Dongjun Jang Sungjoo Byun Hyopil Shin
This study examines whether the attention scores between tokens in the BERT
model significantly vary based on lexical categories during the fine-tuning
process for downstream tasks. Drawing inspiration from the notion that in human
language processing, syntactic and semantic information is parsed differently,
we categorize tokens in sentences according to their lexical categories and
focus on changes in attention scores among these categories. Our hypothesis
posits that in downstream tasks that prioritize semantic information, attention
scores centered on content words are enhanced, while in cases emphasizing
syntactic information, attention scores centered on function words are
intensified. Through experimentation conducted on six tasks from the GLUE
benchmark dataset, we substantiate our hypothesis regarding the fine-tuning
process. Furthermore, our additional investigations reveal the presence of BERT
layers that consistently assign more bias to specific lexical categories,
irrespective of the task, highlighting the existence of task-agnostic lexical
category preferences.