Breaking Down Word Semantics from Pre-trained Language Models through
  Layer-wise Dimension Selection
Nayoung Choi
Contextual word embeddings obtained from pre-trained language model (PLM)
have proven effective for various natural language processing tasks at the word
level. However, interpreting the hidden aspects within embeddings, such as
syntax and semantics, remains challenging. Disentangled representation learning
has emerged as a promising approach, which separates specific aspects into
distinct embeddings. Furthermore, different linguistic knowledge is believed to
be stored in different layers of PLM. This paper aims to disentangle semantic
sense from BERT by applying a binary mask to middle outputs across the layers,
without updating pre-trained parameters. The disentangled embeddings are
evaluated through binary classification to determine if the target word in two
different sentences has the same meaning. Experiments with cased
BERT$_{\texttt{base}}$ show that leveraging layer-wise information is effective
and disentangling semantic sense further improve performance.