---
abstract: 'In this paper, we investigate the modeling power of contextualized embeddings from pre-trained language models, e.g. BERT, on the E2E-ABSA task. Specifically, we build a series of simple yet insightful neural baselines to deal with E2E-ABSA. The experimental results show that even with a simple linear classification layer, our BERT-based architecture can outperform state-of-the-art works. Besides, we also standardize the comparative study by consistently utilizing a hold-out development dataset for model selection, which is largely ignored by previous works. Therefore, our work can serve as a BERT-based benchmark for E2E-ABSA.[^1]'
author:
- |
    Xin Li[^1^]{}, Lidong Bing[^2^]{}, Wenxuan Zhang[^1^]{} and Wai Lam[^1^]{}\
    [[^1^]{}Department of Systems Engineering and Engineering Management\
    The Chinese University of Hong Kong, Hong Kong]{}\
    [[^2^]{}R&D Center Singapore, Machine Intelligence Technology, Alibaba DAMO Academy]{}\
    [`{lixin,wxzhang,wlam}@se.cuhk.edu.hk`]{}\
    [`l.bing@alibaba-inc.com`]{}\
bibliography:
- 'emnlp-ijcnlp-2019.bib'
title: 'Exploiting BERT for End-to-End Aspect-based Sentiment Analysis[^2]'
---

Introduction
============

Aspect-based sentiment analysis (ABSA) is to discover the users’ sentiment or opinion towards an aspect, usually in the form of explicitly mentioned aspect terms [@mitchell-etal-2013-open; @zhang-etal-2015-neural] or implicit aspect categories [@wang-etal-2016-attention], from user-generated natural language texts [@liu2012sentiment]. The most popular ABSA benchmark datasets are from SemEval ABSA challenges [@pontiki-etal-2014-semeval; @pontiki-etal-2015-semeval; @pontiki-etal-2016-semeval] where a few thousand review sentences with gold standard aspect sentiment annotations are provided.

Table \[tab:problem\_settings\] summarizes three existing research problems related to ABSA. The first one is the original ABSA, aiming at predicting the sentiment polarity of the sentence towards the given aspect. Compared to this classification problem, the second one and the third one, namely, Aspect-oriented Opinion Words Extraction (AOWE) [@fan-etal-2019-target] and End-to-End Aspect-based Sentiment Analysis (E2E-ABSA) [@ma-etal-2018-joint; @schmitt-etal-2018-joint; @li2019unified; @li2017learning; @li2019learning], are related to a sequence tagging problem. Precisely, the goal of AOWE is to extract the aspect-specific opinion words from the sentence given the aspect. The goal of E2E-ABSA is to jointly detect aspect terms/categories and the corresponding aspect sentiments.

Many neural models composed of a task-agnostic pre-trained word embedding layer and task-specific neural architecture have been proposed for the original ABSA task (i.e. the aspect-level sentiment classification) [@tang-etal-2016-aspect; @wang-etal-2016-attention; @chen-etal-2017-recurrent-attention; @liu-zhang-2017-attention; @ma2017interactive; @ma2018targeted; @majumder-etal-2018-iarm; @li-etal-2018-transformation; @he-etal-2018-exploiting; @xue-li-2018-aspect; @wang-etal-2018-target; @fan-etal-2018-multi; @huang-carley-2018-parameterized; @lei2019human; @li2019exploiting; @zhang2019aspect][^3], but the improvement of these models measured by the accuracy or F1 score has reached a bottleneck. One reason is that the task-agnostic embedding layer, usually a linear layer initialized with Word2Vec [@mikolov2013distributed] or GloVe [@pennington-etal-2014-glove], only provides context-independent word-level features, which is insufficient for capturing the complex semantic dependencies in the sentence. Meanwhile, the size of existing datasets is too small to train sophisticated task-specific architectures. Thus, introducing a context-aware word embedding[^4] layer pre-trained on large-scale datasets with deep LSTM [@mccann2017learned; @peters-etal-2018-deep; @howard-ruder-2018-universal] or Transformer [@radford2018improving; @radford2019language; @devlin-etal-2019-bert; @lample2019cross; @yang2019xlnet; @dong2019unified] for fine-tuning a lightweight task-specific network using the labeled data has good potential for further enhancing the performance.

@xu-etal-2019-bert [@sun-etal-2019-utilizing; @song2019attentional; @yu2019adapting; @rietzler2019adapt; @huang2019syntax; @hu2019learning] have conducted some initial attempts to couple the deep contextualized word embedding layer with downstream neural models for the original ABSA task and establish the new state-of-the-art results. It encourages us to explore the potential of using such contextualized embeddings to the more difficult but practical task, i.e. E2E-ABSA (the third setting in Table \[tab:problem\_settings\]).[^5] Note that we are not aiming at developing a task-specific architecture, instead, our focus is to examine the potential of contextualized embedding for E2E-ABSA, coupled with various simple layers for prediction of E2E-ABSA labels.[^6]

In this paper, we investigate the modeling power of BERT [@devlin-etal-2019-bert], one of the most popular pre-trained language model armed with Transformer [@vaswani2017attention], on the task of E2E-ABSA. Concretely, inspired by the investigation of E2E-ABSA in @li2019unified, which predicts aspect boundaries as well as aspect sentiments using a single sequence tagger, we build a series of simple yet insightful neural baselines for the sequence labeling problem and fine-tune the task-specific components with BERT or deem BERT as feature extractor. Besides, we standardize the comparative study by consistently utilizing the hold-out development dataset for model selection, which is ignored in most of the existing ABSA works [@tay2018learning].

Model
=====

In this paper, we focus on the aspect term-level End-to-End Aspect-Based Sentiment Analysis (E2E-ABSA) problem setting. This task can be formulated as a sequence labeling problem. The overall architecture of our model is depicted in Figure \[fig:architecture\]. Given the input token sequence $\mathrm{\bf x} = \{x_1, \cdots, x_T \}$ of length $T$, we firstly employ BERT component with $L$ transformer layers to calculate the corresponding contextualized representations $H^{L} = \{h^L_1, \cdots, h^L_T\} \in \mathbb{R}^{T \times \mathrm{dim}_h}$ for the input tokens where $\mathrm{dim}_h$ denotes the dimension of the representation vector. Then, the contextualized representations are fed to the task-specific layers to predict the tag sequence $\mathrm{\bf y} = \{y_1, \cdots, y_T\}$. The possible values of the tag $y_t$ are `B`-{`POS,NEG,NEU`}, `I`-{`POS,NEG,NEU`}, `E`-{`POS,NEG,NEU`}, `S`-{`POS,NEG,NEU`} or `O`, denoting the beginning of aspect, inside of aspect, end of aspect, single-word aspect, with positive, negative or neutral sentiment respectively, as well as outside of aspect.

![Overview of the designed model.[]{data-label="fig:architecture"}](architecture_crop.pdf){width="50.00000%"}

BERT as Embedding Layer
-----------------------

Compared to the traditional Word2Vec- or GloVe-based embedding layer which only provides a single context-independent representation for each token, the BERT embedding layer takes the sentence as input and calculates the token-level representations using the information from the entire sentence. First of all, we pack the input features as $H^0 = \{e_1,\cdots,e_T\}$, where $e_t$ ($t\in[1, T]$) is the combination of the token embedding, position embedding and segment embedding corresponding to the input token $x_t$. Then $L$ transformer layers are introduced to refine the token-level features layer by layer. Specifically, the representations $H^{l} = \{h^l_1, \cdots, h^l_T\}$ at the $l$-th ($l \in [1, L]$) layer are calculated below: $$\label{eq:bert}
    H^l = \text{Transformer}_l(H^{l-1})$$ We regard $H^L$ as the contextualized representations of the input tokens and use them to perform the predictions for the downstream task.

Design of Downstream Model
--------------------------

After obtaining the BERT representations, we design a neural layer, called E2E-ABSA layer in Figure \[tab:problem\_settings\], on top of BERT embedding layer for solving the task of E2E-ABSA. We investigate several different design for the E2E-ABSA layer, namely, linear layer, recurrent neural networks, self-attention networks, and conditional random fields layer.

#### Linear Layer

The obtained token representations can be directly fed to linear layer with softmax activation function to calculate the token-level predictions: $$P(y_t|x_t) = \text{softmax}(W_o h^L_t + b_o)$$ where $W_o \in \mathbb{R}^{\mathrm{dim}_h \times |\mathcal{Y}|}$ is the learnable parameters of the linear layer.

#### Recurrent Neural Networks

Considering its sequence labeling formulation, Recurrent Neural Networks (RNN) [@elman1990finding] is a natural solution for the task of E2E-ABSA. In this paper, we adopt GRU [@cho-etal-2014-learning], whose superiority compared to LSTM [@hochreiter1997long] and basic RNN has been verified in @jozefowicz2015empirical. The computational formula of the task-specific hidden representation $h^{\mathcal{T}}_t \in \mathbb{R}^{\mathrm{dim}_h}$ at the $t$-th time step is shown below: $$\small
    \begin{split}
        \begin{bmatrix}
        r_t \\
        z_t
     \end{bmatrix}
     &= \sigma(\textsc{Ln}(W_x h^L_t)+\textsc{Ln}(W_h h^{\mathcal{T}}_{t-1})) \\
     n_t &= \text{tanh}(\textsc{Ln}(W_{xn} h^L_t)+r_t * \textsc{Ln}(W_{hn} h^{\mathcal{T}}_{t-1})) \\
     h^{\mathcal{T}}_{t} &= (1-z_t) * n_t + z_t * h^{\mathcal{T}}_{t-1}
    \end{split}$$ where $\sigma$ is the sigmoid activation function and $r_t$, $z_t$, $n_t$ respectively denote the reset gate, update gate and new gate. $W_x,W_h \in \mathbb{R}^{2\mathrm{dim}_h \times \mathrm{dim}_h}$, $W_{xn}, W_{hn} \in \mathbb{R}^{\mathrm{dim}_h \times \mathrm{dim}_h}$ are the parameters of GRU. Since directly applying RNN on the output of transformer, namely, the BERT representation $h^L_t$, may lead to unstable training [@chen-etal-2018-best; @liu2019fine], we add additional layer-normalization [@ba2016layer], denoted as $\textsc{Ln}$, when calculating the gates. Then, the predictions are obtained by introducing a softmax layer: $$\label{output}
    p(y_t|x_t) = \text{softmax}(W_o h^{\mathcal{T}}_t + b_o)$$

#### Self-Attention Networks

With the help of self attention [@cheng-etal-2016-long; @lin2017structured], Self-Attention Network [@vaswani2017attention; @shen2018disan] is another effective feature extractor apart from RNN and CNN. In this paper, we introduce two SAN variants to build the task-specific token representations $H^{\mathcal{T}}=\{h^{\mathcal{T}}_1,\cdots,h^{\mathcal{T}}_T\}$. One variant is composed of a simple self-attention layer and residual connection [@he2016deep], dubbed as “SAN”. The computational process of SAN is below: $$\begin{split}
    H^{\mathcal{T}} &= \textsc{Ln}(H^{L}+\textsc{Slf-Att} (Q, K, V)) \\
    Q, K, V &=  H^{L} W^{Q}, H^{L}W^{K}, H^{L} W^{V}
        \end{split}$$ where <span style="font-variant:small-caps;">Slf-Att</span> is identical to the self-attentive scaled dot-product attention [@vaswani2017attention]. Another variant is a transformer layer (dubbed as “TFM”), which has the same architecture with the transformer encoder layer in the BERT. The computational process of TFM is as follows: $$\begin{split}
    \hat{H}^{L} &= \textsc{Ln}(H^{L}+\textsc{Slf-Att} (Q, K, V)) \\
    H^{\mathcal{T}} &= \textsc{Ln}(\hat{H}^{L}+\textsc{Ffn}(\hat{H}^{L}))
        \end{split}$$ where <span style="font-variant:small-caps;">Ffn</span> refers to the point-wise feed-forward networks [@vaswani2017attention]. Again, a linear layer with softmax activation is stacked on the designed SAN/TFM layer to output the predictions (same with that in Eq(\[output\])).

[ll|c|c|c|c]{} & Train & Dev & Test & Total\
& \# sent & 2741 & 304 & 800 & 4245\
& \# aspect & 2041 & 256 & 634 & 2931\
& \# sent & 3490 & 387 & 2158 & 6035\
& \# aspect & 3893 & 413 & 2287 & 6593\

#### Conditional Random Fields

Conditional Random Fields (CRF) [@lafferty2001conditional] is effective in sequence modeling and has been widely adopted for solving the sequence labeling tasks together with neural models [@huang2015bidirectional; @lample-etal-2016-neural; @ma-hovy-2016-end]. In this paper, we introduce a linear-chain CRF layer on top of the BERT embedding layer. Different from the above mentioned neural models maximizing the token-level likelihood $p(y_t|x_t)$, the CRF-based model aims to find the globally most probable tag sequence. Specifically, the sequence-level scores $s(\mathrm{\bf x},\mathrm{\bf y})$ and likelihood $p(\mathrm{\bf y}|\mathrm{\bf x})$ of $\mathrm{\bf y} = \{y_1,\cdots,y_T\}$ are calculated as follows: $$    \begin{split}
        s(\mathrm{\bf x},\mathrm{\bf y}) &= \sum^T_{t=0} M^A_{y_t,y_{t+1}} + \sum^T_{t=1} M^P_{t, y_t} \\
        p(\mathrm{\bf y}|\mathrm{\bf x}) &= \text{softmax}(s(\mathrm{\bf x},\mathrm{\bf y}))
    \end{split}$$ where $M^A \in \mathbb{R}^{|\mathcal{Y}| \times |\mathcal{Y}|}$ is the randomly initialized transition matrix for modeling the dependency between the adjacent predictions and $M^P \in \mathbb{R}^{T \times |\mathcal{Y}|}$ denote the emission matrix linearly transformed from the BERT representations $H^L$. The softmax here is conducted over all of the possible tag sequences. As for the decoding, we regard the tag sequence with the highest scores as output: $$\mathrm{\bf y}^* = \arg\max_{\mathrm{\bf y}} s(\mathrm{\bf x},\mathrm{\bf y})$$ where the solution is obtained via Viterbi search.

![Performances on the Dev set of `REST`.[]{data-label="fig:overfit"}](overfit_v4.pdf){width="40.00000%"}

Experiment
==========

Dataset and Settings
--------------------

We conduct experiments on two review datasets originating from SemEval [@pontiki-etal-2014-semeval; @pontiki-etal-2015-semeval; @pontiki-etal-2016-semeval] but re-prepared in @li2019unified. The statistics are summarized in Table \[tab:dataset\]. We use the pre-trained “bert-base-uncased” model[^7], where the number of transformer layers $L=12$ and the hidden size $\mathrm{dim}_h$ is 768. For the downstream E2E-ABSA component, we consistently use the single-layer architecture and set the dimension of task-specific representation as $\mathrm{dim}_h$. The learning rate is 2e-5. The batch size is set as 25 for `LAPTOP` and 16 for `REST`. We train the model up to 1500 steps. After training 1000 steps, we conduct model selection on the development set for very 100 steps according to the micro-averaged F1 score. Following these settings, we train 5 models with different random seeds and report the average results.

We compare with **Existing Models**, including tailor-made E2E-ABSA models [@li2019unified; @luo-etal-2019-doer; @he-etal-2019-interactive], and competitive **LSTM-CRF** sequence labeling models [@lample-etal-2016-neural; @ma-hovy-2016-end; @liu2018empower].

Main Results
------------

From Table \[tab:main\_results\], we surprisingly find that only introducing a simple token-level classifier, namely, BERT-Linear, already outperforms the existing works without using BERT, suggesting that BERT representations encoding the associations between arbitrary two tokens largely alleviate the issue of context independence in the linear E2E-ABSA layer. It is also observed that slightly more powerful E2E-ABSA layers lead to much better performance, verifying the postulation that incorporating context helps to sequence modeling.

Over-parameterization Issue
---------------------------

Although we employ the smallest pre-trained BERT model, it is still over-parameterized for the E2E-ABSA task (110M parameters), which naturally raises a question: does BERT-based model tend to overfit the small training set? Following this question, we train BERT-GRU, BERT-TFM and BERT-CRF up to 3000 steps on `REST` and observe the fluctuation of the F1 measures on the development set. As shown in Figure \[fig:overfit\], F1 scores on the development set are quite stable and do not decrease much as the training proceeds, which shows that the BERT-based model is exceptionally robust to overfitting.

Finetuning BERT or Not
----------------------

We also study the impact of fine-tuning on the final performances. Specifically, we employ BERT to calculate the contextualized token-level representations but kept the parameters of BERT component unchanged in the training phase. Figure \[fig:finetune\] illustrate the comparative results between the BERT-based models and those keeping BERT component fixed. Obviously, the general purpose BERT representation is far from satisfactory for the downstream tasks and task-specific fine-tuning is essential for exploiting the strengths of BERT to improve the performance.

![Effect of fine-tuning BERT.[]{data-label="fig:finetune"}](finetuning_v3.pdf){width="40.00000%"}

Conclusion
==========

In this paper, we investigate the effectiveness of BERT embedding component on the task of End-to-End Aspect-Based Sentiment Analysis (E2E-ABSA). Specifically, we explore to couple the BERT embedding component with various neural models and conduct extensive experiments on two benchmark datasets. The experimental results demonstrate the superiority of BERT-based models on capturing aspect-based sentiment and their robustness to overfitting.

[^1]: Our code is open-source and available at: <https://github.com/lixin4ever/BERT-E2E-ABSA>

[^2]: The work described in this paper is substantially supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14204418).

[^3]: Due to the limited space, we can not list all of the existing works here, please refer to the survey [@zhou2019deep] for more related papers.

[^4]: In this paper, we generalize the concept of “word embedding” as a mapping between the word and the low-dimensional word representations.

[^5]: Both of ABSA and AOWE assume that the aspects in a sentence are given. Such setting makes them less practical in real-world scenarios since manual annotation of the fine-grained aspect mentions/categories is quite expensive.

[^6]: @hu-etal-2019-open introduce BERT to handle the E2E-ABSA problem but their focus is to design a task-specific architecture rather than exploring the potential of BERT.

[^7]: https://github.com/huggingface/transformers
