---
abstract: 'Unresolved coreference is a bottleneck for relation extraction, and high-quality coreference resolvers may produce an output that makes it a lot easier to extract knowledge triples. We show how to improve coreference resolvers by forwarding their input to a relation extraction system and reward the resolvers for producing triples that are found in knowledge bases. Since relation extraction systems can rely on different forms of supervision and be biased in different ways, we obtain the best performance, improving over the state of the art, using multi-task reinforcement learning.'
author:
- '**Rahul Aralikatte**'
- '**Heather Lent**'
- '**Ana Valeria Gonzalez**'
- '**Daniel Hershcovich**'
- |
    \
    **Chen Qiu**
- '**Anders Sandholm**'
- '**Michael Ringaard**'
- '**Anders S[ø]{}gaard**'
bibliography:
- 'coref-rl.bib'
title: |
    Rewarding Coreference Resolvers for Being Consistent\
    with World Knowledge
---

Introduction {#sec:introduction}
============

Coreference annotations are costly and difficult to obtain, since trained annotators with sufficient world knowledge are necessary for reliable annotations. This paper presents a way to [*simulate*]{} annotators using reinforcement learning. To motivate our approach, we rely on the following example from :

-   $_1$ was formed in [****]{}. Other bands from \[[****]{}\]$_2$ include [****]{} and [****]{}.

cite the association between and as an example of a common source of name-name recall error for state-of-the-art coreference resolution systems. The challenge is that the two names co-occur relatively infrequently and are unlikely to do so in a moderate-sized, manually annotated training corpus. A state-of-the-art system may be able to infer the relation using distributional information about the phrase , but is likely to have limited evidence for the decision that it is coreferential with rather than .

While coreference-annotated data is scarce, knowledge bases including factual information (such as that is from ) are increasingly available. For a human annotator unaware that is sometimes referred to as , the information that is from is sufficient to establish that and are (with high probability) coreferential. This paper explores a novel architecture for making use of such information from knowledge bases by tying a coreference resolution system to a relation extraction system, enabling us to reward the coreference system for making predictions that lead us to infer facts that are consistent with such knowledge bases. This potentially provides us with more evidence for resolving coreference such as (1).

![\[fig:sys\_arch\]Our strategy for training a coreference resolver using reward from relation extraction.](images/arch.pdf){width="\columnwidth"}

![image](images/overall-newest.pdf){width="\textwidth"}

We propose a training strategy (Figure \[fig:sys\_arch\]) in which we pass on the predictions of a neural coreference resolver to an open relation extraction (OpenRE) system, matching relations extracted from resolved sentences with a knowledge base. We show how checking the produced relationships for consistency against the knowledge base produces a reward that is, indirectly, a signal about the quality of the coreference resolution. In order to generalize this signal beyond the coverage of the knowledge base, we train a Universal Schema model [@riedel-etal-2013-relation] and use its confidence as our reward function. With this reward function, we do policy-gradient fine-tuning of our coreference resolver, effectively optimizing its predictions’ consistency with world knowledge.

#### Contributions

We demonstrate that training a coreference resolver by reinforcement learning with rewards from a relation extraction system, results in improvements for coreference resolution. Our code is made publicly available at <https://github.com/rahular/coref-rl>

Consistency Reward for Coreference Resolution {#sec:strategy}
=============================================

In order to reward a coreference resolver for being consistent with world knowledge, we propose a simple training strategy based on relation extraction: (i) Sample a Wikipedia[^1] document at random, (ii) Replace mentions with their antecedents using a coreference resolver, (iii) Apply an off-the-shelf openRE system to each rewritten document, (iv) Score relationships that include coreferent mentions using Universal Schema, and (v) Use the score as a reward for training the coreference resolvers.

#### Reward functions

To model consistency with world knowledge, we train different Universal Schema models [@riedel-etal-2013-relation; @Verga:McCallum:16], resulting in three reward functions (Figure \[fig:overall\]): **RE-KG** (Knowledge Graph Universal Schema) is trained to predict whether two entities are linked in Wikidata[^2]; **RE-Text** (Text-based Universal Schema) is trained to predict whether two entities co-occur in Wikipedia; and **RE-Joint** (Joint Universal Schema) is trained to predict whether two entities are linked [*and*]{} co-occur. The three rewards focus on different aspects of relationships between entities, giving complimentary views of what entities are related.

Similar to , we parameterize candidate relation phrases with a BiLSTM [@graves2005framewise], and use pre-trained Wikidata BigGraph embeddings [@pbg] as the entity representations. We apply a one-layer MLP on the concatenated representations to get the reward value.

#### Updating the coreference resolver

Each resolved document is converted into $n$ subject-relation-object (SRO) triples by an open information retrieval system [@openIE]. Each triple $t_i$ is then scored using a reward function to obtain a reward $r_i$ for $i \in \{1, \ldots, n\}$. The final document-level reward is the normalized sum of the individual rewards as shown in Equation \[eqn:reward\], where $R_h$ is a moving window containing the previous $h=100$ normalized reward values.

$$R = \frac{\sum_{i} r_i - mean(R_h)}{stddev(R_h)}
    \label{eqn:reward}$$

Since $R$ is not differentiable with respect to the coreference resolver’s parameters, we use policy gradient training to update the coreference resolver. We select the best action according to the current policy, using random exploration of the alternative solutions with $p=\frac{1}{10}$.

#### Multi-task reinforcement learning

Our overall training procedure is presented in Algorithm \[alg:distral\]. After training the three aforementioned reward models, we create **RE-Distill** by interpolating their trained weights. Next, we pre-train a coreference resolver using supervised learning, and fine-tune it using each of the three reward functions to get three different coreference policies: **Coref-KG**, **Coref-Text** and **Coref-Joint**, respectively. We then use multi-task reinforcement learning to combine these three policies to get **Coref-Distill**. Our approach is a particular instance of DisTraL [@distral], using policy gradient and model interpolation. Finally, **Coref-Distill** is fine-tuned with rewards from [**RE-Distill**]{}.

Baseline initialized policies $\theta_n$ for $n\in \{1,2,3\}$ \[alg:pretrained-policies\]\
Reward functions `reward_n` for $n\in \{1,2,3\}$ Distilled reward function `reward_*` Sample $k$ documents $D^k$ $\mathcal{C}_d$ = entity clusters with $\theta_n$ $d'$ = resolve $d$ with $\mathcal{C}_d$ $\mathcal{T}$ = obtain OpenIE triples for $d'$ $r$ = reward$_n$($d'$) $\hat{g}_k$ = policy gradient for $\theta_n$ with reward $r$ $\theta_n^{k+1}=\theta_n^k+\alpha_k\hat{g}_k$ Distilled policy $\theta_*=\frac{\theta_1+\theta_2+\theta_3}{3}$ Sample $k$ documents $D^k$ $d'$ = resolve $d$ with $\mathcal{C}_d$ $\mathcal{T}$ = obtain OpenIE triples for $d'$ $r$ = reward$_*$($d'$) $\hat{g}_k$ = policy gradient for $\theta_*$ with reward $r$ $\theta_*^{k+1}=\theta_*^k+\alpha_k\hat{g}_k$ Distilled policy $\theta_*$

Experiments {#sec:experiments}
===========

We use a state-of-the-art neural coreference resolution model [@lee2018higher] as our baseline coreference resolver.[^3] This model extends @lee2017end with coarse-to-fine inference and ELMo pretrained embeddings [@peters2018deep].

#### Data

We use the standard training, validation, and test splits from the English OntoNotes.[^4] We also evaluate on the English WikiCoref [@wikicoref], with a validation and test split of 10 and 20 documents respectively.

#### Reward model training

We use data from English Wikipedia and Wikidata to train our three reward models. For training **RE-KG**, we sample 1 million Wikidata triples, and expand them to 12 million triples by replacing relation phrases with their aliases. For **RE-Text**, we pass the summary paragraphs from 50,000 random Wikipedia pages to Stanford’s OpenIE extractor [@corenlp], creating 2 million triples. For **RE-Joint**, we only use Wikipedia triples that are grounded in Wikidata, resulting in 60,000 triples.[^5] We further sample 200,000 triples from Wikidata and Wikipedia for validation, and train the reward models with early stopping based on the F$_1$ score of their predictions.

#### Evaluation

All models are evaluated using the standard CoNLL metric, which is the average F$_1$ score over MUC, CEAFe, and $B^3$ [@denis2009global].

Results {#sec:results}
=======

Since the quality of our reward models is essential to the performance of the coreference resolver adaptations, we first report the validation accuracy and F$_1$ scores of the four reward models used, in Table \[tab:reward\_results\]. We clearly see the advantage of distillation, with a 5% absolute difference between the best single model ([**RE-Text**]{}) and [**RE-Distill**]{}.

  [**System**]{}    [**Data**]{}   [**Accuracy**]{}   [**F$_1$ score**]{}
  ---------------- -------------- ------------------ ---------------------
  RE-KG                 12M              0.64                0.78
  RE-Text                2M              0.71                0.83
  RE-Joint              60K              0.58                0.73
  RE-Distill             —             **0.78**            **0.88**

  : Training data size, accuracy and F$_1$ scores of the reward models on the 200,000 validation triples.\[tab:reward\_results\]

  [**System**]{}    [**OntoNotes**]{}   [**WikiCoref**]{}
  ---------------- ------------------- -------------------
                          72.60               57.49
  Coref-KG                72.96               57.84
  Coref-Text              72.99               57.54
  Coref-Joint             72.77               57.51
  Coref-Distill         **73.10**           **58.14**

  : Coreference results: average F$_1$ scores on the OntoNotes and WikiCoref test sets. Differences are significant w.r.t. $B^3$ (bootstrap test, $p<0.05$).\[tab:coref\_results\]

Table \[tab:coref\_results\] presents the downstream effects of applying these reward functions to our baseline coreference policy.[^6]

The coreference resolution results are similar to the relation extraction results: using a distilled policy, learned through multi-task reinforcement learning, leads to better results on both datasets.[^7]

While improvements over the current state of the art are relatively small, they reflect significant progress, as they demonstrate the ability to successfully augment coreference resolvers with “free" data from large-scale KB like Wikidata. For relation extraction, this could have positive downstream effects, and also ensure that relations are consistent with real world knowledge. Moreover, this approach has the potential to also be beneficial for coreference resolution in low resource languages, where less annotated data is available, as Wikidata triples are abundant for many languages.

Analysis {#sec:analysis}
========

![image](images/analysis.pdf){width="\textwidth"}

Empirically, we find that fine-tuning the coreference resolver on Wikidata results in two kinds of improvements:

#### Better mention detection

Since the model is rewarded if the SRO triples produced from the resolved document are present in Wikidata, the model can do well only if it correctly resolves the subject and object, which are usually named entities (more generally, noun phrases). Indeed, we see an improvement in mention detection as exemplified in the first example of Figure \[fig:analysis\]. Compared to the baseline, the fine-tuned model identifies a larger number of entities, including “southern hemisphere”, “Cambridge” and “Oxford”, which are missed by the baseline model.

#### Better linking

As a direct consequence of the above, the model is inclined to also link noun phrases that are not entities. In the second example of Figure \[fig:analysis\], we see that “This attempt” is linked to “releasing” by the fine-tuned model. Interestingly, we do not see this type of *eventive* noun phrase linking either in OntoNotes or in the predictions of the baseline model.

This phenomenon, however, also has a side-effect of producing singleton clusters and spurious linking, which adversely affect the recall. On the OntoNotes test data, while the average precision of the best performing fine-tuned model is higher than the baseline (75.62 vs. 73.80), a drop in recall (70.75 vs. 71.34) causes the final F$_1$ score to only marginally improve.

Related Work {#sec:related}
============

#### Coreference resolution

Among neural coreference resolvers [@Wu2017ADL; @Meng2018TriadbasedNN], @lee2017end were the first to propose an end-to-end resolver which did not rely on hand-crafted rules or a syntactic parser. Extending this work, @lee2018higher introduced a novel attention mechanism for iteratively ranking spans of candidate coreferent mentions, thereby improving the identification of long distance coreference chains. improve pronoun coreference resolution by 2.2 F1 using linguistic features (gender, animacy and plurality) and a frequency based predicate-argument selection preference as external knowledge. incorporate knowledge into coreference resolution by means of information retrieval, finding sentences that are syntactically similar to a given instance, and improving F1 by 0.16.

#### Reinforcement learning

RL has been used for many NLP tasks, including coreference resolution [@clark2016deep] and relation extraction [@Zeng2018LargeSR]. @clark2016deep use RL to improve coreference resolution by optimizing their mention ranking model and directly use the standard evaluation metrics as the rewards. We, on the other hand, perform end-to-end optimization by rewarding the model’s consistency with real world knowledge using relation extraction. To our knowledge, we are the first to use consistency with world knowledge as a reward for tasks other than knowledge base construction.[^8]

#### Knowledge bases

Knowledge bases have been leveraged across multiple tasks across NLP [@Bordes2011LearningSE; @Chang2014TypedTD; @Lin2015ModelingRP; @Toutanova2015RepresentingTF; @Yang2017LeveragingKB]. Specifically for coreference resolution, @Prokofyev2015SANAP implement a resolver that ensures semantic relatedness of resulting coreference clusters by leveraging Semantic Web annotations. Their work incorporates knowledge graph information only in the final stage of the resolver’s pipeline, and not during training. In contrast, our work augments information from the knowledge base directly into the training pipeline. Also, they use DBpedia [@dbpedia07] as the ontology. Although both Wikidata and DBpedia are designed to support working with Wikipedia articles, DBpedia can be considered as a subset of Wikidata as Wikipedia infoboxes are its main data source. The advantage of Wikidata over DBpedia is its size, and the fact that it is multilingual, which will allow applying our method to other languages in the future.

Conclusion {#sec:conclusion}
==========

We presented an architecture for adapting coreference resolvers by rewarding them for being consistent with world knowledge. Using simple multi-task reinforcement learning and a knowledge extraction pipeline, we achieved improvements over the state of the art across two datasets. We believe this is an important first step in exploring the usefulness of knowledge bases in the context of coreference resolution and other discourse-level phenomena. In this area, manually annotated data is particularly expensive, and we believe leveraging knowledge bases will eventually reduce the need for manual annotation.

Acknowlegments {#acknowlegments .unnumbered}
==============

We thank the reviewers for their valuable comments. Rahul Aralikatte, Daniel Hershcovich, Heather Lent, and Anders S[ø]{}gaard are funded by a Google Focused Research Award. Heather Lent is also funded by the European Union’s Horizon 2020 research and innovation programme under the Marie Sk[ł]{}odowska-Curie grant agreement No. 801199. Chen Qiu is funded in part by the National Natural Science Foundation of China under grant No. 61773355 and the China Scholarship Council.

[^1]: <https://www.wikipedia.org>

[^2]: <https://www.wikidata.org>

[^3]: <https://github.com/kentonl/e2e-coref>

[^4]: <https://catalog.ldc.upenn.edu/LDC2013T19>

[^5]: That is, we retain only those triples whose subject and object can be linked to an entity in Wikidata.

[^6]: The models were re-trained from scratch, and the scores are slightly different from those reported in .

[^7]: We repeat this experiment three times with different random seeds and observed the same pattern and very robust performance across the board.

[^8]: , for example, use reinforcement learning with consistency-like reward to induce lexical taxonomies.
