---
abstract: 'We describe the NYU-CUBoulder systems for the SIGMORPHON 2020 Task 0 on typologically diverse morphological inflection and Task 2 on unsupervised morphological paradigm completion. The former consists of generating morphological inflections from a lemma and a set of morphosyntactic features describing the target form. The latter requires generating entire paradigms for a set of given lemmas from raw text alone. We model morphological inflection as a sequence-to-sequence problem, where the input is the sequence of the lemma’s characters with morphological tags, and the output is the sequence of the inflected form’s characters. First, we apply a transformer model to the task. Second, as inflected forms share most characters with the lemma, we further propose a pointer-generator transformer model to allow easy copying of input characters. Our best performing system for Task 0 is placed 6th out of 23 systems. We further use our inflection systems as subcomponents of approaches for Task 2. Our best performing system for Task 2 is the 2nd best out of 7 submissions.'
author:
- |
    Assaf Singer\
    New York University\
    USA\
    `as12152@nyu.edu`\
    Katharina Kann\
    University of Colorado Boulder\
    USA\
    `katharina.kann@colorado.edu`\
bibliography:
- 'acl2020.bib'
title: |
    The NYU-CUBoulder Systems for\
    SIGMORPHON 2020 Task 0 and Task 2
---

Introduction
============

In morphologically rich languages, a word’s surface form reflects syntactic and semantic properties that are expressed by the word. For example, most English nouns have both singular and plural forms (e.g., *robot*/*robots*, *process*/*processes*), which are known as the inflected forms of the noun. Some languages display little inflection. In contrast, others have many inflections per base form or lemma: a Polish verb has nearly 100 inflected forms [@Janecki1998] and an Archi verb has around 1.5 million [@kibrik1998archi].

Morphological inflection is the task of, given an input word – a lemma – together with morphosyntactic features defining the target form, generating the indicated inflected form, cf. Figure \[fig:morphexample\]. Morphological inflection is a useful tool for many natural language processing tasks [@DBLP:journals/tacl/SeekerC15; @DBLP:conf/acl/CotterellSE16], especially in morphologically rich languages where handling inflected forms can reduce data sparsity [@DBLP:conf/acl/MinkovTS07].

  ----------- -------------- --------------------
  **Lemma**   **Features**   **Inflected form**
  hug         V;PST          hugged
  seel        V;3;SG;PRS     seels
  ----------- -------------- --------------------

The SIGMORPHON 2020 Shared Task consists of three separate tasks. We participate in Task 0 on typologically diverse morphological inflection [@vylomova2020sigmorphon] and Task 2 on unsupervised morphological paradigm completion [@kann-etal-2020-sigmorphon]. **Task 0** consists of generating morphological inflections from a lemma and a set of morphosyntactic features describing the target form. For this task, we implement a pointer-generator transformer model, based on the vanilla transformer model [@DBLP:conf/nips/VaswaniSPUJGKP17] and the pointer-generator model [@DBLP:conf/acl/SeeLM17]. After adding a copy mechanism to the transformer, it produces a final probability distribution as a combination of generating elements from its output vocabulary and copying elements – characters in our case – from the input. As most inflected forms derive their characters from the source lemma, the use of a mechanism for copying characters directly from the lemma has proven to be effective for morphological inflection generation, especially in the low resource setting [@DBLP:conf/acl/AharoniG17; @DBLP:journals/corr/MakarovRC17].

For our submissions, we further increase the size of all training sets by performing multi-task training on morphological inflection and morphological reinflection, i.e., the task of generating inflected forms from forms *different from the lemma*. For languages with small training sets, we also perform hallucination pretraining [@DBLP:conf/emnlp/AnastasopoulosN19], where we generate pseudo training instances for the task, based on suffixation and prefixation rules collected from the original dataset.

For **Task 2**, participants are given raw text and a source file with lemmas. The objective is to generate the complete paradigms for all lemmas. Our systems for this task consist of a combination of the official baseline system [@jin2020unsupervised] and our systems for Task 0. The baseline system finds inflected forms in the text, decides on the number of inflected forms per lemma, and produces pseudo training files for morphological inflection. Our inflection model then learns from these and, subsequently, generates all missing forms.

Related Work
============

#### SIGMORPHON and CoNLL–SIGMORPHON shared tasks.

In recent years, the SIGMORPHON and CoNLL–SIGMORPHON shared tasks have promoted research on computational morphology, with a strong focus on morphological inflection. Research related to those shared tasks includes @DBLP:conf/acl/KannS16, who used an LSTM [@DBLP:journals/neco/HochreiterS97] sequence-to-sequence model with soft attention [@DBLP:journals/corr/BahdanauCB14] and achieved the best result in the SIGMORPHON 2016 shared task [@DBLP:conf/sigmorphon/KannS16; @DBLP:conf/sigmorphon/CotterellKSYEH16]. Due to the often monotonic alignment between input and output, @DBLP:conf/acl/AharoniG17 proposed a model with hard monotonic attention. Based on this, @DBLP:journals/corr/MakarovRC17 implemented a neural state-transition system which also used hard monotonic attention and achieved the best results for Task 1 of the SIGMORPHON 2017 shared task. In 2018, the best results were achieved by a revised version of the neural transducer, trained with imitation learning [@DBLP:conf/conll/MakarovC18]. That model learned an alignment instead of maximizing the likelihood of gold action sequences given by a separate aligner.

#### Transformers.

Transformers have produced state-of-the-art results on various tasks such as machine translation [@DBLP:conf/nips/VaswaniSPUJGKP17] language modeling [@DBLP:conf/aaai/Al-RfouCCGJ19], question answering [@DBLP:conf/naacl/DevlinCLT19] and language understanding [@DBLP:conf/naacl/DevlinCLT19]. There has been very little work on transformers for morphological inflection, with, to the best of our knowledge, @DBLP:journals/corr/abs-2005-01630 being the only published paper. However, the widespread success of transformers in NLP leads us to believe that a transformer model could perform well on morphological inflection.

#### Pointer-generators.

In addition to the transformer, the architecture of our model is also inspired by @DBLP:conf/acl/SeeLM17, who used a pointer-generator network for abstractive summarization. Their model could choose between generating a new element and copying an element from the input directly to the output. This copying of words from the source text via pointing [@DBLP:conf/nips/VinyalsKKPSH15], improved the handling of out-of-vocabulary words. Copy mechanisms have also been used for other tasks, including morphological inflection [@DBLP:conf/conll/SharmaKS18]. *Transformers* with copy mechanisms have been used for word-level tasks [@DBLP:conf/naacl/ZhaoWSJL19], but, as far as we know, never before on the character level.

SIGMORPHON 2020 Shared Task
===========================

The SIGMORPHON 2020 Shared Task is composed of three tasks: Task 0 on typologically diverse morphological inflection [@vylomova2020sigmorphon], Task 1 on multilingual grapheme-to-phoneme conversion [@Task1-2020-sigmorphon], and Task 2 on unsupervised morphological paradigm completion [@kann-etal-2020-sigmorphon]. We submit systems to Tasks 0 and 2.

Task 0: Typologically Diverse Morphological Inflection
------------------------------------------------------

SIGMORPHON 2020 Task 0 focuses on morphological inflection in a set of typologically diverse languages. Different languages inflect differently, so it is not trivially clear that systems that work on some languages also perform well on others. For Task 0, systems need to generalize well to a large group of languages, including languages unseen during model development.

The task features 90 languages in total. 45 of them are development languages, coming from five families: Austronesian, Niger–Congo, Uralic, Oto-Manguean, and Indo-European. The remaining 45 are surprise languages, and many of those are from language families different from the development languages. Some languages have very small training sets, which makes them hard to model. For those cases, the organizers recommend a family-based multilingual approach to exploit similarities between related languages. While this might be effective, we believe that using multitask training in combination with hallucination pretraining can give the model enough information to learn the task well, while staying true to the specific structure of each individual language.

Task 2: Unsupervised Morphological Paradigm Completion
------------------------------------------------------

Task 2 is a novel task, designed to encourage work on unsupervised methods for computational morphology. As morphological annotations are limited for many of the world’s languages, the study of morphological generation in the low-resource setting is of great interest [@DBLP:conf/conll/CotterellKSWVMK18]. However, a different way to tackle the problem is by creating systems that are able to use data without annotations.

For Task 2, a tokenized Bible in each language is given to the participants, along with a list of lemmas. Participants should then produce complete paradigms for each lemma. As slots in the paradigm are not labeled with gold data paradigm slot descriptions, an evaluation metric called best-match accuracy was designed for this task. First, this metric matches predicted paradigm slots with gold slots in the way which leads to the highest overall accuracy. It then evaluates the correctness of individual inflected forms.

Methods
=======

In this section, we introduce our models for Tasks 0 and 2 and describe all approaches we use, such as multitask training, hallucination pretraining and ensembling. The code for our models is available online.[^1]

Transformer
-----------

Our model is built on top of the transformer architecture [@DBLP:conf/nips/VaswaniSPUJGKP17]. It consists of an encoder and a decoder, each composed of a stack of layers. Each encoder layer consists, in turn, of a self-attention layer, followed by a fully connected layer. Decoder layers contain an additional inter-attention layer between the two.

With inputs $\left( x_1, \cdots, x_T \right)$ being a lemma’s characters followed by tags representing the morphosyntactic features of the target form, the encoder processes the input sequence and outputs hidden states $\left( h_1, \cdots, h_T \right)$. At generation step $t$, the decoder reads the previously generated sequence $\left( y_1, \cdots, y_{t-1} \right)$ to produce states $\left( s_1, \cdots, s_{t-1} \right)$. The last decoder state $s_{t-1}$ is then passed through a linear layer followed by a softmax, to generate a probability distribution over the output vocabulary: $${P_{\text{vocab}}}= \textrm{softmax}(V s_{t-1} + b)$$ During training, the entire target sequence $\left( y_1, \cdots, y_{T_y} \right)$ is input to the decoder at once, along with a sequential mask to prevent positions from attending to subsequent positions.

Pointer-Generator Transformer
-----------------------------

The pointer-generator transformer allows for both generating characters from a fixed vocabulary, as well as copying from the source sequence via pointing [@DBLP:conf/nips/VinyalsKKPSH15]. This is managed by ${p_{\text{gen}}}$ – the probability of generating as opposed to copying – which acts as a soft switch between the two actions. ${p_{\text{gen}}}$ is computed by passing a concatenation of the decoder state $s_{t}$, the previously generated output $y_{t-1}$, and a context vector $c_t$ through a linear layer, followed by the sigmoid function. $${p_{\text{gen}}}= \sigma(w [s_t;c_t;y_{t-1}] + b)$$ The context vector is computed as the weighted sum of the encoder hidden states $$c_t = \sum\nolimits_{i=1}^{T} a^t_i h_i$$ with attention weights $\left( a^t_1, \cdots, a^t_T \right)$. For each inflection example, let the extended vocabulary denote the union of the output vocabulary, and all characters appearing in the source lemma. We then use ${p_{\text{gen}}}$, ${P_{\text{vocab}}}$ produced by the transformer, and the attention weights of the last decoder layer $\left( a^t_1, \cdots, a^t_T \right)$ to compute a distribution over the extended vocabulary: $$P(c) = {p_{\text{gen}}}{P_{\text{vocab}}}(c) + (1-{p_{\text{gen}}}) {P_{\text{copy}}}(c),$$ with $${P_{\text{copy}}}(c) = \sum\nolimits_{i: x_i = c} a^t_i$$ The copy distribution ${P_{\text{copy}}}(c)$ for each character $c$ is the sum of attention weights over all source positions where $x_i = c$. Note that if $c$ is an out-of-vocabulary (OOV) character, then ${P_{\text{vocab}}}(c)$ is zero; similarly, if $c$ does not appear in the source lemma, then $\sum\nolimits_{i: x_i = c} a^t_i$ is zero. The ability to produce OOV characters is one of the primary advantages of pointer-generator models; by contrast models such as our vanilla transformer are restricted to their pre-set vocabulary.

Multitask Training
------------------

  -- --------- --------- ------------ --
     grip      grips     V;SG;3;PRS   
     grip      gripped   V;PST        
     grips     grip      V;LEMMA      
     grips     gripped   V;PST        
     gripped   grip      V;LEMMA      
  -- --------- --------- ------------ --

Some languages in Task 0 have small training sets, which makes them hard to model. In order to handle that, we perform multitask training, and, thereby, increase the amount of examples available for training.

#### Morphological reinflection.

Morphological reinflection is a generalized version of the morphological inflection task, which consists of producing an inflected form for any given source form – i.e., not necessarily the lemma –, and target tag. For example:

$$\begin{aligned}
\textrm{(hugging; V;PST)} \rightarrow \textrm{hugged}.    \end{aligned}$$

This is a more complex task, since a model needs to infer the underlying lemma of the source form in order to inflect it correctly to the desired form.

Many morphological inflection datasets contain lemmas that are converted to several inflected forms. Treating separate instances for the same source lemma as independent is missing an opportunity to utilize the connection between the different inflected forms. We approach this by converting our morphological inflection training set into one for morphological reinflection as described in the following.

#### From inflection to reinflection.

Inflected forms of the same lemma are grouped together to sets of one or more (inflected form, morphological features) pairs. Then, for each set, we create new training instances by inflecting all forms to one another, as shown in Figure \[fig:multitask-exmp\]. We also let the model inflect forms back to the lemma by adding the lemma as one of the inflected forms, marked with the synthetically generated LEMMA tag. The new training set fully utilizes the connections between different forms in the paradigm, and, in that way, provides more training instances to our model.

Hallucination Pretraining
-------------------------

Another effective tool to improve training in the low-resource setting is data hallucination [@DBLP:conf/emnlp/AnastasopoulosN19]. Using hallucination, new pseudo-instances are generated for training, based on suffixation and prefixation rules collected from the original dataset. For languages with less than 1000 training instances, we pretrain our models on a hallucinated training set consisting of 10,000 instances, before training on the multitask training set.

Submissions and Ensembling Strategies
-------------------------------------

  **Hyperparameter**           **Value**
  -------------------------- -----------
  Embedding dimension                256
  Encoder layers                       4
  Decoder layers                       4
  Encoder hidden dimension          1024
  Decoder hidden dimension          1024
  Attention heads                      4

  : The hyperparameters used in our inflection models for both Task 0 and Task 2.[]{data-label="tab:hyperparam-table"}

We submit 4 different systems for Task 0. NYU-CUBoulder-2 consists of one pointer-generator transformer model, and, for NYU-CUBoulder-4, we train one vanilla transformer. Those two are our simplest systems and can be seen as baselines for our other submissions.

Because of the effects of random initialization in non-convex objective functions, we further use ensembling in combination with both architectures: NYU-CUBoulder-1 is an ensemble of three pointer-generator transformers, and NYU-CUBoulder-3 is an ensemble of five pointer-generator transformers. The final decision is made by majority voting. In case of a tie, the answer is chosen randomly among the most frequent predictions. Models participating in the ensembles are from different epochs during the same training run.

As previously stated, all systems are trained on the augmented multitask training sets, and systems trained on languages with less than 1000 training instances were pretrained on the hallucinated datasets.

Task 2: Model description
-------------------------

Our systems for Task 2 consist of a combination of the official baseline system [@jin2020unsupervised] and our inflection systems for Task 0. The system is given raw text and a source file with lemmas, and generates the complete paradigm of each lemma. The baseline system finds inflected forms in the text, decides on the number of inflected forms per lemma, and produces pseudo training files for morphological inflection. Any inflections that the system has not found in the raw text are given as test instances. Our inflection model then learns from the files and, subsequently, generates all missing forms. We use the pointer-generator and vanilla transformers as our inflection models.

For Task 2, we use ensembling for all submissions. NYU-CUBoulder-1 is an ensemble of six pointer-generator transformers, NYU-CUBoulder-2 is an ensemble of six vanilla transformers, and NYU-CUBoulder-3 is an ensemble of all twelve models. For all models in both tasks, we use the hyperparameters described in Table \[tab:hyperparam-table\].

Experiments
===========

Task 0
------

#### Data.

The dataset for Task 0 covers 90 languages in total: 45 development languages and 45 surprise languages. For details on the official dataset please refer to @vylomova2020sigmorphon.

#### Baselines.

This year, several baselines are provided for the task. The first system has also been used as a baseline in previous shared tasks on morphological reinflection [@DBLP:conf/conll/CotterellKSWVXF17; @DBLP:conf/conll/CotterellKSWVMK18]. It is a non-neural system which first scans the dataset to extract suffix- or prefix-based lemma-to-form transformations. Then, based on the morphological tag at inference time, it applies the most frequent suitable transformation to an input lemma to yield the output form [@DBLP:conf/conll/CotterellKSWVXF17]. The other two baselines are neural models. One is a transformer [@DBLP:conf/nips/VaswaniSPUJGKP17; @wu2020applying], and the second one is a hard-attention model [@DBLP:conf/acl/WuC19], which enforces strict monotonicity and learns a latent alignment while learning to transduce. To account for the low-resource settings for some languages, the organizers also employ two additional methods: constructing a multilingual model trained for all languages belonging to each language family [@kann-etal-2017-one], and data augmentation using hallucination [@DBLP:conf/emnlp/AnastasopoulosN19]. Four model types are trained for each neural architecture: a plain model, a family-multilingual model, a data augmented model, and an augmented family-multilingual model. Overall, there are nine baseline systems for each language. We compare our models to an oracle baseline by choosing the best score over all baseline systems for each language.

#### Results.

[lr|r|r|r|r]{} & Sub-1 & Sub-2 & Sub-3 & Sub-4 & Base\
\
Low & **88.71** & 88.02 & 84.90 & 84.07 & -\
Other & 90.46 & 90.63 & 90.20 & **90.94** & -\
All & **90.06** & 90.02 & 88.96 & 89.34 & -\
\
Low & 84.8 & 84.8 & 85.5 & 83.9 & **89.77**\
Other & 89.7 & 89.8 & 89.8 & 90.2 & **92.43**\
All & 88.6 & 88.7 & 88.8 & 88.8 & **91.81**\

[lll|ll|ll|ll|ll]{} System & & & & &\
\
& slots & macro & slots & macro & slots & macro & slots & macro & slots & macro\
Basque & 30 & 0.0006 & 27 & 0.0006 & 30 & 0.0005 & 30 & 0.0005 & 30 & **0.0007**\
Bulgarian & 35 & 0.283 & 34 & **0.3169** & 35 & 0.2769 & 35 & 0.2894 & 35 & 0.2789\
English & 4 & 0.656 & 4 & **0.662** & 4 & 0.502 & 4 & 0.528 & 4 & 0.512\
Finnish & 21 & 0.0533 & 21 & **0.055** & 21 & 0.0536 & 21 & 0.0547 & 21 & 0.0535\
German & 9 & 0.2835 & 9 & **0.29** & 9 & 0.273 & 9 & 0.2735 & 9 & 0.2735\
Kannada & 172 & 0.1549 & 172 & **0.1512** & 172 & 0.111 & 172 & 0.1116 & 172 & 0.111\
Navajo & 3 & 0.0323 & 3 & **0.0327** & 3 & 0.004 & 3 & 0.0043 & 3 & 0.0043\
Spanish & 29 & 0.2296 & 29 & **0.2367** & 29 & 0.2039 & 29 & 0.2056 & 29 & 0.203\
Turkish & 104 & 0.1421 & 104 & **0.1553** & 104 & 0.1488 & 104 & 0.1539 & 104 & 0.1513\
**All** & & 0.2039 & & **0.2112** & & 0.1749 & & 0.1802 & & 0.1765\

Our results for Task 0 are displayed in Table \[tab:task0-results\]. All four systems produce relatively similar results. NYU-CUBoulder-3, our five-model ensemble, performs best overall with $88.8\%$ accuracy on average. We further look at the results for low-resource ($<$ 1000 training examples) and high-resource ($>=$ 1000 training examples) languages separately. This way, we are able to see the advantage of the pointer-generator transformer in the low-resource setting, where all pointer-generator systems achieve an at least $0.9\%$ higher accuracy than the vanilla transformer model. However, in the setting where training data is abundant, the effect of the copy mechanism vanishes, as NYU-CUBoulder-4 – our only vanilla transformer – achieved the best results for our high-resource languages.

Task 2
------

#### Data.

For Task 2, a tokenized Bible in each language is given to the participants, along with a list of lemmas. Participants are required to construct the paradigms for all given lemmas.

The languages for Task 2 are again divided into development and test languages. Development languages are available for model development and hyperparameter tuning, but are not used during the final evaluation. The test languages are used for evaluation only, and do not have development sets. The development languages are: Maltese, Persian, Portuguese, Russian, Swedish. The test languages are: Basque, Bulgarian, English, Finnish, German, Kannada, Navajo, Spanish and Turkish.

#### Baselines.

The baseline system for the task is composed of four components, eventually producing morphological paradigms [@jin2020unsupervised]. The first three modules perform edit tree [@Chrupala2020] retrieval, additional lemma retrieval from the corpus, and paradigm size discovery, using distributional information. After the first three steps, pseudo training and test files for morphological inflection are produced. Finally, the non-neural Task 0 baseline system [@DBLP:conf/conll/CotterellKSWVXF17] or the neural transducer by @DBLP:conf/conll/MakarovC18 are used to create missing inflected forms.

#### Results.

Systems for Task 2 are evaluated using macro-averaged best-match accuracy [@jin2020unsupervised]. Results are shown in in Table \[tab:task2-results-average\]. All three systems produce relatively similar results. NYU-CUBoulder-2, our vanilla transformer ensemble, performed slightly better overall with an average best-match accuracy of $18.02\%$. Since our system is close to the baseline models, it performs similarly, achieving slightly worse results. For Basque, our all-round ensemble NYU-CUBoulder-2 outperformed both baselines with a best-match accuracy of $00.07\%$, achieving the highest result in the shared task.

Low-resource Setting
--------------------

As most inflected forms derive their characters from the source lemma, the use of a mechanism for copying characters directly from the lemma has proven to be effective for morphological inflection generation, especially in the low-resource setting [@DBLP:conf/acl/AharoniG17; @DBLP:journals/corr/MakarovRC17]. As all Task 0 datasets are fairly large, we further design a low-resource experiment to investigate the effectiveness of our model.

  System       Trm   Trm-PG    Baseline
  -------- ------- -------- -----------
  All        63.06    67.61   **70.06**

  : Results on the official development data for our low-resource experiment. Trm=Vanilla transformer, Trm-PG=Pointer-generator transformer, Baseline=neural transducer by @DBLP:conf/conll/MakarovC18.[]{data-label="tab:low-res-results-average"}

#### Data.

We simulate a low-resource setting by sampling 100 instances from all languages that we already consider low-resource, i.e., all languages with less than 1000 training instances. We then keep their development and test sets unchanged. Overall, we perform this experiment on 21 languages.

#### Experimental setup.

We train a pointer-generator transformer and a vanilla transformer on the modified datasets to examine the effects of the copy mechanism. We keep the hyperparameters unchanged, i.e., they are as mentioned in Table \[tab:hyperparam-table\]. We use a majority-vote ensemble consisting of 5 individual models for each architecture.

#### Baseline.

We additionally train the neural transducer by @DBLP:conf/conll/MakarovC18, which has achieved the best results for the 2018 shared task in the low-resource setting [@DBLP:conf/conll/CotterellKSWVMK18]. The neural transducer uses hard monotonic attention [@DBLP:conf/acl/AharoniG17] and transduces the lemma into the inflected form by a sequence of explicit edit operations. It is trained with an imitation learning method [@DBLP:conf/conll/MakarovC18]. We use this model as a reference for the state of the art in the low-resource setting.

#### Results.

As seen in Table \[tab:low-res-results-average\], for the low-resource dataset, the pointer-generator transformer clearly outperforms the vanilla transformer by an average accuracy of $4.46\%$. For some languages, such as Chichicapan Zapotec, the difference is up to $14\%$. While the neural transducer achieves a higher accuracy, our model performs only $2.45\%$ worse than this state-of-the-art model.[^2] We are also able to observe the use of the copy mechanism for copying of OOV characters in the test sets of some languages.

Ablation Studies
================

  Model:             1   2   3   4   5
  ----------------- --- --- --- --- ---
  Copy                              
  Multitask Train                   
  Hallucination                     

  : System components for the ablation study for Task 0. Each model is a transformer which contains a combination of the following components: copy mechanism, multitask training and hallucination pretraining.[]{data-label="tab:ablation-models"}

Our systems use three components on top of the vanilla transformer: a copy mechanism, multitask training and hallucination pretraining. We further perform an ablation study to measure the contribution of each component to the overall system performance. For this, we additionally train five different systems with different combinations of components. A description of which component is used in which system for this ablation study is shown in Table \[tab:ablation-models\].

Results
-------

#### Copy mechanism.

Comparing models 2 and 4, which are both trained on the original dataset, pretrained with hallucination and differ only by the use of the copy mechanism, we are able to see that adding this component slightly improves performance by $0.06-0.16\%$. When comparing models 1 and 3, the copy mechanism decreases performance slightly by $0.3\%$ for the high-resource languages and $0.11\%$ overall, but increases performance for low-resource languages by $0.68\%$.

#### Multitask training.

Unlike the copy mechanism, multitask training actually consistently decreases the performance of the models. Looking at models 1 and 2, training the pointer-generator transformer on the multitask dataset decreases accuracy by $1.8-2.03\%$ for all three language groups. The same happens for the vanilla transformer with an accuracy decrease of $1.67-2.32\%$. A possible explanation are the relatively large training sets provided for the shared task, as this method is more suitable for the low-resource setting.

#### Hallucination pretraining.

In order to examine the effect of hallucination pretraining on our submitted models, we now compare the pointer-generator transformers trained on the multitask data with and without hallucination pretraining (models 1 and 5). Hallucination pretraining shows to be helpful: it increases the accuracy on low-resource languages by $1.85\%$. The performance on the high-resource languages is necessarily the same, as only models for low-resource languages are actually pretrained.

[lc|c|c|c|c]{} [Model:]{} & 1 & 2 & 3 & 4 & 5\
\
Low & 88.20 & **90.00** & 87.52 & 89.84 & 86.35\
Other & 90.63 & **92.66** & 90.93 & 92.60 & 90.63\
All & 90.02 & **92.04** & 90.13 & 91.96 & 89.63\

Conclusion
==========

We presented the NYU-CUBoulder submissions for SIGMORPHON 2020 Task 0 and Task 2.

We developed morphological inflection models, based on a transformer and a new model for the task, a pointer-generator transformer, which is a transformer-analogue of a pointer-generator model. For Task 0, we further added multitask training and hallucination pretraining. For Task 2, we combined our inflection models with additional components from the provided baseline to obtain a fully functional system for unsupervised morphological paradigm completion.

We performed an ablation study to examine the effects of all components of our inflection system. Finally, we designed a low-resource experiment to show that using the copy mechanism on top of the vanilla transformer is beneficial if training sets are small, and achieved results close to a state-of-the-art model for low-resource morphological inflection.

Acknowledgments {#acknowledgments .unnumbered}
===============

We would like to thank the organizers of SIGMORPHON 2020 Task 0 and Task 2.

[^1]: [github.com/AssafSinger94/sigmorphon-2020-inflection](https://github.com/AssafSinger94/sigmorphon-2020-inflection)

[^2]: We could probably obtain better results with appropriate hyperparameter tuning.
