Learning to Substitute Spans towards Improving Compositional
  Generalization
Zhaoyi Li Ying Wei Defu Lian
Despite the rising prevalence of neural sequence models, recent empirical
evidences suggest their deficiency in compositional generalization. One of the
current de-facto solutions to this problem is compositional data augmentation,
aiming to incur additional compositional inductive bias. Nonetheless, the
improvement offered by existing handcrafted augmentation strategies is limited
when successful systematic generalization of neural sequence models requires
multi-grained compositional bias (i.e., not limited to either lexical or
structural biases only) or differentiation of training sequences in an
imbalanced difficulty distribution. To address the two challenges, we first
propose a novel compositional augmentation strategy dubbed \textbf{Span}
\textbf{Sub}stitution (SpanSub) that enables multi-grained composition of
substantial substructures in the whole training set. Over and above that, we
introduce the \textbf{L}earning \textbf{to} \textbf{S}ubstitute \textbf{S}pan
(L2S2) framework which empowers the learning of span substitution probabilities
in SpanSub in an end-to-end manner by maximizing the loss of neural sequence
models, so as to outweigh those challenging compositions with elusive concepts
and novel surroundings. Our empirical results on three standard compositional
generalization benchmarks, including SCAN, COGS and GeoQuery (with an
improvement of at most 66.5\%, 10.3\%, 1.2\%, respectively), demonstrate the
superiority of SpanSub, %the learning framework L2S2 and their combination.