On the Transferability of Visually Grounded PCFGs
Yanpeng Zhao Ivan Titov
There has been a significant surge of interest in visually grounded grammar
induction in recent times. While a variety of models have been developed for
the task and have demonstrated impressive performance, they have not been
evaluated on text domains that are different from the training domain, so it is
unclear if the improvements brought by visual groundings are transferable. Our
study aims to fill this gap and assess the degree of transferability. We start
by extending VC-PCFG (short for Visually-grounded Compound
PCFG~\citep{zhao-titov-2020-visually}) in such a way that it can transfer
across text domains. We consider a zero-shot transfer learning setting where a
model is trained on the source domain and is directly applied to target
domains, without any further training. Our experimental results suggest that:
the benefits from using visual groundings transfer to text in a domain similar
to the training domain but fail to transfer to remote domains. Further, we
conduct data and result analysis; we find that the lexicon overlap between the
source domain and the target domain is the most important factor in the
transferability of VC-PCFG.