Ensemble Distillation for Unsupervised Constituency Parsing
Behzad Shayegh Yanshuai Cao Xiaodan Zhu Jackie C. K. Cheung Lili Mou
We investigate the unsupervised constituency parsing task, which organizes
words and phrases of a sentence into a hierarchical structure without using
linguistically annotated data. We observe that existing unsupervised parsers
capture differing aspects of parsing structures, which can be leveraged to
enhance unsupervised parsing performance. To this end, we propose a notion of
"tree averaging," based on which we further propose a novel ensemble method for
unsupervised parsing. To improve inference efficiency, we further distill the
ensemble knowledge into a student model; such an ensemble-then-distill process
is an effective approach to mitigate the over-smoothing problem existing in
common multi-teacher distilling methods. Experiments show that our method
surpasses all previous approaches, consistently demonstrating its effectiveness
and robustness across various runs, with different ensemble components, and
under domain-shift conditions.