RaTE: a Reproducible automatic Taxonomy Evaluation by Filling the Gap
Tianjian Gao Phillipe Langlais
Taxonomies are an essential knowledge representation, yet most studies on
automatic taxonomy construction (ATC) resort to manual evaluation to score
proposed algorithms. We argue that automatic taxonomy evaluation (ATE) is just
as important as taxonomy construction. We propose RaTE, an automatic label-free
taxonomy scoring procedure, which relies on a large pre-trained language model.
We apply our evaluation procedure to three state-of-the-art ATC algorithms with
which we built seven taxonomies from the Yelp domain, and show that 1) RaTE
correlates well with human judgments and 2) artificially degrading a taxonomy
leads to decreasing RaTE score.