Analyzing Text Representations by Measuring Task Alignment
Cesar Gonzalez-Gutierrez Audi Primadhanty Francesco Cazzaro Ariadna Quattoni
Textual representations based on pre-trained language models are key,
especially in few-shot learning scenarios. What makes a representation good for
text classification? Is it due to the geometric properties of the space or
because it is well aligned with the task? We hypothesize the second claim. To
test it, we develop a task alignment score based on hierarchical clustering
that measures alignment at different levels of granularity. Our experiments on
text classification validate our hypothesis by showing that task alignment can
explain the classification performance of a given representation.