Cross-functional Analysis of Generalisation in Behavioural Learning
Pedro Henrique Luz de Araujo Benjamin Roth
In behavioural testing, system functionalities underrepresented in the
standard evaluation setting (with a held-out test set) are validated through
controlled input-output pairs. Optimising performance on the behavioural tests
during training (behavioural learning) would improve coverage of phenomena not
sufficiently represented in the i.i.d. data and could lead to seemingly more
robust models. However, there is the risk that the model narrowly captures
spurious correlations from the behavioural test suite, leading to
overestimation and misrepresentation of model performance -- one of the
original pitfalls of traditional evaluation. In this work, we introduce BeLUGA,
an analysis method for evaluating behavioural learning considering
generalisation across dimensions of different granularity levels. We optimise
behaviour-specific loss functions and evaluate models on several partitions of
the behavioural test suite controlled to leave out specific phenomena. An
aggregate score measures generalisation to unseen functionalities (or
overfitting). We use BeLUGA to examine three representative NLP tasks
(sentiment analysis, paraphrase identification and reading comprehension) and
compare the impact of a diverse set of regularisation and domain generalisation
methods on generalisation performance.