Estimating Lexical Complexity from Document-Level Distributions
Sondre Wold Petter Mæhlum Oddbjørn Hove
Existing methods for complexity estimation are typically developed for entire
documents. This limitation in scope makes them inapplicable for shorter pieces
of text, such as health assessment tools. These typically consist of lists of
independent sentences, all of which are too short for existing methods to
apply. The choice of wording in these assessment tools is crucial, as both the
cognitive capacity and the linguistic competency of the intended patient groups
could vary substantially. As a first step towards creating better tools for
supporting health practitioners, we develop a two-step approach for estimating
lexical complexity that does not rely on any pre-annotated data. We implement
our approach for the Norwegian language and verify its effectiveness using
statistical testing and a qualitative evaluation of samples from real
assessment tools. We also investigate the relationship between our complexity
measure and certain features typically associated with complexity in the
literature, such as word length, frequency, and the number of syllables.