Probabilistic Method of Measuring Linguistic Productivity
Sergei Monakhov
In this paper I propose a new way of measuring linguistic productivity that
objectively assesses the ability of an affix to be used to coin new complex
words and, unlike other popular measures, is not directly dependent upon token
frequency. Specifically, I suggest that linguistic productivity may be viewed
as the probability of an affix to combine with a random base. The advantages of
this approach include the following. First, token frequency does not dominate
the productivity measure but naturally influences the sampling of bases.
Second, we are not just counting attested word types with an affix but rather
simulating the construction of these types and then checking whether they are
attested in the corpus. Third, a corpus-based approach and randomised design
assure that true neologisms and words coined long ago have equal chances to be
selected. The proposed algorithm is evaluated both on English and Russian data.
The obtained results provide some valuable insights into the relation of
linguistic productivity to the number of types and tokens. It looks like
burgeoning linguistic productivity manifests itself in an increasing number of
types. However, this process unfolds in two stages: first comes the increase in
high-frequency items, and only then follows the increase in low-frequency
items.