Word segmentation granularity in Korean
Jungyeul Park Mija Kim
This paper describes word {segmentation} granularity in Korean language
processing. From a word separated by blank space, which is termed an eojeol, to
a sequence of morphemes in Korean, there are multiple possible levels of word
segmentation granularity in Korean. For specific language processing and corpus
annotation tasks, several different granularity levels have been proposed and
utilized, because the agglutinative languages including Korean language have a
one-to-one mapping between functional morpheme and syntactic category. Thus, we
analyze these different granularity levels, presenting the examples of Korean
language processing systems for future reference. Interestingly, the
granularity by separating only functional morphemes including case markers and
verbal endings, and keeping other suffixes for morphological derivation results
in the optimal performance for phrase structure parsing. This contradicts
previous best practices for Korean language processing, which has been the de
facto standard for various applications that require separating all morphemes.