Exploring Strategies for Modeling Sign Language Phonology
Lee Kezar Riley Carlin Tejas Srinivasan Zed Sehyr Naomi Caselli Jesse Thomason
Like speech, signs are composed of discrete, recombinable features called
phonemes. Prior work shows that models which can recognize phonemes are better
at sign recognition, motivating deeper exploration into strategies for modeling
sign language phonemes. In this work, we learn graph convolution networks to
recognize the sixteen phoneme "types" found in ASL-LEX 2.0. Specifically, we
explore how learning strategies like multi-task and curriculum learning can
leverage mutually useful information between phoneme types to facilitate better
modeling of sign language phonemes. Results on the Sem-Lex Benchmark show that
curriculum learning yields an average accuracy of 87% across all phoneme types,
outperforming fine-tuning and multi-task strategies for most phoneme types.