The Sem-Lex Benchmark: Modeling ASL Signs and Their Phonemes
Lee Kezar Elana Pontecorvo Adele Daniels Connor Baer Ruth Ferster Lauren Berger Jesse Thomason Zed Sevcikova Sehyr Naomi Caselli
Sign language recognition and translation technologies have the potential to
increase access and inclusion of deaf signing communities, but research
progress is bottlenecked by a lack of representative data. We introduce a new
resource for American Sign Language (ASL) modeling, the Sem-Lex Benchmark. The
Benchmark is the current largest of its kind, consisting of over 84k videos of
isolated sign productions from deaf ASL signers who gave informed consent and
received compensation. Human experts aligned these videos with other sign
language resources including ASL-LEX, SignBank, and ASL Citizen, enabling
useful expansions for sign and phonological feature recognition. We present a
suite of experiments which make use of the linguistic information in ASL-LEX,
evaluating the practicality and fairness of the Sem-Lex Benchmark for isolated
sign recognition (ISR). We use an SL-GCN model to show that the phonological
features are recognizable with 85% accuracy, and that they are effective as an
auxiliary target to ISR. Learning to recognize phonological features alongside
gloss results in a 6% improvement for few-shot ISR accuracy and a 2%
improvement for ISR accuracy overall. Instructions for downloading the data can
be found at https://github.com/leekezar/SemLex.