Task-Agnostic Low-Rank Adapters for Unseen English Dialects
Zedian Xiao William Held Yanchen Liu Diyi Yang
Large Language Models (LLMs) are trained on corpora disproportionally
weighted in favor of Standard American English. As a result, speakers of other
dialects experience significantly more failures when interacting with these
technologies. In practice, these speakers often accommodate their speech to be
better understood. Our work shares the belief that language technologies should
be designed to accommodate the diversity in English dialects and not the other
way around. However, prior works on dialect struggle with generalizing to
evolving and emerging dialects in a scalable manner. To fill this gap, our
method, HyperLoRA, leverages expert linguistic knowledge to enable
resource-efficient adaptation via hypernetworks. By disentangling
dialect-specific and cross-dialectal information, HyperLoRA improves
generalization to unseen dialects in a task-agnostic fashion. Not only is
HyperLoRA more scalable in the number of parameters, but it also achieves the
best or most competitive performance across 5 dialects in a zero-shot setting.
In this way, our approach facilitates access to language technology for
billions of English dialect speakers who are traditionally underrepresented.