Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse
  Finetuning
Sarkar Snigdha Sarathi Das Ranran Haoran Zhang Peng Shi Wenpeng Yin Rui Zhang
Unified Sequence Labeling that articulates different sequence labeling
problems such as Named Entity Recognition, Relation Extraction, Semantic Role
Labeling, etc. in a generalized sequence-to-sequence format opens up the
opportunity to make the maximum utilization of large language model knowledge
toward structured prediction. Unfortunately, this requires formatting them into
specialized augmented format unknown to the base pretrained language model
(PLMs) necessitating finetuning to the target format. This significantly bounds
its usefulness in data-limited settings where finetuning large models cannot
properly generalize to the target format. To address this challenge and
leverage PLM knowledge effectively, we propose FISH-DIP, a sample-aware dynamic
sparse finetuning strategy that selectively focuses on a fraction of
parameters, informed by feedback from highly regressing examples, during the
fine-tuning process. By leveraging the dynamism of sparsity, our approach
mitigates the impact of well-learned samples and prioritizes underperforming
instances for improvement in generalization. Across five tasks of sequence
labeling, we demonstrate that FISH-DIP can smoothly optimize the model in low
resource settings offering upto 40% performance improvements over full
fine-tuning depending on target evaluation settings. Also, compared to
in-context learning and other parameter-efficient fine-tuning approaches,
FISH-DIP performs comparably or better, notably in extreme low-resource
settings.