Parameterizing Context: Unleashing the Power of Parameter-Efficient
  Fine-Tuning and In-Context Tuning for Continual Table Semantic Parsing
Yongrui Chen Shenyu Zhang Guilin Qi Xinnan Guo
Continual table semantic parsing aims to train a parser on a sequence of
tasks, where each task requires the parser to translate natural language into
SQL based on task-specific tables but only offers limited training examples.
Conventional methods tend to suffer from overfitting with limited supervision,
as well as catastrophic forgetting due to parameter updates. Despite recent
advancements that partially alleviate these issues through semi-supervised data
augmentation and retention of a few past examples, the performance is still
limited by the volume of unsupervised data and stored examples. To overcome
these challenges, this paper introduces a novel method integrating
\textit{parameter-efficient fine-tuning} (PEFT) and \textit{in-context tuning}
(ICT) for training a continual table semantic parser. Initially, we present a
task-adaptive PEFT framework capable of fully circumventing catastrophic
forgetting, which is achieved by freezing the pre-trained model backbone and
fine-tuning small-scale prompts. Building on this, we propose a teacher-student
framework-based solution. The teacher addresses the few-shot problem using ICT,
which procures contextual information by demonstrating a few training examples.
In turn, the student leverages the proposed PEFT framework to learn from the
teacher's output distribution, and subsequently compresses and saves the
contextual information to the prompts, eliminating the need to store any
training examples. Experimental evaluations on two benchmarks affirm the
superiority of our method over prevalent few-shot and continual learning
baselines across various metrics.