F-MALLOC: Feed-forward Memory Allocation for Continual Learning in
  Neural Machine Translation
Junhong Wu Yuchen Liu Chengqing Zong
In the evolving landscape of Neural Machine Translation (NMT), the
pretrain-then-finetune paradigm has yielded impressive results. However, the
persistent challenge of Catastrophic Forgetting (CF) remains a hurdle. While
previous work has introduced Continual Learning (CL) methods to address CF,
these approaches grapple with the delicate balance between avoiding forgetting
and maintaining system extensibility. To address this, we propose a CL method,
named $\textbf{F-MALLOC}$ ($\textbf{F}$eed-forward $\textbf{M}$emory
$\textbf{ALLOC}ation)$. F-MALLOC is inspired by recent insights highlighting
that feed-forward layers emulate neural memories and encapsulate crucial
translation knowledge. It decomposes feed-forward layers into discrete memory
cells and allocates these memories to different tasks. By learning to allocate
and safeguard these memories, our method effectively alleviates CF while
ensuring robust extendability. Besides, we propose a comprehensive assessment
protocol for multi-stage CL of NMT systems. Experiments conducted following
this new protocol showcase the superior performance of F-MALLOC, evidenced by
higher BLEU scores and almost zero forgetting.