NAPA-VQ: Neighborhood Aware Prototype Augmentation with Vector
  Quantization for Continual Learning
Tamasha Malepathirana Damith Senanayake Saman Halgamuge
Catastrophic forgetting; the loss of old knowledge upon acquiring new
knowledge, is a pitfall faced by deep neural networks in real-world
applications. Many prevailing solutions to this problem rely on storing
exemplars (previously encountered data), which may not be feasible in
applications with memory limitations or privacy constraints. Therefore, the
recent focus has been on Non-Exemplar based Class Incremental Learning (NECIL)
where a model incrementally learns about new classes without using any past
exemplars. However, due to the lack of old data, NECIL methods struggle to
discriminate between old and new classes causing their feature representations
to overlap. We propose NAPA-VQ: Neighborhood Aware Prototype Augmentation with
Vector Quantization, a framework that reduces this class overlap in NECIL. We
draw inspiration from Neural Gas to learn the topological relationships in the
feature space, identifying the neighboring classes that are most likely to get
confused with each other. This neighborhood information is utilized to enforce
strong separation between the neighboring classes as well as to generate old
class representative prototypes that can better aid in obtaining a
discriminative decision boundary between old and new classes. Our comprehensive
experiments on CIFAR-100, TinyImageNet, and ImageNet-Subset demonstrate that
NAPA-VQ outperforms the State-of-the-art NECIL methods by an average
improvement of 5%, 2%, and 4% in accuracy and 10%, 3%, and 9% in forgetting
respectively. Our code can be found in https://github.com/TamashaM/NAPA-VQ.git.