Leveraging AI Predicted and Expert Revised Annotations in Interactive
  Segmentation: Continual Tuning or Full Training?
Tiezheng Zhang Xiaoxi Chen Chongyu Qu Alan Yuille Zongwei Zhou
Interactive segmentation, an integration of AI algorithms and human
expertise, premises to improve the accuracy and efficiency of curating
large-scale, detailed-annotated datasets in healthcare. Human experts revise
the annotations predicted by AI, and in turn, AI improves its predictions by
learning from these revised annotations. This interactive process continues to
enhance the quality of annotations until no major revision is needed from
experts. The key challenge is how to leverage AI predicted and expert revised
annotations to iteratively improve the AI. Two problems arise: (1) The risk of
catastrophic forgetting--the AI tends to forget the previously learned classes
if it is only retrained using the expert revised classes. (2) Computational
inefficiency when retraining the AI using both AI predicted and expert revised
annotations; moreover, given the dominant AI predicted annotations in the
dataset, the contribution of newly revised annotations--often account for a
very small fraction--to the AI training remains marginal. This paper proposes
Continual Tuning to address the problems from two perspectives: network design
and data reuse. Firstly, we design a shared network for all classes followed by
class-specific networks dedicated to individual classes. To mitigate
forgetting, we freeze the shared network for previously learned classes and
only update the class-specific network for revised classes. Secondly, we reuse
a small fraction of data with previous annotations to avoid over-computing. The
selection of such data relies on the importance estimate of each data. The
importance score is computed by combining the uncertainty and consistency of AI
predictions. Our experiments demonstrate that Continual Tuning achieves a speed
16x greater than repeatedly training AI from scratch without compromising the
performance.