Inconsistency-Based Data-Centric Active Open-Set Annotation
Ruiyu Mao Ouyang Xu Yunhui Guo
Active learning is a commonly used approach that reduces the labeling effort
required to train deep neural networks. However, the effectiveness of current
active learning methods is limited by their closed-world assumptions, which
assume that all data in the unlabeled pool comes from a set of predefined known
classes. This assumption is often not valid in practical situations, as there
may be unknown classes in the unlabeled data, leading to the active open-set
annotation problem. The presence of unknown classes in the data can
significantly impact the performance of existing active learning methods due to
the uncertainty they introduce. To address this issue, we propose a novel
data-centric active learning method called NEAT that actively annotates
open-set data. NEAT is designed to label known classes data from a pool of both
known and unknown classes unlabeled data. It utilizes the clusterability of
labels to identify the known classes from the unlabeled pool and selects
informative samples from those classes based on a consistency criterion that
measures inconsistencies between model predictions and local feature
distribution. Unlike the recently proposed learning-centric method for the same
problem, NEAT is much more computationally efficient and is a data-centric
active open-set annotation method. Our experiments demonstrate that NEAT
achieves significantly better performance than state-of-the-art active learning
methods for active open-set annotation.