Model Evaluation for Domain Identification of Unknown Classes in
  Open-World Recognition: A Proposal
Gusti Ahmad Fanshuri Alfarisy Owais Ahmed Malik Ong Wee Hong
Open-World Recognition (OWR) is an emerging field that makes a machine
learning model competent in rejecting the unknowns, managing them, and
incrementally adding novel samples to the base knowledge. However, this broad
objective is not practical for an agent that works on a specific task. Not all
rejected samples will be used for learning continually in the future. Some
novel images in the open environment may not belong to the domain of interest.
Hence, identifying the unknown in the domain of interest is essential for a
machine learning model to learn merely the important samples. In this study, we
propose an evaluation protocol for estimating a model's capability in
separating unknown in-domain (ID) and unknown out-of-domain (OOD). We evaluated
using three approaches with an unknown domain and demonstrated the possibility
of identifying the domain of interest using the pre-trained parameters through
traditional transfer learning, Automated Machine Learning (AutoML), and Nearest
Class Mean (NCM) classifier with First Integer Neighbor Clustering Hierarchy
(FINCH). We experimented with five different domains: garbage, food, dogs,
plants, and birds. The results show that all approaches can be used as an
initial baseline yielding a good accuracy. In addition, a Balanced Accuracy
(BACCU) score from a pre-trained model indicates a tendency to excel in one or
more domains of interest. We observed that MobileNetV3 yielded the highest
BACCU score for the garbage domain and surpassed complex models such as the
transformer network. Meanwhile, our results also suggest that a strong
representation in the pre-trained model is important for identifying unknown
classes in the same domain. This study could open the bridge toward open-world
recognition in domain-specific tasks where the relevancy of the unknown classes
is vital.