OneCAD: One Classifier for All image Datasets using multimodal learning
Shakti N. Wadekar Eugenio Culurciello
Vision-Transformers (ViTs) and Convolutional neural networks (CNNs) are
widely used Deep Neural Networks (DNNs) for classification task. These model
architectures are dependent on the number of classes in the dataset it was
trained on. Any change in number of classes leads to change (partial or full)
in the model's architecture. This work addresses the question: Is it possible
to create a number-of-class-agnostic model architecture?. This allows model's
architecture to be independent of the dataset it is trained on. This work
highlights the issues with the current architectures (ViTs and CNNs). Also,
proposes a training and inference framework OneCAD (One Classifier for All
image Datasets) to achieve close-to number-of-class-agnostic transformer model.
To best of our knowledge this is the first work to use Mask-Image-Modeling
(MIM) with multimodal learning for classification task to create a DNN model
architecture agnostic to the number of classes. Preliminary results are shown
on natural and medical image datasets. Datasets: MNIST, CIFAR10, CIFAR100 and
COVIDx. Code will soon be publicly available on github.