More for Less: Compact Convolutional Transformers Enable Robust Medical
  Image Classification with Limited Data
Andrew Kean Gao
Transformers are very powerful tools for a variety of tasks across domains,
from text generation to image captioning. However, transformers require
substantial amounts of training data, which is often a challenge in biomedical
settings, where high quality labeled data can be challenging or expensive to
obtain. This study investigates the efficacy of Compact Convolutional
Transformers (CCT) for robust medical image classification with limited data,
addressing a key issue faced by conventional Vision Transformers - their
requirement for large datasets. A hybrid of transformers and convolutional
layers, CCTs demonstrate high accuracy on modestly sized datasets. We employed
a benchmark dataset of peripheral blood cell images of eight distinct cell
types, each represented by approximately 2,000 low-resolution (28x28x3 pixel)
samples. Despite the dataset size being smaller than those typically used with
Vision Transformers, we achieved a commendable classification accuracy of
92.49% and a micro-average ROC AUC of 0.9935. The CCT also learned quickly,
exceeding 80% validation accuracy after five epochs. Analysis of per-class
precision, recall, F1, and ROC showed that performance was strong across cell
types. Our findings underscore the robustness of CCTs, indicating their
potential as a solution to data scarcity issues prevalent in biomedical
imaging. We substantiate the applicability of CCTs in data-constrained areas
and encourage further work on CCTs.