A novel method to enhance pneumonia detection via a model-level
  ensembling of CNN and vision transformer
Sandeep Angara Nishith Reddy Mannuru Aashrith Mannuru Sharath Thirunagaru
Pneumonia remains a leading cause of morbidity and mortality worldwide. Chest
X-ray (CXR) imaging is a fundamental diagnostic tool, but traditional analysis
relies on time-intensive expert evaluation. Recently, deep learning has shown
immense potential for automating pneumonia detection from CXRs. This paper
explores applying neural networks to improve CXR-based pneumonia diagnosis. We
developed a novel model fusing Convolution Neural networks (CNN) and Vision
Transformer networks via model-level ensembling. Our fusion architecture
combines a ResNet34 variant and a Multi-Axis Vision Transformer small model.
Both base models are initialized with ImageNet pre-trained weights. The output
layers are removed, and features are combined using a flattening layer before
final classification. Experiments used the Kaggle pediatric pneumonia dataset
containing 1,341 normal and 3,875 pneumonia CXR images. We compared our model
against standalone ResNet34, Vision Transformer, and Swin Transformer Tiny
baseline models using identical training procedures. Extensive data
augmentation, Adam optimization, learning rate warmup, and decay were employed.
The fusion model achieved a state-of-the-art accuracy of 94.87%, surpassing the
baselines. We also attained excellent sensitivity, specificity, kappa score,
and positive predictive value. Confusion matrix analysis confirms fewer
misclassifications. The ResNet34 and Vision Transformer combination enables
jointly learning robust features from CNNs and Transformer paradigms. This
model-level ensemble technique effectively integrates their complementary
strengths for enhanced pneumonia classification.