How You Split Matters: Data Leakage and Subject Characteristics Studies
  in Longitudinal Brain MRI Analysis
Dewinda Julianensi Rumala
Deep learning models have revolutionized the field of medical image analysis,
offering significant promise for improved diagnostics and patient care.
However, their performance can be misleadingly optimistic due to a hidden
pitfall called 'data leakage'. In this study, we investigate data leakage in 3D
medical imaging, specifically using 3D Convolutional Neural Networks (CNNs) for
brain MRI analysis. While 3D CNNs appear less prone to leakage than 2D
counterparts, improper data splitting during cross-validation (CV) can still
pose issues, especially with longitudinal imaging data containing repeated
scans from the same subject. We explore the impact of different data splitting
strategies on model performance for longitudinal brain MRI analysis and
identify potential data leakage concerns. GradCAM visualization helps reveal
shortcuts in CNN models caused by identity confounding, where the model learns
to identify subjects along with diagnostic features. Our findings, consistent
with prior research, underscore the importance of subject-wise splitting and
evaluating our model further on hold-out data from different subjects to ensure
the integrity and reliability of deep learning models in medical image
analysis.