Towards trustworthy seizure onset detection using workflow notes
Khaled Saab Siyi Tang Mohamed Taha Christopher Lee-Messer Christopher Ré Daniel Rubin
A major barrier to deploying healthcare AI models is their trustworthiness.
One form of trustworthiness is a model's robustness across different subgroups:
while existing models may exhibit expert-level performance on aggregate
metrics, they often rely on non-causal features, leading to errors in hidden
subgroups. To take a step closer towards trustworthy seizure onset detection
from EEG, we propose to leverage annotations that are produced by healthcare
personnel in routine clinical workflows -- which we refer to as workflow notes
-- that include multiple event descriptions beyond seizures. Using workflow
notes, we first show that by scaling training data to an unprecedented level of
68,920 EEG hours, seizure onset detection performance significantly improves
(+12.3 AUROC points) compared to relying on smaller training sets with
expensive manual gold-standard labels. Second, we reveal that our binary
seizure onset detection model underperforms on clinically relevant subgroups
(e.g., up to a margin of 6.5 AUROC points between pediatrics and adults), while
having significantly higher false positives on EEG clips showing
non-epileptiform abnormalities compared to any EEG clip (+19 FPR points). To
improve model robustness to hidden subgroups, we train a multilabel model that
classifies 26 attributes other than seizures, such as spikes, slowing, and
movement artifacts. We find that our multilabel model significantly improves
overall seizure onset detection performance (+5.9 AUROC points) while greatly
improving performance among subgroups (up to +8.3 AUROC points), and decreases
false positives on non-epileptiform abnormalities by 8 FPR points. Finally, we
propose a clinical utility metric based on false positives per 24 EEG hours and
find that our multilabel model improves this clinical utility metric by a
factor of 2x across different clinical settings.