Statistical validation of a deep learning algorithm for dental anomaly
  detection in intraoral radiographs using paired data
Pieter Van Leemput Johannes Keustermans Wouter Mollemans
This article describes the clinical validation study setup, statistical
analysis and results for a deep learning algorithm which detects dental
anomalies in intraoral radiographic images, more specifically caries, apical
lesions, root canal treatment defects, marginal defects at crown restorations,
periodontal bone loss and calculus. The study compares the detection
performance of dentists using the deep learning algorithm to the prior
performance of these dentists evaluating the images without algorithmic
assistance. Calculating the marginal profit and loss of performance from the
annotated paired image data allows for a quantification of the hypothesized
change in sensitivity and specificity. The statistical significance of these
results is extensively proven using both McNemar's test and the binomial
hypothesis test. The average sensitivity increases from $60.7\%$ to $85.9\%$,
while the average specificity slightly decreases from $94.5\%$ to $92.7\%$. We
prove that the increase of the area under the localization ROC curve (AUC) is
significant (from $0.60$ to $0.86$ on average), while the average AUC is
bounded by the $95\%$ confidence intervals ${[}0.54, 0.65{]}$ and ${[}0.82,
0.90{]}$. When using the deep learning algorithm for diagnostic guidance, the
dentist can be $95\%$ confident that the average true population sensitivity is
bounded by the range $79.6\%$ to $91.9\%$. The proposed paired data setup and
statistical analysis can be used as a blueprint to thoroughly test the effect
of a modality change, like a deep learning based detection and/or segmentation,
on radiographic images.