10.14639/0392-100X-N2580
&nbsp;
&nbsp;
text/xml
1827-675X
2580
en
Sara Moccia
http://creativecommons.org/licenses/by-nc-nd/4.0
ACTA Otorhinolaryngologica Italica
larynx
Deep learning in endoscopy: the importance of standardisation
Text.Serial.Journal
https://www.actaitalica.it/
43
Letters to the Editor-in-Chief
2023-10-10
2023-03-26
2023-12-19
2023-12-19
430-432
1
Sara Moccia
2023/10/10
10.14639/0392-100X-N2580
430
The BioRobotics Institute, Scuola Superiore Sant’Anna, Pisa, Italy; Department of Excellence in Robotics and AI, Scuola Superiore Sant’Anna, Pisa, Italy
1827-675X
larynx
en
432
Deep learning in endoscopy: the importance of standardisation
https://www.actaitalica.it/article/download/2580/1044
43
ACTA OTORHINOLARYNGOLOGICA ITALICA 2023;43:430-432. doi: 10.14639/0392-100X-N2580




Letter to the Editor


Deep learning in endoscopy:
the importance of standardisation
Deep learning in endoscopia: l’importanza della standardizzazione
Alberto Paderno1,2, Francesca Pia Villani3, Alessandra Sordi1,2, Claudia Montenegro1,2, Sara Moccia4,5
1
  Unit of Otorhinolaryngology - Head and Neck Surgery, ASST Spedali Civili of Brescia, Brescia, Italy; 2 Department of Medical
and Surgical Specialties, Radiological Sciences, and Public Health, University of Brescia, School of Medicine, Brescia, Italy;
3
  Department of Humanities, Università degli Studi di Macerata, Macerata, Italy; 4 The BioRobotics Institute, Scuola Superiore
Sant’Anna, Pisa, Italy; 5 Department of Excellence in Robotics and AI, Scuola Superiore Sant’Anna, Pisa, Italy


KEY WORDS: deep learning, endoscopy, outcomes, computer vision, larynx
PAROLE CHIAVE: deep learning, endoscopia, risultati, computer vision, laringe



Dear Editor,
In recent years, there has been an explosion of interest in the use of deep learn-
ing (DL) for medical image analysis. Different DL architectures have been
proposed to address a variety of tasks, including image classification, object
detection, segmentation and characterisation 1. These advancements have led
to significant progress in assisting clinicians in the evaluation of endoscopic
frames, giving birth to a field of artificial intelligence-associated applications
called “videomics” 2. However, a major challenge is that the performance of
these methods is often not directly comparable among different institutions/
series because of the lack of standardised evaluation methodology.                       Received: March 26, 2023
The importance of standardising outcomes in DL for medical imaging can-                  Accepted: April 3, 2023
                                                                                         Published online: October 10, 2023
not be overemphasised and its lack is a major barrier to translation into actual
clinical practice of such algorithms for videomics. To enable fair algorithm             Correspondence
comparisons, it is essential to have common evaluation metrics that are agreed           Alberto Paderno
upon by the research community from both clinical and technical perspec-                 Department of Otorhinolaryngology - Head
                                                                                         and Neck Surgery, Spedali Civili di Brescia,
tives 3.                                                                                 piazzale Spedali Civili 1, 25123 Brescia, Italy
The dataset used for algorithm training, validation and testing should be rep-           Tel. +39 030 3995319
resentative of the real-world clinical scenario, possibly including data from            E-mail: albpaderno@gmail.com
different centres and annotated by different clinicians.
Herein, we propose a general guideline to standardise reporting in studies fo-             How to cite this article: Paderno A, Villani
                                                                                           FP, Sordi A, et al. Deep learning in endoscopy:
cused on the automatic analysis of endoscopic images (i.e. videomics). Fig-                the importance of standardisation. Acta Otorhi-
ure 1 shows a schematic framework aimed at addressing study definition and                 nolaryngol Ital 2023;43:430-432. https://doi.
outcome metrics.                                                                           org/10.14639/0392-100X-N2580

                                                                                         © Società Italiana di Otorinolaringoiatria
Study definition                                                                         e Chirurgia Cervico-Facciale


The first step is to clearly describe the setting and methodology:                                      OPEN ACCESS
1. Objective: clearly state the type of task to be assessed.                             This is an open access article distributed in accordance with
                                                                                         the CC-BY-NC-ND (Creative Commons Attribution-Non-
2. Algorithm: describe the DL algorithm, its architecture, and technical fea-            Commercial-NoDerivatives 4.0 International) license. The
   tures (e.g. loss function, optimisation, learning rate, batch size, validation        article can be used by giving appropriate credit and mentio-
                                                                                         ning the license, but only for non-commercial purposes and
   metric, strategy to stop training, training curves).                                  only in the original version. For further information: https://
3. Describe the number of patients and number of frames extracted from each              creativecommons.org/licenses/by-nc-nd/4.0/deed.en




430
                                                                                                                                Deep learning in endoscopy




   (with selection criteria). Technical information regard-                     balance, appropriate metrics should be considered (e.g. F1-
   ing endoscopic and recording equipment, as well as the                       score instead of accuracy, precision-recall curve).
   use of optical filters (e.g. narrow band imaging), should                    Conversely, special considerations should be applied to de-
   also be available.                                                           tection and segmentation tasks. It is essential to define the
4. The training-validation-test set split ratio should be de-                   percentage of frames in which a region of interest (ROI) has
   scribed, specifying the type of cross-validation. Patients                   been detected. Here, the most commonly used parameter is
   should be clustered in different sets.                                       accuracy (Acc), which measures the percentage of correctly
                                                                                classified pixels. However, endoscopic frames are common-
Definition of primary outcomes                                                  ly highly class-imbalanced. Frames usually contain a single
                                                                                region of interest involving only a small portion of pixels,
A further step is to clearly define the study outcomes by                       whereas the remaining image is labelled as background.
selecting measures that are actually representative of diag-                    Because of the positive impact of true negatives (i.e. back-
nostic performance. Figure 1 shows the suggested metrics                        ground), Acc will always result in high scores. Similarly,
for each task, selecting those with a lower risk of result-                     specificity (Spec) indicates the model capability to detect
ing in an illegitimate high score. Adjunctive metrics can be                    the background in an image. Due to the large fraction of
added, but should not be considered as primary outcomes                         pixels annotated as background compared to the ROI, speci-
unless there is a clear rationale.                                              ficity values close to 1 are typical. For this reason, Acc and
Regarding classification, thanks to the clear dichotomic                        Spec should not be used as primary outcomes.
distinction between positive and negative results, the same                     Sensitivity (also defined as Recall) is another popular met-
rationale used for standard diagnostic tests should be ap-                      ric, but in detection and segmentation tasks it is less repre-
plied. A wide range of metrics (Fig. 1) should be provided                      sentative than overlap metrics such as the Dice Similarity
to allow a comprehensive assessment of the algorithm’s                          Coefficient (DSC) and Intersection over Union (IoU). In
diagnostic characteristics. In addition, in case of class un-                   particular, the DSC is calculated by taking the number of




 aHD, average Hausdorff distance; AUC, area under the curve;
 CNN, convolutional neural network;
 DSC, Dice Similarity Coefficient; IoU, Intersection over Union.

Figure 1. Schematic framework detailing a proposal to standardise definitions and outcome measures in studies assessing deep learning in videoendoscopy.



                                                                                                                                                      431
A. Paderno et al.




data points that are shared by both sets and dividing it by the   ing agencies in the public, commercial, or not-for-profit
total number of data points; IoU is calculated by taking the      sectors.
intersection of the predicted segmentation and the ground
truth segmentation, and dividing it by the union of the two.      Author contributions
Finally, the Hausdorff Distance (HD) measures the distance        AP, FPV, and SM conceived and designed the study; AS
between two sets of points (i.e. ground truth and predicted       and CM discussed the contents and commented on the
segmentation), and allows scoring localisation similarity         manuscript.
by focusing on the delineation of margins. Since the HD is
sensitive to outliers, the average HD may be better suited        Ethical consideration
for most applications.                                            Not applicable.
This is a first proposal to standardise reporting in videom-
ics studies. Some technical concepts should be taken into
consideration to improve collaboration and reliably assess        References
the performance of novel DL algorithms before introducing         1
                                                                      Paderno A, Gennarini F, Sordi A, et al. Artificial intelligence in
them into a clinical setting.                                         clinical endoscopy: insights in the field of videomics. Front Surg
                                                                      2022;9:933297. https://doi.org/10.3389/fsurg.2022.933297

Conflict of interest statement
                                                                  2
                                                                      Paderno A, Holsinger FC, Piazza C. Videomics: bringing deep learn-
                                                                      ing to diagnostic endoscopy. Curr Opin Otolaryngol Head Neck Surg
The authors declare no conflict of interest.                          2021;29:143-148. https://doi.org/10.1097/MOO.0000000000000697
                                                                  3
                                                                      Taha AA, Hanbury A. Metrics for evaluating 3D medical image
Funding                                                               segmentation: analysis, selection, and tool. BMC Med Imaging
This research did not receive any specific grant from fund-           2015;15:29. https://doi.org/10.1186/s12880-015-0068-x




432
