10.14639/0392-100X-N2336
Objective. To achieve instance segmentation of upper aerodigestive tract (UADT) neoplasms using a deep learning (DL) algorithm, and to identify differences in its diagnostic performance in three different sites: larynx/hypopharynx, oral cavity and oropharynx. Methods. A total of 1034 endoscopic images from 323 patients were examined under narrow band imaging (NBI). The Mask R‑CNN algorithm was used for the analysis. The dataset split was: 935 training, 48 validation and 51 testing images. Dice Similarity Coefficient (Dsc) was the main outcome measure. Results. Instance segmentation was effective in 76.5% of images. The mean Dsc was 0.90 ± 0.05. The algorithm correctly predicted 77.8%, 86.7% and 55.5% of lesions in the larynx/hypopharynx, oral cavity, and oropharynx, respectively. The mean Dsc was 0.90 ± 0.05 for the larynx/hypopharynx, 0.60 ± 0.26 for the oral cavity, and 0.81 ± 0.30 for the oropharynx. The analysis showed inferior diagnostic results in the oral cavity compared with the larynx/hypopharynx (p &lt; 0.001). Conclusions. The study confirms the feasibility of instance segmentation of UADT using DL algorithms and shows inferior diagnostic results in the oral cavity compared with other anatomic areas.
text/xml
1827-675X
2336
en
Cesare Piazza
http://creativecommons.org/licenses/by-nc-nd/4.0
ACTA Otorhinolaryngologica Italica
videomics
Instance segmentation of upper aerodigestive tract cancer: site‑specific outcomes
Text.Serial.Journal
https://www.actaitalica.it/
43
Clinical techniques and technologies
2023-07-24
2022-10-10
2023-07-25
2023-07-25
283-290
1
Cesare Piazza
2023/07/24
10.14639/0392-100X-N2336
283
Unit of Otorhinolaryngology, Head and Neck Surgery, ASST Spedali Civili of Brescia, Brescia, Italy; Department of Medical and Surgical Specialties, Radiological Sciences, and Public Health, University of Brescia, School of Medicine, Brescia, Italy
1827-675X
videomics
en
290
Instance segmentation of upper aerodigestive tract cancer: site‑specific outcomes
https://www.actaitalica.it/article/download/2336/1018
43
ACTA OTORHINOLARYNGOLOGICA ITALICA 2023;43:283-290; doi: 10.14639/0392-100X-N2336




Clinical techniques and technologies


Instance segmentation of upper aerodigestive tract
cancer: site‑specific outcomes
Instance segmentation nei tumori delle vie areo‑digestive superiori
Alberto Paderno1,2, Francesca Pia Villani3, Milena Fior2, Giulia Berretti2, Francesca Gennarini2, Gabriele Zigliani2,
Emanuela Ulaj2, Claudia Montenegro2, Alessandra Sordi2, Claudio Sampieri4, Giorgio Peretti4, Sara Moccia5,6,
Cesare Piazza1,2
1
 Unit of Otorhinolaryngology, Head and Neck Surgery, ASST Spedali Civili of Brescia, Brescia, Italy; 2 Department of Medical and
Surgical Specialties, Radiological Sciences, and Public Health, University of Brescia, School of Medicine, Brescia, Italy; 3 Department
of Humanities, Università degli Studi di Macerata, Macerata, Italy; 4 Unit of Otorhinolaryngology, Head and Neck Surgery, IRCCS
Ospedale Policlinico San Martino, Genoa, Italy; 5 The BioRobotics Institute, Scuola Superiore Sant’Anna, Pisa, Italy; 6 Department of
Excellence in Robotics and AI, Scuola Superiore Sant’Anna, Pisa, Italy


SUMMARY
Objective. To achieve instance segmentation of upper aerodigestive tract (UADT) neo-
plasms using a deep learning (DL) algorithm, and to identify differences in its diagnostic
performance in three different sites: larynx/hypopharynx, oral cavity and oropharynx.
Methods. A total of 1034 endoscopic images from 323 patients were examined under nar-
row band imaging (NBI). The Mask R‑CNN algorithm was used for the analysis. The data-
set split was: 935 training, 48 validation and 51 testing images. Dice Similarity Coefficient
(Dsc) was the main outcome measure.
Results. Instance segmentation was effective in 76.5% of images. The mean Dsc was
0.90 ± 0.05. The algorithm correctly predicted 77.8%, 86.7% and 55.5% of lesions in
the larynx/hypopharynx, oral cavity, and oropharynx, respectively. The mean Dsc was
0.90 ± 0.05 for the larynx/hypopharynx, 0.60 ± 0.26 for the oral cavity, and 0.81 ± 0.30 for
the oropharynx. The analysis showed inferior diagnostic results in the oral cavity compared
with the larynx/hypopharynx (p < 0.001).                                                         Received: October 14, 2022
Conclusions. The study confirms the feasibility of instance segmentation of UADT using           Accepted: January 8, 2023
DL algorithms and shows inferior diagnostic results in the oral cavity compared with other
anatomic areas.                                                                                  Correspondence
KEY WORDS: artificial intelligence, instance segmentation, deep learning, videomics              Alberto Paderno
                                                                                                 Department of Otorhinolaryngology, Head and
                                                                                                 Neck Surgery, Spedali Civili di Brescia, piazzale
RIASSUNTO                                                                                        Spedali Civili 1, Brescia, Italy
Obiettivo. Valutare l’instance segmentation utilizzando un algoritmo di intelligenza artifi-     Tel. +39 030 3995319
ciale (IA) nei tumori delle vie aerodigestive superiori. Si è poi confrontata la performance     E-mail: albpaderno@gmail.com
diagnostica in tre diversi siti anatomici: laringe/ipofaringe, cavo orale e orofaringe.
Metodi. Sono state analizzate 1034 immagini NBI di 323 pazienti. Lo studio si è avvalso            How to cite this article: Paderno A, Villani
dell’algoritmo Mask‑R‑CNN. Il dataset è stato suddiviso in 935 immagini per il training,           FP, Fior M et al. Instance segmentation of up-
48 per la validazione e 51 per il testing. Il principale outcome misurato è stato il Dice          per aerodigestive tract cancer: site-specific out-
Similarity Coefficient (Dsc).                                                                      comes. Acta Otorhinolaryngol Ital Acta Otorhi-
Risultati. L’algoritmo ha identificato le lesioni nel 76.5% delle immagini. Il valore totale       nolaryngol Ital 2023;43:283-290. https://doi.
del Dsc è stato 0,90 ± 0,05. Considerando le diverse sottosedi, sono state segmentate il           org/10.14639/0392-100X-N2336
77,8% delle lesioni laringo‑ipofaringee, l’86,7% delle orali e il 55,5% delle orofaringee. Il
Dsc per le tre sottosedi è stato 0,90 ± 0,05, 0,60 ± 0,26 e 0,81 ± 0,30 rispettivamente per      © Società Italiana di Otorinolaringoiatria
                                                                                                 e Chirurgia Cervico-Facciale
laringe/ipofaringe, cavo orale e orofaringe. L’analisi ha dimostrato risultati migliori per la
laringe/ipofaringe e l’orofaringe rispetto a quelli del cavo orale (p < 0,001).
                                                                                                                OPEN ACCESS
Conclusioni. Questo studio dimostra la fattibilità dell’instance segmentation nelle vie ae-
rodigestive superiori utilizzando un algoritmo di IA, mostrando risultati inferiori nel cavo     This is an open access article distributed in accordance with
orale rispetto alle altre sottosedi.                                                             the CC-BY-NC-ND (Creative Commons Attribution-Non-
                                                                                                 Commercial-NoDerivatives 4.0 International) license. The
PAROLE CHIAVE: intelligenza artificiale, instance segmentation, deep learning, videomica         article can be used by giving appropriate credit and mentio-
                                                                                                 ning the license, but only for non-commercial purposes and
                                                                                                 only in the original version. For further information: https://
                                                                                                 creativecommons.org/licenses/by-nc-nd/4.0/deed.en




                                                                                                                                                        283
A. Paderno et al.




Introduction                                                     Materials and methods
The application of computer vision techniques in diagnos-        A retrospective study was performed including videoen-
tic videoendoscopies (i.e. Videomics) 1,2 is a promising re-     doscopies performed between September 2009 and January
search field that is currently showing a fast rate of growth     2021 in patients treated at the Unit of Otorhinolaryngol-
in many medical specialties. The recent refinement of deep       ogy – Head and Neck Surgery, University of Brescia, Italy
learning (DL) algorithms for image processing and their          for SCC of the UADT. A total of 7,567 videoendoscopies
application in the medical field opened novel possibilities      were collected from a dedicated archive. All recordings
in the management of endoscopic exams that, in the past,         were anonymised and associated with the corresponding
had only subjective value. In particular, videoendoscopy is      histopathologic report.
a key component in the management of upper aerodiges-            The study primary endpoint was the definition of the di-
tive tract (UADT) tumours, influencing their entire diag-        agnostic accuracy (in terms of Dice Similarity Coefficient
nostic process, treatment and follow‑up 3. Notwithstanding,      [Dsc]) of the Mask R‑CNN algorithm when applied to NBI
it remains a operator‑dependent and time‑consuming pro-          UADT videoendoscopic frames. The secondary endpoint
cedure, which is substantially limited by the variables of       was the comparison of the algorithm’s Dsc in the three dif-
human experience and perception. This is especially true         ferent anatomical areas herein considered.
when endoscopy is applied in conjunction with optical bi-        Inclusion criteria were as follows:
opsy techniques such as Narrow Band Imaging (NBI) 4, re-         • primary or recurrent SCC of the UADT (distinguished
quiring even more specialised training and adding a further         between those occurring in the oral cavity, oropharynx,
layer of complexity and subjectivity. Finally, no easily clas-      and larynx/hypopharynx);
sifiable and structured data can be drawn from these exami-      • NBI evaluation with adequate quality (without pooling
nations, significantly limiting their integration with other        of saliva, blood spots, swallowing reflex, coughing or
                                                                    other technical issues);
technologies (e.g., cross sectional imaging, ultrasound,
                                                                 • available histological examination obtained at the time
genomic markers, and so on). This is also highlighted by
                                                                    of videoendoscopy or subsequent surgery.
initial attempts to standardise endoscopic evaluation and
                                                                 All patients were examined both under white light (WL)
improve implementation of new analytic techniques 5.
                                                                 and NBI through transnasal videolaryngoscopy (HD Vid-
Our study aimed to explore the potential of a novel DL
                                                                 eo Rhino‑laryngoscope Olympus ENF‑VH, ENF‑VQ, or
algorithm, Mask R‑CNN 6, in the diagnostic approach to
                                                                 ENF‑V2, Olympus Medical System Corporation, Tokyo,
UADT squamous cell carcinoma (SCC). The primary goal
                                                                 Japan) or through transoral endoscopy by 0° rigid telescope
was to detect and classify neoplastic lesions and, at the
                                                                 coupled to an Evis Exera II HD camera connected to an
same time, precisely define their margins, a task overall        Evis Exera II CLV‑180B/III CV‑190 light source (Olympus
defined as “instance segmentation”. In fact, Mask R‑CNN          Medical Systems Corporation, Tokyo, Japan). Endoscopic
provides a flexible and general framework for object in-         videos were selected independently by two otolaryngolo-
stance segmentation that can also be potentially applied to      gists with extensive experience (at least 4 years) in endo-
medical images. This approach combines elements from             scopic assessment of UADT lesions by NBI and indepen-
the tasks of object detection (where the goal is to local-       dently reviewed by an adjunctive expert. Images were then
ise the lesion using a bounding box), object classification 7    manually quality‑controlled, with exclusion of those that
(where the purpose is to classify each pixel into a set of       were blurred, obscured by blood or secretions, or without
categories – e.g., tumour vs. normal mucosa), and semantic       adequate NBI evaluation.
segmentation (where the aim is to automatically delineate
the lesion’s margins). Finally, we included in our analysis      Image processing
three different areas of the UADT (oral cavity, oropharynx,      Three representative frames per video were selected for
larynx/hypopharynx) in order to identify potential site‑re-      every lesion and saved in jpeg format. The most representa-
lated differences in the diagnostic capability of this DL al-    tive NBI videoframe was chosen and subsequent frames at
gorithm, an information that is still lacking in the current     0.3 second time intervals were then automatically selected.
literature. In fact, studies assessing the value of artificial   Frame annotation was performed manually using the La-
intelligence in endoscopy are generally focused on a single      belMe application 8. Annotations consisted of a variable
site and are difficult to generalise in the context of UADT      number of key points marking the lesion margins in the
SCC, which can arise from a wide variety of anatomical           videoendoscopic frame taking into account positive NBI
structures, as well as epithelial and mucosal types.             patterns. The resulting masks were then saved in json for-



284
                                                                        Instance segmentation of upper aerodigestive tract cancer: site-specific outcomes




mat and stored in a dedicated folder. Two clinical experts                 the input frame at multiple scales. Starting from the features
concomitantly annotated the images and a further review                    computed with the backbone, the RPN identifies candidate
was performed by a senior staff member. When an agree-                     regions containing the tumour. For each of the proposed
ment regarding lesion margins was not reached, the frame                   regions, the final bounding box containing the tumour and
was excluded from the analysis.                                            the tumour segmentation are obtained from the three heads.
After this selection process, a total of 1034 endoscopic                   To cope with the relatively limited size of the dataset, we
images were obtained. Three different sub‑datasets were                    used the weights computed on the COCO dataset 12 to ini-
generated according to the lesion primary site: oral cavity,               tialise the layers of Mask R‑CNN. To reduce the risk of
oropharynx, and larynx/hypopharynx. In this way, the total                 overfitting, we performed on‑the‑fly data augmentation
frames analysed were 653 for the larynx/hypopharynx, 246                   during training by applying: random brightness changes in
for the oral cavity, and 135 for the oropharynx.                           the range (0.5, 1.1), random contrast changes in the range
                                                                           (0.8, 3) and random rotation in the range (-20, 20).
Dataset                                                                    The model was trained for 100 epochs, using the Stochas-
The dataset included 1034 images from 323 patients. For                    tic Gradient Descent (SGD) as optimiser with an initial
algorithm training and testing the dataset was split over pa-              learning rate of 0.001 and momentum of 0.9. We used a
tients and balancing the three classes into three sets: 935                loss which is the combination of different contributions:
images from 290 subjects for training, 48 images from 16
subjects for validation, and 51 images from 17 subjects for
testing. All images were resized to the same dimension of                  where       is the loss in the classification head,             is
480 x 640 pixels.                                                          the loss in bounding‑box regression head,                  is the
                                                                           classification loss in the RPN,               is the localisation
DL analysis
In this work, Mask R‑CNN 9 was used to segment the tu-                     loss in the RPN, and           is the loss in segmentation
mour in endoscopic frames. This convolutional neural                       head. The loss equations can be found in the original Mask
network (CNN) consists of backbone, Region Proposal                        R‑CNN paper 9.
Network (RPN), and three heads for classification, bound-
ing‑box regression and segmentation (Fig. 1).                              Performance metrics and statistical analysis
As backbone, we used the ResNet50 10 combined with the                     As a primary endpoint the segmentation performance was
Feature Pyramid Network (FPN) 11 to extract features from                  evaluated using the Dsc, which is a statistical validation




Figure 1. Schematic representation of the proposed architecture. The Mask R‑CNN is made of a backbone (composed by a ResNet50 and a feature pyramid
network), a region proposal network (RPN), ROIAlign, and three heads, for classification, bounding‑box regression, and segmentation.



                                                                                                                                                   285
A. Paderno et al.




metric based on the spatial overlap between the predicted        The Kruskal‑Wallis H‑test was used for the overall compar-
(       ) and ground-truth (    ) segmentation:                  ison and the Mann‑Whitney U rank test for pair compari-
                                                                 sons. Statistical analysis was performed using Jupiter Note-
                                                                 book 6.4.5 with pandas 1.4.1 and ScyPy 1.8.0 libraries.

                                                                 Results
Dsc can assume values in a range from 0, indicating no           Overall performance
overlap, to 1, indicating complete overlap.                      The total number of images used for the test set was 51.
Furthermore, outcomes were also evaluated using the fol-         The algorithm demonstrated the ability to correctly predict
lowing spatial overlap‑based metrics:                            39 of 51 images (76.5%). The average Dsc score was 0.79
Pixel accuracy (Acc) represents the percent of pixels in the     (range, 0.26‑0.97; standard deviation (SD), 0.22). Overall
image which are correctly classified.                            and site‑specific performance metrics are summarised in
                                                                 Table I and Figure 2. Samples of the segmentation results
It is defined as:                                                are presented in Figure 3.

where TP, TN, FP, FN denote the true positives, true nega-       Laryngeal/hypopharyngeal lesions
tives, false positives and false negatives, respectively.        The total number of laryngeal and hypopharyngeal lesions
Recall (Rec), also known as Sensitivity or True Posi-            in the test set were 27 (52.9% of the test dataset). Out of
tive Rate, defines the portion of positive pixels in the         that number, our algorithm correctly predicted 21 lesions
ground‑truth which are also identified as positive in the        (77.8%). The mean Dsc score was 0.90 ± 0.05, the first
predicted segmentation.
                                                                 quartile was 0.90 and the third quartile 0.94 (Tab. I).
It is defined as:
                                                                 Oral lesions
Specificity (Spec), or True Negative Rate, measures the por-     The oral lesions comprised in the test set were 15 (29.4% of
tion of negative pixels (background) in the ground‑truth that    the total). The algorithm performed a correct prediction in
are also identified as negative in the predicted segmentation.   13 cases (86.7%). The mean Dsc score was 0.60 ± 0.26, the
It is defined as:                                                first quartile was 0.34 and the third quartile 0.84 (Tab. I).

Precision (Prec), or Positive Predictive Value, measures         Oropharyngeal lesions
how accurate the predictions are, i.e. the percentage of cor-    In the test set, the oropharyngeal lesions were 9 of 51 imag-
rect predictions.                                                es (17.6%). The algorithm predicted 5 images (55.5%). The
It is defined as:                                                mean value of Dsc score was 0.81 ± 0.30, the first quartile
                                                                 was 0.92 and the third quartile 0.95 (Tab. I).
F1‑score is a balance between precision and recall, also
known as harmonic mean.                                          Comparison between three different UADT sites
                                                                 Results for each site are summarised in Table I. The overall
It is defined as:                                                diagnostic performance, defined by the Dsc score, was sig-
                                                                 nificantly different between the different sites (p = 0.002).
Intersection over Union (IoU), also referred to as Jaccard in-   Pairwise analysis showed that the difference was related to
dex, represents the area of overlap between the predicted seg-   significantly inferior results in the oral cavity when com-
mentation and the ground truth divided by the area of union      pared with larynx/hypopharynx (p < 0.001).
between the predicted segmentation and the ground truth.         Diagnostic results proved to be significantly correlated
It is defined as:                                                with the site analysed also considering other performance
                                                                 metrics: accuracy (p < 0.001), specificity (p = 0.02), IoU
Mean Average Precision (mAP), which represents the aver-         (p = 0.002), and F1 score (p = 0.002). As above, this differ-
age of the area under the Recall‑Precision curve, was also       ence is related to inferior results in the oral cavity vs larynx/
computed.                                                        hypopharynx. However, when considering accuracy, it is
Outcomes were compared between the different subsites            also possible to evidence a significant difference between
analysed using non‑parametric statistics.                        oral cavity and oropharynx (p = 0.03).



286
                                                                                          Instance segmentation of upper aerodigestive tract cancer: site-specific outcomes




Figure 2. Box plots detailing the diagnostic accuracy of the algorithm in different sites according to various metrics. (A) Dice similarity coefficient (Dsc); (B) Ac‑
curacy; (C) Specificity; (D) Precision; (E) Recall; (F) Intersection over Union (IoU); (G) F1 score.


Table I. Summary of the diagnostic performance according to different metrics.
 Mean value (SD)                          Overall                      Larynx/hypopharynx                                  Oral cavity                Oropharynx
 Dsc                                              0.79 ± 0.23                             0.90 ± 0.05                     0.60± 0.26                  0.80 ± 0.30
 Accuracy                                         0.91 ± 0.12                             0.98 ± 0.01                     0.79 ± 0.13                 0.92 ± 0.14
 Specificity                                      0.93 ± 0.12                             0.98 ± 0.01                     0.86 ± 0.16                 0.92 ± 0.15
 Precision                                        0.85 ± 0.24                             0.94 ± 0.06                     0.73 ± 0.32                 0.79 ± 0.36
 Recall                                           0.86 ± 0.22                             0.91 ± 0.08                     0.73 ± 0.33                 0.95 ± 0.04
 IoU                                              0.73 ± 0.27                             0.87 ± 0.09                     0.49 ± 0.30                 0.76 ± 0.14
 F1 score                                         0.80 ± 0.23                             0.92 ± 0.05                     0.61 ± 0.27                 0.81 ± 0.31
Dsc: Dice similarity coefficient; IoU: Intersection over Union; SD: standard deviation.



Discussion                                                                                   was able to identify and segment the lesion in 76.5% of
                                                                                             cases, and showed remarkable diagnostic accuracy, espe-
In this study, we evaluated for the first time the specific
task of instance segmentation in clinical endoscopy for                                      cially in consideration of the complex task to be performed.
head and neck SCC. The analysis included three sites of the                                  Interestingly, results were significantly inferior in the oral
UADT to allow comparison of the algorithm’s diagnostic                                       cavity, where all outcome measures underperformed when
performance in different anatomical areas. The algorithm                                     compared with larynx/hypopharynx and, in some cases



                                                                                                                                                                     287
A. Paderno et al.




Figure 3. Visual samples of the segmentation results. From left to right: raw endoscopic frames, ground truth annotation, and predictions obtained with the
proposed method.



288
                                                                 Instance segmentation of upper aerodigestive tract cancer: site-specific outcomes




(i.e., accuracy), oropharynx. This is in line with what pre-        formed previously published results on the external valida-
viously observed by Piazza and coworkers 13 when applying           tion cohorts. The model was initially trained on WL and
bioendoscopic tools such as NBI. This result is possibly            NBI endoscopic frames of laryngeal SCC, but also showed
related to the wide array of epithelial subtypes observed           to be effective in the segmentation of independent frames
in the oral cavity, adjunctive limits specifically correlated       of oral and oropharyngeal cancer. The authors stated that
with oral examination (e.g., presence of light artifacts), and      the model demonstrated potential for improved detection of
confounding factors (e.g., tongue blade, teeth, or dentures)        early tumours, more precise biopsies and better selection of
that the ML software must learn to take into account.               resection margins.
Instance segmentation represents the ultimate step in video         In general, results of automatic segmentation are infe-
analysis since it allows at the same time detection, clas-          rior to those obtained in more straightforward tasks such
sification, and segmentation of multiple elements in each           as frame classification 18‑20 or lesion detection 21,22 since a
single frame, which is possible thanks to the integration           more in‑depth conceptual model of UADT lesions is re-
of different analytic components in the same general algo-          quired to allow accurate definition of margins. However,
rithm. This approach is particularly suited to the context          semantic segmentation is a key objective when striving to-
of UADT endoscopy since different alterations (e.g., con-           wards more complex tasks involving computer vision and
comitant inflammatory or benign lesions) can be frequently          human‑machine interaction. In fact, other than providing
encountered in the field of view together with the target           a purely diagnostic tool, a comprehensive understanding
lesion, and due to the fact that patients with head and neck        of all UADT alterations and suspicious lesions may grant
SCC can develop distinct islands of neoplastic or dysplastic        significant aid in intra-operative management. This is even
mucosa (i.e., field of cancerisation) that might involve vari-      more true when considering instance segmentation, which
ous portions of the videoframe, even without continuity.
                                                                    epitomises in itself all the needs and requirements of the
In general, recent CNN‑based methods have demonstrat-
                                                                    visual examination of endoscopic images, allowing a full
ed remarkable results in segmentation of the UADT and
                                                                    automatic understanding of complex endoscopic scenarios,
proved to be well‑suited for such a complex task. Laves
                                                                    even those involving more than one lesion and/or more than
et al. 14 first demonstrated that a weighted average ensem-
                                                                    one pathology.
ble network of UNet and ErfNet were the best suited for
                                                                    Potential issues have been addressed to limit biases related
laryngeal segmentation of intra-operative images under di-
                                                                    to the analysis technique:
rect laryngoscopy, with a mean IoU of 84.7%. However,
                                                                    • patients (and their related frames) in the training, valida-
different authors subsequently strived toward development
of diagnostic algorithms that could be applied in real time            tion, and test sets have been distinguished into separated
in office‑based and intra-operative endoscopy. Paderno                 groups to avoid overfitting;
et al. 15 explored the use of fully CNNs for real‑time seg-         • frames were annotated and reviewed by 3 experts to lim-
mentation of SCC in the oral cavity and oropharynx. In this            it subjective errors;
work, different architectures were compared detailing their         • frame selection and data augmentation were performed
diagnostic performance and inference time, demonstrating               to reduce the impact of artifacts or technical biases.
the possibility to achieve real‑time segmentation. In ac-           However, intrinsic limits should be acknowledged. In par-
cordance with previous findings in literature, the present          ticular, the gold standard over which the algorithm has
study confirms that the oral cavity may have inferior diag-         been trained (i.e., the “ground truth”) is represented by an
nostic results due to the high variability of subsites when         expert opinion of the tumor margins and not by the histo-
compared with other areas of the UADT (i.e., oropharynx,            pathological definition per se. In fact, as of today, it is not
larynx, and hypopharynx). When dealing with normal la-              technically possible to provide a direct in situ, in vivo mor-
ryngeal anatomy, Fehling et al. 16 explored the possibility to      phologic correlation between endoscopic images and their
achieve a fully automated segmentation of the glottic area          histopathological specimen.
using a CNN in high‑speed laryngeal videos. The algorithm
obtained a Dsc over 0.85 for all subsites analysed. Finally,        Conflict of interest statement
Li et al. 17 proposed a method to segment nasopharyngeal            The authors declare no conflict of interest.
malignancies in endoscopic images based on DL, reach-
ing an accuracy of 88.0%. However, progressive advances             Funding
in automatic segmentation of the UADT can be observed               This research did not receive any specific grant from fund-
thanks to a recent article by Azam et al. 2, in which Seg-          ing agencies in the public, commercial, or not-for-profit
MENT, a novel CNN‑based segmentation model, outper-                 sectors.



                                                                                                                                            289
A. Paderno et al.




Author contributions                                                        9
                                                                                 He K, Gkioxari G, Dollár P, et al. Mask R-CNN, 2017 IEEE Interna-
                                                                                 tional Conference on Computer Vision (ICCV), Venice, Italy; 2017.
AP, FPV, FG, MF, GB, GZ, EU, CM, AS, SM, CP: con-
                                                                                 pp. 2980-2988. https://doi.org/10.1109/ICCV.2017.322
tributed to data collection and analysis; AP, CP: performed                 10
                                                                                 He K, Zhang X, Ren S, et al. Deep residual learning for image recog-
manuscript preparation; AP, FPV, FG, SM, CP: performed                           nition, 2016 IEEE Conference on Computer Vision and Pattern Rec-
final edits and revisions; AP, FPV, MF, GB, FG, GZ, EU,                          ognition (CVPR), Las Vegas, NV, USA; 2016. pp. 770-778. https://
CM, AS, CS, GP, SM, CP: reviewed contributed conceptu-                           doi.org/10.1109/CVPR.2016.90
ally to the article and approved the submitted version.
                                                                            11
                                                                                 Lin T, Dollár P, Girshick RB, et al. Feature pyramid networks for
                                                                                 object detection. IEEE Conference on Computer Vision and Pattern
                                                                                 Recognition (CVPR), Honolulu, HI, USA; 2017. pp. 936-944, https://
Ethical consideration                                                            doi.org/10.1109/CVPR.2017.106
This study was approved by the Institutional Ethics Com-                    12
                                                                                 Lin T, Maire M, Belongie S, et al. Microsoft COCO: common objects
mittee (please specify name of the Institution University of                     in context. European Conference on Computer Vision. Cham: Sprin-
                                                                                 ger; 2014. pp. 740-755.
Brescia) (protocol number 4267).                                            13
                                                                                 Piazza C, Del Bon F, Peretti G, et al. “Biologic endosco-
The research was conducted ethically, with all study pro-                        py”: optimization of upper aerodigestive tract cancer evalua-
cedures being performed in accordance with the require-                          tion. Curr Opin Otolaryngol Head Neck Surg 2011;19:67‑76.
ments of the World Medical Association’s Declaration of                          https://doi.org/10.1097/MOO.0b013e328344b3ed
Helsinki.                                                                   14
                                                                                 Laves MH, Bicker J, Kahrs LA, et al. A dataset of laryngeal endo-
                                                                                 scopic images with comparative study on convolution neural net-
Written informed consent was obtained from each partici-                         work‑based semantic segmentation. Int J Comput Assist Radiol Surg
pant/patient for study participation and data publication.                       2019;14:483‑492. https://doi.org/10.1007/s11548-018-01910-0
                                                                            15
                                                                                 Paderno A, Piazza C, Del Bon F, et al. Deep learning for automatic
                                                                                 segmentation of oral and oropharyngeal cancer using narrow band im-
References                                                                       aging: preliminary experience in a clinical perspective. Front Oncol
1
    Paderno A, Holsinger FC, Piazza C. Videomics: bringing deep learn-           2021;11:626602. https://doi.org/10.3389/fonc.2021.626602
    ing to diagnostic endoscopy. Curr Opin Otolaryngol Head Neck Surg       16
                                                                                 Fehling MK, Grosch F, Schuster ME et al. Fully automatic segmen-
    2021;29:143‑148. https://doi.org/10.1097/MOO.0000000000000697                tation of glottis and vocal folds in endoscopic laryngeal high‑speed
2
    Azam MA, Sampieri C, Ioppi A, et al. Videomics of the upper aero‑di-         videos using a deep Convolutional LSTM Network. PloS One
    gestive tract cancer: deep learning applied to white light and narrow        2020;15:E0227791. https://doi.org/10.1371/journal.pone.0227791
    band imaging for automatic segmentation of endoscopic images. Front     17
                                                                                 Li C, Jing B, Ke L, et al. Development and validation of an en-
    Oncol 2022;12:900451. https://doi.org/10.3389/fonc.2022.900451               doscopic images‑based deep learning model for detection with
3
    Paderno A, Gennarini F, Sordi A, et al. Artificial intelligence in           nasopharyngeal malignancies. Cancer Commun 2018;38:59.
    clinical endoscopy: insights in the field of videomics. Front Surg           https://doi.org/10.1186/s40880‑018‑0325‑9
    2022;9:933297. https://doi.org/10.3389/fsurg.2022.933297                18
                                                                                 Song B, Sunny S, Uthoff RD, et al. Automatic classification of
4
    Piazza C, Del Bon F, Paderno A, et al. The diagnostic value of nar-          dual‑modalilty, smartphone‑based oral dysplasia and malignancy
    row band imaging in different oral and oropharyngeal subsites. Eur           images using deep learning. Biomed Opt Express 2018;9:5318.
    Arch Otorhinolaryngol 2016;273:3347‑3353. https://doi.org/10.1007/           https://doi.org/10.1364/BOE.9.005318
    s00405‑016‑3925‑5                                                       19
                                                                                 Esmaeili N, Sharaf E, Gomes Ataide EJ, et al. Deep convolu-
5
    Nogal P, Buchwald M, Staśkiewicz M, et al. Endoluminal lar-                  tion neural network for laryngeal cancer classification on con-
    ynx anatomy model – towards facilitating deep learning and                   tact endoscopy‑narrow band imaging. Sensors 2021;21:8157.
    defining standards for medical images evaluation with arti-                  https://doi.org/10.3390/s21238157
    ficial intelligence algorithms. Otolaryngol Pol 2022;76:1‑9.            20
                                                                                 Dunham ME, Kong KA, McWhorter AJ, et al. Optical biopsy: au-
    https://doi.org/10.5604/01.3001.0015.9501                                    tomated classification of airway endoscopic findings using a convo-
6
    He K, Gkioxari G, Dollár P, et al. Mask R‑CNN. IEEE Trans Pat-               lutional neural network. Laryngoscope 2022;132(Suppl. 4):S1‑S8.
    tern Anal Mach Intell 2020;42:386‑397. https://doi.org/10.1109/              https://doi.org/10.1002/lary.28708
    TPAMI.2018.2844175                                                      21
                                                                                 Inaba A, Hori K, Yoda Y, et al. Artificial intelligence system for detect-
7
    Cho WK, Lee YJ, Joo HA, et al. Diagnostic accuracies of laryn-               ing superficial laryngopharyngeal cancer with high efficiency of deep
    geal diseases using a convolutional neural network‑based im-                 learning. Head Neck 2020;42:2581‑2592. https://doi.org/10.1002/
    age classification system. Laryngoscope 2021;131:2558‑2566.                  hed.26313
    https://doi.org/10.1002/lary.29595                                      22
                                                                                 Azam MA, Sampieri C, Ioppi A, et al. Deep learning applied to white
8
    Russell BC, Torralba A, Murphy KP et al. LabelMe: a database and             light and narrow band imaging videolaryngoscopy: toward real‑time
    web‑based tool for image annotation. Int J Comput Vis 2008;77:57-            laryngeal cancer detection. Laryngoscope 2022;132:1798‑1806. htt-
    173. https://doi.org/10.1007/s11263-007-0090-8                               ps://doi.org/10.1002/lary.29960




290
