10.5826/dpc.1203a126
Introduction: Automated classification of dermatoscopic images via neural networks shows comparable performance to clinicians in experimental conditions, but can be affected by artefacts like skin markings or rulers. It is unknown whether specialized neural networks are equally affected, or more robust to artefacts.
Objectives: Analyse robustness of three neural network architectures, namely ResNet34, Faster R-CNN and Mask R-CNN. Methods: We identified common artefacts in the public HAM10000, PH2 and the 7-point criteria evaluation datasets, and established a template-based method to superimpose artefacts on dermatoscopic images. The HAM10000-dataset with and without superimposed artefacts was used to train the networks, followed by analysing their robustness against artefacts in test images.
Results: ResNet-34 and Faster R-CNN models trained on regular images perform worse than the Mask R-CNN models when tested on images with superimposed artefacts. Artefacts in all tested images led to a decrease in area under the precision-recall curve values of 0.030 for ResNet-34 and 0.045 for Faster R-CNN in comparison to only 0.011 for Mask R-CNN. However, changes in model’s performance only became significant with 40% or more of the images having superimposed artefacts. We could also show that loss in performance occurs when the training was biased by selectively superimposing artefacts on images belonging to a certain class.
Conclusions: Instance segmentation architectures may be helpful to counter the effects of artefacts, and further research on related architectures of this family should be promoted. Our introduced template-based artefact insertion mechanism could be useful for future research.
text/xml
2160-9381
1863
en
Philipp Tschandl
https://creativecommons.org/licenses/by-nc/4.0
Dermatology Practical & Conceptual
dermatoscopy
Comparison of convolutional neural network architectures for robustness against common artefacts in dermatoscopic images
Text.Serial.Journal
https://dpcj.org/index.php/dpc
Original Article
2022-07-28
2021-07-03
2022-07-28
2022-09-13
e2022126
Dermatol Pract Concept
Introduction: Automated classification of dermatoscopic images via neural networks shows comparable performance to clinicians in experimental conditions, but can be affected by artefacts like skin markings or rulers. It is unknown whether specialized neural networks are equally affected, or more robust to artefacts.
Objectives: Analyse robustness of three neural network architectures, namely ResNet34, Faster R-CNN and Mask R-CNN. Methods: We identified common artefacts in the public HAM10000, PH2 and the 7-point criteria evaluation datasets, and established a template-based method to superimpose artefacts on dermatoscopic images. The HAM10000-dataset with and without superimposed artefacts was used to train the networks, followed by analysing their robustness against artefacts in test images.
Results: ResNet-34 and Faster R-CNN models trained on regular images perform worse than the Mask R-CNN models when tested on images with superimposed artefacts. Artefacts in all tested images led to a decrease in area under the precision-recall curve values of 0.030 for ResNet-34 and 0.045 for Faster R-CNN in comparison to only 0.011 for Mask R-CNN. However, changes in model’s performance only became significant with 40% or more of the images having superimposed artefacts. We could also show that loss in performance occurs when the training was biased by selectively superimposing artefacts on images belonging to a certain class.
Conclusions: Instance segmentation architectures may be helpful to counter the effects of artefacts, and further research on related architectures of this family should be promoted. Our introduced template-based artefact insertion mechanism could be useful for future research.
Philipp Tschandl
2022/07/28
10.5826/dpc.1203a126
e2022126
Department of Dermatology, Medical University of Vienna, Vienna, Austria
2160-9381
dermatoscopy
en
e2022126
Comparison of convolutional neural network architectures for robustness against common artefacts in dermatoscopic images
https://dpcj.org/index.php/dpc/article/download/1863/1653
                    Dermatology Practical & Conceptual


       Comparison of Convolutional Neural Network
       Architectures for Robustness Against Common
             Artefacts in Dermatoscopic Images
                              Florian Katsch1, Christoph Rinner1, Philipp Tschandl2


1 Center for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna, Vienna, Austria
2 Department of Dermatology, Medical University of Vienna, Vienna, Austria



Key words: image classification, object detection, instance segmentation, artefacts, dermatoscopy,
Citation: Katsch F, Rinner C, Tschandl P. Comparison of convolutional neural network architectures for robustness against common
artefacts in dermatoscopic images. Dermatol Pract Concept. 2022;12(3):e2022126. DOI: https://doi.org/10.5826/dpc.1203a126
Accepted: December 7, 2021; Published: July 2022
Copyright: ©2022 Katsch et al. This is an open-access article distributed under the terms of the Creative Commons Attribution-
NonCommercial License (BY-NC-4.0), https://creativecommons.org/licenses/by-nc/4.0/, which permits unrestricted noncommercial use,
distribution, and reproduction in any medium, provided the original authors and source are credited.
Funding: None.
Competing interests: PT reports consulting fees from Silverchair, honoraria from FotoFinder, Novartis and Lilly, and grants from
MetaOptima Technology and Lilly, outside the submitted work. All other authors declare no conflict of interest.
Authorship: All authors have contributed significantly to this publication.
Corresponding author: Philipp Tschandl, PhD, Department of Dermatology, Medical University of Vienna, Währinger Gürtel 18-20, 1090
Vienna, Austria, E-mail: philipp.tschandl@meduniwien.ac.at




  ABSTRACT              Introduction: Classification of dermatoscopic images via neural networks shows comparable perfor-
                        mance to clinicians in experimental conditions but can be affected by artefacts like skin markings or
                        rulers. It is unknown whether specialized neural networks are more robust to artefacts.
                        Objectives: Analyze robustness of 3 neural network architectures, namely ResNet-34, Faster R-CNN
                        and Mask R-CNN.
                        Methods: We identified common artefacts in the HAM10000, PH2 and the 7-point criteria evaluation
                        datasets, and established a template-based method to superimpose artefacts on dermatoscopic images.
                        The HAM10000-dataset with and without superimposed artefacts was used to train the networks,
                        followed by analyzing their robustness against artefacts in test images. Performance was assessed via
                        area under the precision recall curve and classification results.
                        Results: ResNet-34 and Faster R-CNN models trained on regular images perform worse than Mask
                        R-CNN on images with superimposed artefacts. Artefacts added to all tested images led to a decrease
                        in area under the precision-recall curve values of 0.030 for ResNet-34 and 0.045 for Faster R-CNN in
                        comparison to only 0.011 for Mask R-CNN. However, changes in model performance only became sig-
                        nificant with 40% or more of the images having superimposed artefacts. A loss in performance occurred
                        when the training was biased by selectively superimposing artefacts on images belonging to a certain class.
                        Conclusions: As Mask R-CNN showed the least decrease in performance when confronted with artefacts,
                        instance segmentation architectures may be helpful to counter the effects of artefacts, warranting further
                        research on related architectures. Our artefact insertion mechanism could be useful for future research.




Original Article | Dermatol Pract Concept. 2022;12(3):e2022126                                                                        1
Introduction                                                                [15,16]. Object detection has been used in the field of au-
                                                                            tomated skin cancer detection on clinical images [17], but
Epidemiological studies show an increasing trend in the in-                 training of instance segmentation neural networks in der-
cidence rates of melanoma and non-melanoma skin cancer                      matoscopy has not yet been reported on successfully, most
worldwide over the last 30 years [1]. According to the Amer-                probably because of missing ground-truth data.
ican Joint Committee on Cancer melanoma staging system,
stage I malignant skin alterations with a five-year survival
rate of more than 90% contrasts with a survival rate of less                Objectives
than 15% for stage IV patients. This indicates a clear need
                                                                            Our hypothesis is that in contrast to ResNet, the other net-
for early, reliable and consistent diagnosis and treatment [2].
                                                                            work architectures intrinsically have to “concentrate” on re-
The desire for automatic lesion analysis is further intensified
                                                                            gions of the classified object in an image and hence may offer
by a high dependency between the diagnostic quality and the
                                                                            robustness against artefacts surrounding the lesions. Robust-
examiners experience in dermoscopy, as well as a high degree
                                                                            ness in this case describes the consistency of the obtained
of inter- and intra-variability of diagnoses [3,4].
                                                                            diagnoses under the influence of artefacts in the input image
    Methods of automatic skin lesion analysis have been the
                                                                            data. These networks could potentially be used as off-the-
focus of research for decades, and have gained interest in recent
                                                                            shelf methods with little customization-effort needed and
years [5,6]. These methods are intended to support tele-der-
                                                                            could enable us to focus less on tedious image pre-processing
matologic settings, improve management decisions or aid in
                                                                            such as removal of bubbles or hairs [18].
difficult clinical scenarios, but often suffer, among other things,
from the presence of artefacts in dermatoscopic images [7-12].
    A common neural network used for classification is Res-                 Methods
Net, two well-known neural network architectures in com-
puter vision are Faster R-CNN and Mask R-CNN (Figure 1)                     Image datasets
[13,14]. The first is performing “object detection”, a process              The primary source of dermatoscopic images was the
where one or multiple objects in an image can be detected                   HAM10000 dataset [19]. This dataset also includes pub-
and located with a rectangular “bounding box”. The latter                   licly available lesion segmentation masks for every image,
is performing “instance segmentation” where one or more                     as described previously, which are necessary for training the
objects in an image can be found and their respective area                  Faster R-CNN and Mask R-CNN architectures [8]. It con-
(i.e. pixels) in the image outlined (“segmented”), and can                  tains 10,015 images, each with 600x450 pixels and 3-8 bit
be regarded as a CNN-based multi-instance generalisation                    color channels. Each image is assigned one of seven diag-
of computer-vision based techniques of lesion segmentation                  nostic classes: actinic keratosis / intraepithelial carcinoma
                         APPROACH




                                            Classification         Object Detection         Instance Segmentation
                         INPUT
                         ARCHITECTURE




                                        ResNet-34            Faster R-CNN                 Mask R-CNN
                         OUTPUT




                                               Nevus


                                                                                                            Nevus
                                                                                 Nevus

                          Figure 1. A visual representation of the outputs of the three approaches. Image classifi-
                          cation (ie ResNet-34) classifies the image as a whole, object detection (ie Faster R-CNN)
                          finds objects and their approximate position in the image and instance segmentation (ie
                          Mask R-CNN) finds objects and their exact spatial delimitation.



2                                                                               Original Article | Dermatol Pract Concept. 2022;12(3):e2022126
(akiec), basal cell carcinoma (bcc), benign keratotic lesion          From those images we extracted the artefacts by manually re-
(bkl), dermatofibroma (df), nevus (nv), melanoma (mel), or            pairing the images areas with Adobe® Photoshop’s® (version
vascular lesion (vasc). Also, the PH2 and the 7-point criteria        CC 2018 (19.1.9), Adobe Inc.) content aware image repair
evaluation dataset were reviewed and several images were              mechanism and using the difference, per RGB channel, to the
utilized to extract artefacts from [20,21]. Images from those         untouched image as a template (Figure 2). The insertion of those
datasets were not used for other purposes within this study.          templates was done in a way that the position of artefacts var-
We used the ISIC2018 test-set as the test-set to keep varia-          ies according to observed patterns, using the provided segmen-
tion as low as possible, as it sources from the same origin as        tation mask of the target image. In Figure 3, a dermatoscopic
the HAM10000 dataset and includes the same classes.                   image with automatically superimposed artefacts is shown. The
                                                                      source code will be made available upon publication of this
Artefact generation                                                   work at https://github.com/thisismexp/artefact_insertion.
As with every real-world picture, dermatoscopic images can                Using the artefact insertion mechanism, several dataset
contain content considered as “artefacts”. Examples are hairs,        mutations of the original HAM10000 dataset were created,
dark corners, vignettes, medical devices, different sorts of rul-     where artefacts were superimposed on either none or all of the
ers, ink markings in different shapes, styles and colors, air         images and on every image belonging to a certain diagnosis.
bubbles or reflections. This work focuses on three of them:           The test portion of the HAM10000 dataset, corresponding to
“bubbles” that originate from trapped air in the liquid be-           the ISIC2018 challenge Task 3 test-set with 1,511 images, was
tween skin and the dermatoscope, “rulers” used to show the            altered in the same way. Additionally, artefacts were inserted
spatial dimension of a lesion, and ink “markings” on the pa-          in a certain percentage of images in 20% step increments.
tient’s skin used to highlight the lesion for excision or review.
    In order to generate artefact-modified cases, we selected 60      Neural Network Training
images from the HAM10000, PH2 and 7-point criteria dataset            As representatives for image classification, object detection
which contain either a bubble, a ruler or a marking artefact.         and instance segmentation we trained a ResNet-34, a Faster




                              Input: Original Image

                                                                             2.) Extraction of repaired regions




                                                                                      3.) RGB channel-Wise
                                                                                            difference

                        1.) Content-aware image repair




                                                                                   Output: Artefact Template




                  Figure 2. Workflow for extracting artefact templates. Manually selected original images (Input) were
                  repaired manually (1), and corresponding image areas extracted (2). The channel-wise difference
                  (3) was stored as a template for the corresponding artefact type.



Original Article | Dermatol Pract Concept. 2022;12(3):e2022126                                                                      3
Figure 3. Example of automatically superimposed artefacts on a dermatoscopic image. (A) In the top left the original image without artefact
is shown. The other 3 images show the lesion with the superimposed artefacts bubbles (B), ink markings (C) and a ruler (D).




R-CNN (with ResNet-34 backbone) and a Mask R-CNN                       Results
(also with a ResNet-34 backbone) model as provided by the
Torchvision package of the open source machine learning                Baseline performance in terms of PR-AUC of our models
framework PyTorch [22]. All models were trained on all of              trained and tested with no additional artefacts was 0.8 for
the 9 generated datasets in a 5-fold cross validation fashion.         ResNet-34 and 0.72 for Faster R-CNN as well as Mask
Transfer-learning and data augmentation including random               R-CNN. Introduction of artefacts in only the test dataset
crops, resize, rotations, mirroring operations as well as color        led to a reduction in performance for all three architectures
jitter operations were used.                                           (Figure 4) increasing with the proportion of artefacts pres-
                                                                       ent in the test dataset, and more severe for the ResNet-34
Statistics                                                             and Faster R-CNN model. With a maximum relative reduc-
To evaluate diagnostic accuracy, all trained network mod-              tion of 0.05 PR-AUC the Faster R-CNN model was affected
els are tested against the 13 test datasets and performance            the most, ResNet-34 (-0.03) the second most, and Mask
was reported in terms of area under the precision recall               R-CNN was the most robust (-0.01). For ResNet-34 and
curve (PR-AUC), precision, recall, false positive (FPR) and            Faster R-CNN, changes in predictive performance compared
false negative rates (FNR) and differences thereof (calcu-             to baseline was significant at and above 40% of introduced
lated using scikit-learn version 0.24.1) [23]. To visualize            artefacts in the test set (P < 0.01; tested using McNemar
spatial activations, Gradient based Class Activation Map               test with Edwards correction on binarized predictions). For
(Grad-CAM) visualizations were used. A two-sided p-value               Mask R-CNN we did not detect a significant difference in
of 0.05 was regarded as statistically significant, and all             predictions in all used test sets (all P values > 0.17).
                                                                           Introducing artefacts in the training data led to biased re-
calculations were performed using statsmodels version
                                                                       sults in all three examined architectures. Artefacts introduced
0.12.2 [24].



4                                                                           Original Article | Dermatol Pract Concept. 2022;12(3):e2022126
into all images of the melanocytic nevi class during training           (Figure 5). These mappings indicate an increase in robustness
decreased recall values on average by 0.218 for ResNet-34,              against these very artefacts for Faster and Mask R-CNN mod-
0.129 for Faster R-CNN and by 0.155 for Mask R-CNN                      els, if trained with inserted artefacts in the dataset.
in comparison to the respective unbiased models. Reduc-
tion in recall values indicate that those are indeed biased by
                                                                                           0.00
artefacts for specific classes. This effect was more apparent
the bigger the proportion of biased samples in the dataset is.




                                                                       Change in PR AUC
                                                                                          –0.02
Considering the FPR and FNR for specific classes, a selective
bias towards classes that were corrupted by artefacts during                              –0.04          Mask R-CNN
training could be observed for all three architectures. The                                              ResNet-34
                                                                                                         Faster R-CNN
increase in FPR for the class with inserted artefacts during                              –0.06

training, and a simultaneous increase in FNR for all others,                                      0.0        0.2         0.4          0.6         0.8   1.0
                                                                                                          Proportion of artefacts in the test set
in fact showed a shift in classifications towards the biased
class. This effect could not be observed if artefacts were in-          Figure 4. Neural networks show different robustness to inserted ar-
serted into none or all of the images.                                  tefacts on the test set. Precision recall curve (PR-AUC) as achieved
                                                                        by training without additional artefacts in the train and test set was
    When inspecting heat map representations of the Grad-
                                                                        used as the baseline (0%). With increasing proportion of inserted ar-
CAM we observed that training with artefacts shifted the                tefacts, PR-AUC decreases for ResNet-34 (blue) and Faster R-CNN
attention of the object detection and instance segmentation net-        (green), but almost not for Mask R-CNN (purple). Shaded areas
work away from the artefact itself towards areas of the lesion          denote 95%-confidence intervals.


                                                Trained without inserted artefacts
                                   Input                ResNet-34                         Faster R-CNN           Mask R-CNN

                           A                     B                     C                                   D




                           E                     F                     G                                   H




                                                     Trained with inserted artefacts
                                   Input                ResNet-34                         Faster R-CNN           Mask R-CNN
                            I                    J                     K                                    L




                           M                     N                     O                                   P




                           Figure 5. Grad-CAM for used network architectures. The first column shows the
                           input image for the corresponding row, in its original form (top) and with bubble
                           artefacts inserted (bottom). Grad-CAM heatmaps show the ResNet-34 increases
                           ­attention towards the bubble-area after training with artefacts (N), where the Faster
                            R-CNN network loses its initial attention towards the artefact (G) afterwards (O).
                            The Mask R-CNN architecture seems to ignore the artefact throughout (H and P).
                            Black boxes denote positions of inserted bubble artefacts.



Original Article | Dermatol Pract Concept. 2022;12(3):e2022126                                                                                                5
Conclusions                                                             5. Rubegni P, Burroni M, Cevenini G, et al. Digital dermoscopy
                                                                           analysis and artificial neural network for the differentiation of
We compared representatives of three neural network archi-                 clinically atypical pigmented skin lesions: a retrospective study.
                                                                           J Invest Dermatol. 2002;119(2):471-474. DOI: 10.1046/j.1523-
tectures to classify lesions in dermatoscopic images in regard
                                                                           1747.2002.01835.x. PMID: 12190872.
to their robustness against artefacts. Although as a limita-            6. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level clas-
tion the baseline performance of the examined models were                    sification of skin cancer with deep neural networks. Nature.
not the same, we found differences in their vulnerability to                 2017;542(7639):115-118. DOI: 10.1038/nature21056. PMID:
performance changes under the influence of artefacts. Mask                   28117445. PMCID: PMC8382232.
R-CNN tends to be the most robust. The influence on classi-             7.   Muñoz-López C, Ramírez-Cornejo C, Marchetti MA, et al. Per-
                                                                             formance of a deep neural network in teledermatology: a sin-
fication results by artefacts in test images can be reduced by
                                                                             gle-centre prospective diagnostic study. J Eur Acad Dermatol
augmenting training data with artificially superimposed arte-                Venereol. 2021;35(2):546-553. DOI: 10.1111/jdv.16979. PMID:
facts for all three architectures. This is in line with findings by          33037709. PMCID: PMC8274350.
Maron et al, who reduced - but not eliminated - brittleness of          8.   Tschandl P, Rinner C, Apalla Z, et al. Human–computer collab-
their system through data augmentation [25]. We anticipate                   oration for skin cancer recognition. Nat Med. 2020;26(8):1229-
that automated superimposition of artefacts as presented                     1234. DOI: 10.1038/s41591-020-0942-0. PMID: 32572267.
                                                                        9.   Fink C, Blum A, Buhl T, et al. Diagnostic performance of a deep
here as a further evolution of data augmentation, that to-
                                                                             learning convolutional neural network in the differentiation of
gether with integrating more diverse variants, will enhance                  combined naevi and melanomas. J Eur Acad Dermatol Vene-
robustness of automated classifiers and decision support sys-                reol. 2020;34(6):1355-1361. DOI: 10.1111/jdv.16165. PMID:
tems further [26,27]. The initial data, in our view, warrants                31856342.
more in-depth follow up research on this topic, to understand          10.   Okuboyejo DA, Olugbara OO. A review of prevalent meth-
which approaches are the most effective and efficient.                       ods for automatic skin lesion diagnosis. Open Dermatol J.
                                                                             2018;12(1):14-53. DOI: 10.2174/187437220181201014
    However, this work failed to find evidence for a clinically
                                                                       11.   Winkler JK, Fink C, Toberer F, et al. Association Between Surgi-
relevant robustness against artefacts of instance segmentation               cal Skin Markings in Dermoscopic Images and Diagnostic Per-
for several reasons. On the one hand we used a shallow back-                 formance of a Deep Learning Convolutional Neural Network for
bone network architecture for our experiments, even though                   Melanoma Recognition. JAMA Dermatol. 2019;155(10):1135-
current research and commercial products commonly use                        1141.     DOI:     10.1001/jamadermatol.2019.1735.       PMID:
deeper models, and an increase in robustness against image                   31411641. PMCID: PMC6694463.
                                                                       12.   Navarrete-Dechent C, Dusza SW, Liopyris K, Marghoob AA,
distortions has been demonstrated by others with increased
                                                                             Halpern AC, Marchetti MA. Automated Dermatological Diag-
backbone capacity [28]. We also used a new template-based                    nosis: Hype or Reality? J Invest Dermatol. 2018;138(10):2277-
approach to superimpose artefacts on images. This approach                   2279. DOI: 10.1016/j.jid.2018.04.040. PMID: 29864435.
leaves room for improvement with regard to the number of                     PMCID: PMC7701995.
images the artefacts are extracted from, and a detailed anal-          13.   Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards Re-
ysis on how different artefact types affect the classification               al-Time Object Detection with Region Proposal Networks. IEEE
                                                                             Transactions on Pattern Analysis and Machine Intelligence.
performance. Alternatively, lesions with existing artefacts
                                                                             2017;39(6):1137-1149. . DOI: 10.1109/TPAMI.2016.2577031.
could be used after manual or automated annotations.                         PMID: 27295650.
                                                                       14.   He K, Gkioxari G, Dollar P, Girshick R. Mask R-CNN. 2017
                                                                             IEEE International Conference on Computer Vision (ICCV).
References                                                                   2017:2980-2988. DOI: 10.1109/ICCV.2017.322.
 1. Apalla Z, Lallas A, Sotiriou E, Lazaridou E, Ioannides D. Ep-      15.   Kaur R, LeAnder R, Mishra NK, et al. Thresholding methods
    idemiological trends in skin cancer. Dermatol Pract Concept.             for lesion segmentation of basal cell carcinoma in dermoscopy
    2017;7(2):1-6. DOI: 10.5826/dpc.0702a01. PMID: 28515985.                 images. Skin Res Technol. 2017;23(3):416-428. DOI: 10.1111/
    PMCID: PMC5424654.                                                       srt.12352. PMID: 27892649.
 2. Balch CM, Soong S-J, Atkins MB, et al. An evidence-based           16.   Mishra NK, Kaur R, Kasmi R, et al. Automatic lesion border
    staging system for cutaneous melanoma. CA Cancer J Clin.                 selection in dermoscopy images using morphology and color
    2004;54(3):131-149; quiz 182-184. DOI: 10.3322/canj-                     features. Skin Res Technol. 2019;25(4):544-552. DOI: 10.1111/
    clin.54.3.131. PMID: 15195788.                                           srt.12685. PMID: 30868667. PMCID: PMC7173402.
 3. Kittler H, Pehamberger H, Wolff K, Binder M. Diagnostic ac-        17.   Han SS, Moon IJ, Lim W, et al. Keratinocytic Skin Cancer De-
    curacy of dermoscopy. Lancet Oncol. 2002;3(3):159-165. DOI:              tection on the Face Using Region-Based Convolutional Neu-
    10.1016/s1470-2045(02)00679-4. PMID: 11902502.                           ral Network. JAMA Dermatol. 2020;156(1):29-37. DOI:
 4. Korotkov K, Garcia R. Computerized analysis of pigmented                 10.1001/jamadermatol.2019.3807. PMID: 31799995. PMCID:
    skin lesions: a review. Artif Intell Med. 2012;56(2):69-90. DOI:         PMC6902187.
    10.1016/j.artmed.2012.08.002. PMID: 23063256.                      18.   Lee T, Ng V, Gallagher R, Coldman A, McLean D. Dull-
                                                                             razor: A software approach to hair removal from images.




6                                                                            Original Article | Dermatol Pract Concept. 2022;12(3):e2022126
      Comput Biol Med. 1997;27(6):533-543. DOI: 10.1016/s0010-           23. Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: Ma-
      4825(97)00020-6. PMID: 9437554.                                        chine learning in Python. the Journal of machine Learning re-
19.   Tschandl P, Rosendahl C, Kittler H. The HAM10000 dataset, a            search. 2011;12:2825-2830. Available from: https://www.jmlr.
      large collection of multi-source dermatoscopic images of common        org/papers/volume12/pedregosa11a/pedregosa11a.pdf
      pigmented skin lesions. Sci Data. 2018;5:180161. DOI: 10.1038/     24. Seabold S, Perktold J. Statsmodels: Econometric and statistical
      sdata.2018.161. PMID: 30106392. PMCID: PMC6091241.                     modeling with python. In: Proceedings of the 9th Python in Science
20.   Mendonça T, Ferreira PM, Marques JS, Marcal ARS, Rozeira J.            Conference. Vol 57. Austin, TX; 2010:61. Available from: https://
      PH2 - A dermoscopic image database for research and bench-             conference.scipy.org/proceedings/scipy2010/pdfs/seabold.pdf
      marking. In: 2013 35th Annual International Conference of the      25. Maron RC, Haggenmüller S, von Kalle C, et al. Robustness of
      IEEE Engineering in Medicine and Biology Society (EMBC).;              convolutional neural networks in recognition of pigmented
      2013:5437-5440. DOI: 10.1109/EMBC.2013.6610779. PMID:                  skin lesions. Eur J Cancer. 2021;145:81-91. DOI: 10.1016/j.
      24110966.                                                              ejca.2020.11.020. PMID: 33423009.
21.   Kawahara J, Daneshvar S, Argenziano G, Hamarneh G. Sev-            26. Aggarwal SLP. Data augmentation in dermatology image recogni-
      en-Point Checklist and Skin Lesion Classification Using Mul-           tion using machine learning. Skin Res Technol. 2019;25(6):815-
      titask Multimodal Neural Nets. IEEE Journal of Biomedical              820. DOI: 10.1111/srt.12726. PMID: 31140653.
      and Health Informatics. 2019;23(2):538-546. DOI: 10.1109/          27. Winkler JK, Sies K, Fink C, et al. Association between different
      JBHI.2018.2824327. PMID: 29993994.                                     scale bars in dermoscopic images and diagnostic performance of
22.   Paszke A, Gross S, Massa F, et al. PyTorch: An Imperative Style,       a market-approved deep learning convolutional neural network
      High-Performance Deep Learning Library. In: Wallach H, La-             for melanoma recognition. Eur J Cancer. 2021;145:146-154.
      rochelle H, Beygelzimer A, d’ Alché-Buc F, Fox E, Garnett R,           DOI: 10.1016/j.ejca.2020.12.010. PMID: 33465706.
      eds. Advances in Neural Information Processing Systems 32.         28. Michaelis C, Mitzkus B, Geirhos R, et al. Benchmarking Robust-
      Curran Associates, Inc.; 2019:8026-8037. Available from:               ness in Object Detection: Autonomous Driving when Winter is
      https://papers.nips.cc/paper/2019/hash/bdbca288fee7f92f2b-             Coming. arXiv [csCV]. Published online July 17, 2019. Available
      fa9f7012727740-Abstract.html                                           from: http://arxiv.org/abs/1907.07484




Original Article | Dermatol Pract Concept. 2022;12(3):e2022126                                                                               7
