10.5826/dpc.1302a105
Introduction: Case-based training improves novices pattern recognition and diagnostic accuracy in skin cancer diagnostics. However, it is unclear how pattern recognition is best taught in conjunction with the knowledge needed to justify a diagnosis.
Objectives: The aim of this study was to examine whether an explanation of the underlying histopathological reason for dermoscopic criteria improves skill acquisition and retention during case-based training in skin cancer diagnostics.
Methods: In this double-blinded randomized controlled trial, medical students underwent eight days of case-based training in skin cancer diagnostics, which included access to written diagnosis modules. The modules dermoscopic subsections differed between the study groups. All participants received a general description of the criteria, but the intervention group additionally received a histopathological explanation.
Results: Most participants (78%) passed a reliable test in skin cancer diagnostics, following a mean training time of 217 minutes. Access to histopathological explanations did not affect participants' learning curves or skill retention.
Conclusions: The histopathological explanation did not affect the students, but the overall educational approach was efficient and scalable.
text/xml
2160-9381
2751
en
Annette Chakera
https://creativecommons.org/licenses/by-nc/4.0
Dermatology Practical & Conceptual
carcinoma
Effect of Histopathological Explanations for Dermoscopic Criteria on Learning Curves in Skin Cancer Training: A Randomized Controlled Trial
Text.Serial.Journal
https://dpcj.org/index.php/dpc
Original Article
2023-04-29
2022-08-08
2023-04-29
2023-05-31
e2023105
Dermatol Pract Concept
Introduction: Case-based training improves novices pattern recognition and diagnostic accuracy in skin cancer diagnostics. However, it is unclear how pattern recognition is best taught in conjunction with the knowledge needed to justify a diagnosis.
Objectives: The aim of this study was to examine whether an explanation of the underlying histopathological reason for dermoscopic criteria improves skill acquisition and retention during case-based training in skin cancer diagnostics.
Methods: In this double-blinded randomized controlled trial, medical students underwent eight days of case-based training in skin cancer diagnostics, which included access to written diagnosis modules. The modules dermoscopic subsections differed between the study groups. All participants received a general description of the criteria, but the intervention group additionally received a histopathological explanation.
Results: Most participants (78%) passed a reliable test in skin cancer diagnostics, following a mean training time of 217 minutes. Access to histopathological explanations did not affect participants' learning curves or skill retention.
Conclusions: The histopathological explanation did not affect the students, but the overall educational approach was efficient and scalable.
Annette Chakera
2023/04/29
10.5826/dpc.1302a105
e2023105
Department of Plastic Surgery, Copenhagen University Hospital, Herlev and Gentofte, Copenhagen, Denmark
2160-9381
carcinoma
en
e2023105
Fox, M. C., Ericsson, K. A. & Best, R. Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychol Bull 137, 316–344 (2011).
Effect of Histopathological Explanations for Dermoscopic Criteria on Learning Curves in Skin Cancer Training: A Randomized Controlled Trial
https://dpcj.org/index.php/dpc/article/download/2751/2064
      Effect of Histopathological Explanations for
    Dermoscopic Criteria on Learning Curves in Skin
     Cancer Training: a Randomized Controlled Trial
  Niels Kvorning Ternov1, Martin Tolsgaard2,3, Lars Konge2,3, Anders Nymark Christensen4,
   Sigrid Isabella Pilgaard Kristensen1, Lisbet Rosenkrantz Hölmich1,2, Jonathan Stretch5,6,
             Richard Anthony Scolyer5,6,7,8, Tine Vestergaard9, Pascale Guitera5,6,10,
                                 Annette Hougaard Chakera1,2


 1 Department of Plastic Surgery, Copenhagen University Hospital, Herlev and Gentofte, Copenhagen, Denmark
 2 Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
 3 Copenhagen Academy for Medical Education and Simulation, Rigshospitalet, Copenhagen, Denmark
 4 Department of Applied Mathematics and Computer Science, Technical University of Denmark, Kongens Lyngby, Denmark
 5 Melanoma Institute Australia, The University of Sydney, Sydney, New South Wales, Australia
 6 Faculty of Medicine and Health, The University of Sydney, Sydney, New South Wales, Australia
 7 Tissue Pathology and Diagnostic Oncology, Royal Prince Alfred Hospital and New South Wales Health Pathology, Sydney,
   New South Wales, Australia
 8 Charles Perkins Centre, The University of Sydney, Sydney, New South Wales, Australia
 9 Department of Dermatology and Allergy Center, Odense University Hospital, Denmark
10 The Sydney Melanoma Diagnosis Centre, Royal Prince Alfred Hospital, Sidney, Australia



Key words: education, melanoma, pigmented lesions, dermatopathology, carcinoma
Citation: Kvorning Ternov N, Tolsgaard M, Konge L, et al. Effect of Histopathological Explanations for Dermoscopic Criteria on
Learning Curves in Skin Cancer Training: A Randomized Controlled Trial. Dermatol Pract Concept. 2023;13(2):e2023105.
DOI: https://doi​.org/10.5826/dpc.1302a105
Accepted: October 20, 2022; Published: April 2023
Copyright: ©2023 Kvorning Ternov et al. This is an open-access article distributed under the terms of the Creative Commons
Attribution-NonCommercial License (BY-NC-4.0), https://creativecommons.org/licenses/by-nc/4.0/, which permits unrestricted
noncommercial use, distribution, and reproduction in any medium, provided the original authors and source are credited.
Funding: None.
Competing Interests: NKT is CEO and co-founder of the start-up MelaTech ApS that developed the educational mobile application used
in this trial. RAS has received fees for professional services from F. Hoffmann-La Roche Lt​d, Evaxion, Provectus Biopharmaceuticals
Australia, Qbiotics, Novartis, Merck Sharp & Dohme, NeraCare, AMGEN Inc., Bristol-Myers Squibb, Myriad Genetics, GlaxoSmithKline.
Authorship: All authors have contributed significantly to this publication.
Corresponding Author: Niels Kvorning Ternov, Dep. of Plastic Surgery, Herlev and Gentofte University Hospital, Borgmester Ib Juuls Vej 1,
2370 Herlev, Denmark. E-mail: niels.kvorning.ternov@regionh.dk




  ABSTRACT             Introduction: Case-based training improves novices pattern recognition and diagnostic accuracy in
                       skin cancer diagnostics. However, it is unclear how pattern recognition is best taught in conjunction
                       with the knowledge needed to justify a diagnosis.
                       Objectives: The aim of this study was to examine whether an explanation of the underlying histo-
                       pathological reason for dermoscopic criteria improves skill acquisition and retention during case-based
                       training in skin cancer diagnostics.




Original Article | Dermatol Pract Concept. 2023;13(2):e2023105                                                                         1
                     Methods: In this double-blinded randomized controlled trial, medical students underwent eight days
                     of case-based training in skin cancer diagnostics, which included access to written diagnosis modules.
                     The modules dermoscopic subsections differed between the study groups. All participants received a
                     general description of the criteria, but the intervention group additionally received a histopathological
                     explanation.
                     Results: Most participants (78%) passed a reliable test in skin cancer diagnostics, following a mean
                     training time of 217 minutes. Access to histopathological explanations did not affect participants’
                     learning curves or skill retention.
                     Conclusions: The histopathological explanation did not affect the students, but the overall education-
                     al approach was efficient and scalable.




Introduction                                                     Methods
Skin cancer is the most common malignancy among                  In this double-blinded randomized controlled trial (alloca-
fair-skinned people worldwide [1,2]. Early detection and         tion ratio 1:1), we enrolled medical students with no prior
treatment of skin cancers reduce patient mortality and           experience in skin cancer diagnostics that had previously
the associated socioeconomic costs [3]. In most countries,       passed an exam in general histology. The students were in-
skin cancer triage is performed by personnel without stan-       vited to participate through a Facebook group for Danish
dardized training in the domain [4,5]. Courses that teach        medical students. Participants were enrolled through virtual
structured checklists for skin cancer diagnostics such as the    meetings between the 8th and 27th of July 2021. During in-
ABCD (Asymmetry, Borders, Colour, and Diameter) algo-            clusion, we helped participants download and get started
rithm improve novices short-term accuracy, but frequent          in the educational application (onboarding process). Par-
refresher courses are needed to maintain the skill [6,7]. It     ticipants were automatically and randomly (simple ran-
is unclear if the training effect is caused by the structured    domization) assigned to the intervention or control group
checklists or the simultaneous exposure to many skin lesion      during the onboarding process. The principal investigator
images. A Cochrane review recently stated that structured        (N.K.T.), a student assistant (S.K.), and participants were
checklists do not improve clinicians accuracy in skin cancer     all blinded towards trial group allocation. Following the on-
diagnostics [8]. Case-based training improves novices ac-        boarding process, all participants underwent a pre-test and
curacies in skin cancer diagnostics significantly more than      were instructed to diagnose 500 skin lesion cases (including
structured checklist practice [9-11]. Exposure to annotated      the pre-test cases) over 8 days (training phase), pause for
skin lesion images improves novices pattern recognition,         14 days (washout phase 1), diagnose another 100 cases in
which is the primary diagnostic strategy of experts [12-14].     2 days (retention phase), pause for another 7 days (washout
It is unclear how pattern recognition and the declarative        phase 2), and finally, complete a retention test (Figure 1). All
knowledge needed to justify a diagnosis, are best taught in      participants received daily reminders during the training and
conjunction. Related work from odontology suggests that          retention phases, and those that completed the entire study
teaching the underlying biomedical reason for visual crite-      received a certificate of completion. During the training and
ria improves the students ability to recognize and remem-        retention phase, participants had access to written learning
ber the criteria [15,16].                                        modules that described the most common skin lesion diag-
                                                                 noses. The learning modules included a dermoscopy sub-
                                                                 section that differed between the trial groups. Participants
Objectives                                                       in the control group saw a brief description of the dermo-
This study investigated a novel approach towards teaching        scopic criteria, while the intervention group saw the same
skin cancer diagnostics through a mobile educational appli-      description supplemented with a histopathological expla-
cation. Our primary objective was to examine whether an          nation (Figure 2). Neither group were informed about the
explanation of the underlying histopathological reason for       group-dependent learning module differences. We planned
the dermoscopic criteria used in skin cancer diagnostics af-     and conducted the study per the principles of the Declara-
fects medical students’ learning curves and skill retention.     tion of Helsinki. Participants were informed about the study
We hypothesized that a deeper biomedical understanding of        before participation and gave informed consent. The study
the dermoscopic criteria would improve the students’ ability     was voluntary, held no consequences for the participants,
to recognize and remember them.                                  and received a waiver from the Regional Ethics Committee



2                                                                     Original Article | Dermatol Pract Concept. 2023;13(2):e2023105
 Figure 1. Trial flow.
 Participants performed a pre-test (12 cases) at the beginning and a retention test (25 cases) at the end of the trial. During the training
 and retention phases, each participant practiced skin lesion diagnostics on 500 (including the pre-test cases) and 100 training cases, re-
 spectively, while accessing the learning modules of their own accord. We instructed participants to abstain from any training during both
 washout phases.




           Figure 2. Educational intervention (mobile application).
           The educational intervention consisted of an educational mobile application that included quizzes and written learning
           modules. The red circles within the figure indicate where users “press” the mobile screen to proceed towards the next
           screen, indicated by the red arrow. The “Quiz feature” presents skin lesions for diagnostics. The small images representing
           the clinical image (A) and the avatar (B) are buttons that open the clinical image and 3D avatar. When users press “Benign”
           (C) or “Malignant”, an array of new buttons representing the various benign or malignant differential diagnoses appear.
           When users press one of the diagnosis buttons (D), they receive immediate feedback. The feedback consists of the chosen
           diagnosis, the correct diagnosis, and access to learning modules on both the chosen and correct diagnoses (E). Each learn-
           ing module consists of the following sections: introduction, pathology, clinical presentation, dermoscopy, differential diag-
           noses, and references. The dermoscopy sections included an overview and subsections describing the primary dermoscopic
           criteria. Each subsection included a detailed description of the dermoscopic criterium (F). Users from both trial groups
           received descriptions and annotated images representing the dermoscopic criteria. However, the subsections presented to
           the intervention group participants also explained the underlying histopathological correlation for the dermoscopic crite-
           ria. When the learning modules in the application are closed (G), users return to the previous training case feedback page.



Original Article | Dermatol Pract Concept. 2023;13(2):e2023105                                                                             3
of Region Hovedstaden, Denmark (jr nr. H-20066667). The             Participants received immediate feedback on their quiz di-
Danish Health Data Authorities and Data Protection Agency           agnoses, including the correct diagnosis and access to the
approved access, anonymization, handling, and storage of            aforementioned learning modules. Time spent reading the
the skin lesion cases (jr. nr. 21/5103 and 18/53664). We sub-       learning modules and diagnosing the quiz cases were auto-
mitted a study protocol on clinicaltrial.gov prior to initiating    matically registered throughout the study.
the study (identifier: NCT05087485).
                                                                    Pre- and Retention Test
Skin Lesions Library                                                The pre- and retention tests consisted of skin lesion cases from
We developed a case library consisting of 2,376 anonymous           a test item library with validity evidence previously described
skin lesions for this study. Each lesion belonged to one of the     by our group [4]. The pre-test consisted of 12 randomly sam-
following seven diagnostic groups: nevus, seborrheic kerato-        pled test items (Generalizability coefficient of 0.7), while the
sis/solar lentigo, dermatofibroma, hemangioma, melanoma,            retention test included all 25 test items (Cronbach α of 0.83).
basal cell carcinoma, and squamous cell carcinoma. Each             A former pass-fail test revealed a pass-fail limit of twelve, ie a
case included a clinical and dermoscopic image of the lesion,       score above 12/25 is enough to pass the test [4].
the lesion location on a human 3D avatar, a diagnosis, and
the patient age and gender. The lesions diagnoses were based        Statistics
on either a histopathological assessment (N = 1,293) or a           Data was divided into a training (0-500 cases) and a reten-
clinical consensus (N = 1,083), consisting of a joint judgment      tion phase (501-600 cases). A mixed-effects logistic regres-
by 2-3 clinicians. All images were captured by nurses and           sion model with correct or false case-answers as an outcome
doctors at the Department of Dermatology and Allergy Cen-           and a random intercept and slope for each individual was
tre, Odense University Hospital, in Denmark, from the 1st           applied to estimate the learning curves on the log-odds scale.
of September 2010 until the 8th of May 2021. D
                                             ­ ermoscopic           We sought the most straightforward description of the partic-
images were photographed using digital dermoscopes                  ipants learning curves by comparing (likelihood-ratio tests)
(­Medicam 800 and 1000, Fotofinder Systems GmbH).                   cubic and linear spline models with four, one, or zero knots.
                                                                    The retention phase data were described using a simple line.
Written Learning Modules                                            Once we had located the optimal statistical equation for de-
The mobile application’s written content consisted of 38 di-        scribing the training phase, we compared the control and
agnosis (eg melanoma) and sub-diagnosis learning modules            intervention groups training and retention learning curves
(eg superficial spreading melanoma). Each sub-diagnosis             using likelihood-ratio tests. The control group retention test
module included the following subsections: introduction,            results were compared to those of the intervention group us-
pathology, clinical presentation, dermoscopy, differential di-      ing the Welch t-test. For exploratory post hoc analyses, we
agnoses, and references (Figure 2). We created two versions         used the participants test scores on the retention test to di-
of each dermoscopic sub-section; one described the dermo-           vided them into three equally big performance groups; low-
scopic criteria with annotated images (control group), and          (1st tertile), intermediate- (2nd tertile) and high- (3rd tertile)
another version that additionally explained the histopatho-         performance. We compared the learning curves, time spent
logical correlation for each dermoscopic criterion (inter-          reading, and time spent diagnosing training cases between
vention group) (Figure 2). All diagnosis and sub-diagnosis          the 1st and 3rd tertile using likelihood ratio tests and Welch
modules were written by the first author (N.T.) and reviewed        t-tests. All statistical analyses were performed in R version
by content experts in pathology, dermatology, and skin can-         4.1.0 (R Foundation for Statistical Computing).
cer surgery (co-authors: A.C., P.G., T.V., R.S., L.H., and J.S.).

                                                                    Results
Mobile Application
In this study, we employed a mobile application for training        Eighty-seven medical students were enrolled, and 76 com-
skin lesion diagnostics, called Dermloop Learn (Melatech            pleted the entire trial, see the consort diagram in Figure 3.
ApS), developed in cooperation with our group (Figure 2).
A continuously updated version of the application can be            Retention Test
accessed online (https://training.dermloop.io/) or through          Results on the retention test were equal (t= 0.13, degrees of
the app store (“Dermloop Learn”). The application included          freedom (df) = 71.3, P = 0.90) for the intervention (mean:
three functionalities: skin lesion quizzes, written learn-          13.8, SD: 3.06) and control (mean: 13.9, SD: 3.35) groups.
ing modules, and user tracking. Each quiz consisted of ten          Fifty-nine (78%) out of the 76 participants passed the re-
randomly sampled skin lesion cases from the case library.           tention test (>12/25 correct answers). The 76 participants




4                                                                       Original Article | Dermatol Pract Concept. 2023;13(2):e2023105
                                                  ASSESSED FOR ELIGIBILITY (N=87)



                                                                         EXCLUDED (N= 0)



                                                    RANDOMIZED (N= 87)




  INTERVENTION GROUP (N=44)                                              CONTROL GROUP (N= 43)
  (DESCRIPTION OF CRITERIA + HISTOPATHOLOGICAL EXPLANATIONS)             (DESCRIPTION OF CRITERIA)
  -MALE (N=9)                                                            -MALE (N= 8)
  -FEMALE (N= 35)
                                                                         -FEMALE (N= 35)



   FALLOUT (N=2)                                                         FALLOUT (N=3)



                    TRAINING PHASE (N=42)                                             TRAINING PHASE (N= 40)



   FALLOUT (N= 1)                                                        FALLOUT (N=4)



                    RETENTION PHASE (N=41)                                                RETENTION PHASE (N= 36)



   FALLOUT (N= 1)                                                        FALLOUT (N= 0)



                    RETENTION TEST (N= 40)                                               RETENTION TEST (N= 36)



    PART OF THE INTENDED PROGRAM (N=42)                                  PART OF THE INTENDED PROGRAM (N= 40)
    PARTICIPANTS WITH DATA FOR ANALYSIS OF:                              PARTICIPANTS WITH DATA FOR ANALYSIS OF:
    -LEARNING CURVES (N=42)                                              -LEARNING CURVES (N=40)
    -T-TEST RETENTION TEST (N=40)                                        -T-TEST RETENTION TEST (N= 36)
    -TIME USED READING AND PRACTICING (N= 40)                            -TIME USED READING AND PRACTICING (N= 36)


 Figure 3. Consort diagram.


who completed the retention test were split into low-,              intervention and control groups (training phase: χ2= 0.35,
intermediate-, and high-performance groups (N = 25, 25,             df = 2, P = 0.83, retention phase: χ2=0.94, df = 1, P = 0.33).
and 26) based on their test results. Intervention and con-          The learning curves of the intervention and control group
trol participants were equally distributed across the low           participants were also equal within the low- and high-perfor-
(N = 14/11) and high-performance (N = 14/12) groups.                mance groups (training phase: χ2= 0.80, df = 2, P = 0.67, re-
                                                                    tention phase: χ2=0.15, df = 1, P = 0.70). However, there was
Learning Curves and Time Spent Training                             a significant difference between the overall (intervention +
An almost straight line with a single knot, ie breaking point,      control) learning curves of the low- versus high-performance
at 100 cases provided a significantly better data-fit for the       groups, (training phase: χ2= 25.71, df = 1, P = <0.01, reten-
                                                        2
training phase than a straight line without knots (χ = 125.0,       tion phase: χ2=15.29, df = 1, P < 0.01).
df = 2, P = <0.01). There was no added benefit from adding               The mean time spent training was 217 minutes,
three additional knots at 200, 300, and 400 training cases          117 ­
                                                                        minutes diagnosing training cases, and 100 minutes
(χ2= 7.5, df = 6, P = 0.28), or performing a cubic transforma-      reading the learning modules. There was no difference in
tion (χ2= 4.3, df = 4, P = 0.37) (Figure 4).                        time spent diagnosing cases (t= -0.03, df = 51.9, P = 0.98)
    There were no significant learning curve differences            or reading (t= -0.02, df = 73.9, P = 0.98) between the in-
in the training or retention phase, when comparing the              tervention and control participants. The high-performance




Original Article | Dermatol Pract Concept. 2023;13(2):e2023105                                                                  5
                                                                                    SIMPLE LINE                                                                                                                SPLINE WITH 1 KNOT
                                      A                                                                                                                                            B

                                                   0.6




                                                                                                                                                                                   0.6
                                                                                                                                                                                                    KNOT 1




                                                                                                                                         ESTIMATED PROBABILITY OF CORRECT ANSWER
         ESTIMATED PROBABILITY OF CORRECT ANSWER
                                                   0.5




                                                                                                                                                                                   0.5
                                                   0.4




                                                                                                                                                                                   0.4
                                                   0.3




                                                                                                                                                                                   0.3
                                                                                                                         CONTROL                                                                                                                   CONTROL
                                                                                                                         INTERVENTION                                                                                                              INTERVENTION
                                                   0.2




                                                                                                                                                                                   0.2
                                                         0        100            200             300             400               500                                                   0           100        200          300             400             500
                                                                          NUMBER OF COMPLETED CASES                                                                                                          NUMBER OF COMPLETED CASES
                                                                                SPLINE WITH 4 KNOTS                                                                                                               CUBIC SPLINE
                                               C                                                                                                                               D
                                                   0.6




                                                                                                                                                                                   0.6
                                                                                                                                         ESTIMATED PROBABILITY OF CORRECT ANSWER
                                                                KNOTS 1        KNOTS 2         KNOTS 3         KNOTS 4
     ESTIMATED PROBABILITY OF CORRECT ANSWER




                                                                                                                                                                                   0.5
                                                   0.5




                                                                                                                                                                                   0.4
                                                   0.4




                                                                                                                                                                                   0.3
                                                   0.3




                                                                                                                         CONTROL                                                                                                                   CONTROL
                                                                                                                         INTERVENTION                                                                                                              INTERVENTION
                                                                                                                                                                                   0.2
                                                   0.2




                                                         0        100            200            300             400                500                                                   0          100         200          300             400             500
                                                                          NUMBER OF COMPLETED CASES                                                                                                          NUMBER OF COMPLETED CASES


         Figure 4. Learning curve models for the training phase.
         The figure depicts the various statistical equations applied to the data. Red dashed, and solid black lines represent the intervention and
         control groups, respectively. (B) An almost straight line with one knot provided a simple yet reliable fit for the data. (C,D) Increasing
         the complexity of the model by introducing additional knots (C) or a cubic function (D) did not provide any additional value com-
         pared to model B.


                                                                              Table 1. Time spent training within the mobile application.
    Activity                                                                  Time spent reading (min)                                                                                          Time spent diagnosing training cases (min)
                                                                                                          st                   rd
 Group                                                       Intervention          Control               1 tertile            3 tertile                                                      Intervention    Control           1st tertile         3rd tertile
 Mean                                                           100.7                  100.1               94.7                    90.4                                                         117.1          116.6               80.9               124.5
 SD                                                             121.9                  105.7              150.1                    56.6                                                         49.2             93                32.2               48.4

min = minutes; SD = standard deviation.
Time spent reading learning modules and diagnosing training cases among the participants within both study groups (intervention, control)
and performance groups (1st and 3rd tertile). The 1st and 3rd tertile groups consist of the 33% participants with the lowest and highest per-
formance on the retention test.

group spent the same amount of time reading (t= 0.13,                                                                                                                              (78%), irrespective of trial groups, passed a reliable test in
df = 30.5, P = 0.90) and significantly more time diagnosing                                                                                                                        skin cancer diagnostics, following a mean training time of
the training cases (t=-3.8, df = 43.8, P = <0.001) compared                                                                                                                        217 minutes. Access to explanations of the histopathological
to the low-performance group (Table 1). Participants spent                                                                                                                         correlation for dermoscopic criteria did not affect partici-
40% of their training time on the first 100 cases (mean: 93.8                                                                                                                      pants learning curves or skill retention.
min) and the remaining time on the last 500 cases (mean:                                                                                                                                 The rapid diagnostic improvement observed in this study
146.8 min).                                                                                                                                                                        resonates with former studies on case-based pattern recog-
                                                                                                                                                                                   nition training for skin cancer diagnostics [9,10]. This pro-

Conclusions                                                                                                                                                                        posed model for teaching complex visual diagnostics could
                                                                                                                                                                                   potentially democratize skin cancer diagnostics. Primary
This study explored a novel approach towards digital training                                                                                                                      care providers, nurses, and medical students can be educated
in skin cancer diagnostics. The vast majority of participants                                                                                                                      in mass, improving access to high-quality skin cancer triage



6                                                                                                                                                                                        Original Article | Dermatol Pract Concept. 2023;13(2):e2023105
                                                                       LEARNING CURVES


                                                                   TRAINING PHASE                                     RETENTION PHASE
                                             0,6



                                            0,55
  ESTIMATED PROBABILIY OF CORRECT ANSWERS




                                             0,5



                                            0,45



                                             0,4
                                                                                                             WASHOUT PERIOD
                                                                                                                (14 DAYS)
                                            0,35
                                                                             TOTAL INTERVENTION

                                                                             TOTAL CONTROL
                                             0,3
                                                                             LOW INTERVENTION

                                                                             LOW CONTROL
                                            0,25                             HIGH INTERVENTION

                                                                             HIGH CONTROL
                                             0,2
                                                   0   100   200               300                400              500               600
                                                                      NUMBER OF COMPLETED CASES

 Figure 5. Learning curves during the training and retention phases.
 The diagram depicts the learning curves for the entire group (intervention: dark blue, control: red), the high-performance group (inter-
 vention: light blue, control: orange), and the low-performance group (intervention: green, control: purple).


and reducing some of the current inequality in melanoma                       postpone the deceleration of learning curves, further dis-
mortality [17,18]. Improved competencies could also pave                      cussed below.
the way for clinical implementation diagnostic artificial in-                       During post hoc analyses, we found that high-performers
telligence, providing a human safeguard against the algo-                     were more accurate than the low-performers through-
rithms erroneous predictions and inherent biases [19].                        out the study. High-performers were more accurate than
                    According to the “deliberate practice framework” for      low-performers already at the beginning of the trial, despite
teaching diagnostics, novice learners need assistance from                    all participants being supposedly equally inexperienced.
a domain expert, as they lack the competencies needed to                      These findings suggest that some participants were dis-
identify and address their knowledge gaps [20]. A synthetic                   honest about their expertise during inclusion or possessed
algorithm-driven domain expert could, in theory, be devel-                    a superior innate diagnostic accuracy. Regardless of the
oped and integrated as a digital mentor within learning inter-                difference in diagnostic accuracy, both groups maintained
ventions such as the one used in this study. A digital mentor                 parallel learning curves throughout the trial. In theory, the
could be taught how to identify a students learning pattern,                  accuracy of low-performers should have increased faster
weaknesses, and strengths based on prior students training                    than it did for the high-performers. It is significantly more
data. The identified patterns could then be used to select                    challenging to increase one’s diagnostic accuracy when it
and present the instructional material (cases, modules) most                  is high compared to low [21]. High-performers likely had
likely to increase the student competencies at any given time.                a more efficient and intentional learning strategy through-
Such individualized approaches could potentially reduce the                   out the trial. This hypothesis is supported by the fact that
observed difference between low- and high-performers and                      high-performers spent 35% more time diagnosing training



Original Article | Dermatol Pract Concept. 2023;13(2):e2023105                                                                              7
cases than low-performers. Learning curves decelerated fol-         explanations for dermoscopic criteria did not affect the stu-
lowing the first 100 training cases, possibly due to a faltering    dents knowledge acquisition and retention.
engagement, poor clinical contextualization, or a plateau in
the trainees ability to recognize and mitigate their diagnos-
                                                                    Acknowledgments: We wish to extend a special thanks to
tic errors. On average, each participant spent almost the
                                                                    Assistant Professor Morten Hannemose and statistician
                                                                    ­
same amount of time (40% of total time) diagnosing and
                                                                    Søren Grimstrup for their statistical assistance.
reviewing the first 100 training cases as they did on the re-
maining 500 cases. This indicates that they shifted from a
                                                                    References
slow but actively engaged learning strategy towards a faster
trial-and-error strategy that yielded a significantly worse          1. Lomas A, Leonardi-Bee J, Bath-Hextall F. A systematic re-
                                                                        view of worldwide incidence of nonmelanoma skin cancer. Br
outcome [22,23]. Educational engagement is strongly asso-
                                                                        J Dermatol. 2012;166(5):1069-1080. DOI: 10.1111/j.1365-
ciated with internal and external motivation [24]. The aca-
                                                                        2133.2012.10830.x. PMID: 22251204.
demic curiosity, which was the sole motivation of the trial          2. Whiteman DC, Green AC, Olsen CM. The Growing Burden of
participants, may have faltered following 100 training cases,           Invasive Melanoma: Projections of Incidence Rates and Num-
resulting in the learning curve deceleration. Adopting new              bers of New Cases in Six Susceptible Populations through 2031.
clinical skills and knowledge becomes easier if the trainee             J Invest Dermatol. 2016;136(6):1161-1171. DOI: 10.1016/j.
                                                                        jid.2016.01.035. PMID: 26902923.
has former clinical experience within the given domain, pos-
                                                                     3. Petrie T, Samatham R, Witkowski AM, Esteva A, Leachman SA.
sibly because it is easier to conceptualize the clinical rele-
                                                                        Melanoma Early Detection: Big Data, Bigger Picture. J Invest
vance of the skills being taught [22,25]. Participants in this          Dermatol. 2019;139(1):25-30. DOI: 10.1016/j.jid.2018.06.187.
trial had no former clinical experience, potentially impeding           PMID: 30482597. PMCID: PMC6685706.
their ability to contextualize and acquire complex skills in         4. Ternov NK, Vestergaard T, Hölmich LR, et al. Reliable test of
visual diagnostics. Finally, the deceleration may have been             clinicians’ mastery in skin cancer diagnostics. Arch Dermatol
                                                                        Res. 2021;313(4):235-243. DOI: 10.1007/s00403-020-02097-8.
caused by a plateau in the participants ability to analyze and
                                                                        PMID: 32596742.
address their knowledge gaps, ie failure of metacognition
                                                                     5. Tschandl P, Codella N, Akay BN, et al. Comparison of the accu-
[26]. Additional studies, such as think-aloud verbal protocol           racy of human readers versus machine-learning algorithms for
studies, are needed to further our understanding of the low-            pigmented skin lesion classification: an open, web-based, inter-
versus high-performers learning strategies [27,28]. During              national, diagnostic study. Lancet Oncol. 2019;20(7):938-947.
think-aloud verbal protocol studies, participants are asked             DOI: 10.1016/S1470-2045(19)30333-X. PMID: 31201137.
                                                                        PMCID: PMC8237239.
to perform a task and “think aloud” while being recorded.
                                                                     6. Friedman RJ, Rigel DS, Kopf AW. Early detection of ma-
The thought processes are later deconstructed by a trained
                                                                        lignant melanoma: the role of physician examination and
observer and converted into standardized code blocks prior              self-examination of the skin. CA Cancer J Clin. 1985;35(3):130-
to statistical analyses.                                                151. DOI: 10.3322/canjclin.35.3.130. PMID: 3921200.
    Our study has several limitations. First, we did not collect     7. HHarkemanne E, Baeck M, Tromme I. Training general prac-
data on which subsections of the learning modules partici-              titioners in melanoma diagnosis: a scoping review of the
                                                                        literature. BMJ Open. 2021;11(3):e043926. DOI: 10.1136/bm-
pants had read during the trial, limiting our ability to perform
                                                                        jopen-2020-043926. PMID: 33757946. PMCID: PMC7993310.
sub-analyses on the effect of the histopathological explana-
                                                                     8. Dinnes J, Deeks JJ, Chuchu N, et al. Dermoscopy, with and
tions. Secondly, the retention test is a proxy rather than an ac-       without visual inspection, for diagnosing melanoma in adults.
curate measure of clinical skills. Finally, it is unclear whether       Cochrane Database Syst Rev. 2018;12(12):CD011902. DOI:
our results can be reproduced among clinicians. Thirdly, do-            10.1002/14651858.CD011902.pub2. PMID: 30521682. PM-
main experience generally increases a student ability to con-           CID: PMC6517096.
                                                                     9. Speelman C, Martin K, Flower S, Simpson T. Skill acquisition in
textualize knowledge, and we theorize that our results would
                                                                        skin cancer detection. Percept Mot Skills. 2010;110(1):277-297.
have been more pronounced among clinicians compared to
                                                                        DOI: 10.2466/PMS.110.1.277-297. PMID: 20391892.
medical students. Finally, several factors such as the partici-     10. Lacy FA, Coman GC, Holliday AC, Kolodney MS. Assessment
pants general interest, motivation, and social or cultural sta-         of Smartphone Application for Teaching Intuitive Visual Diagno-
tus may have introduced biases in our results.                          sis of Melanoma. JAMA Dermatol. 2018;154(6):730-731. DOI:
    This study describes an efficient and scalable approach             10.1001/jamadermatol.2018.1525. PMID: 29799954. PMCID:
                                                                        PMC6145647.
towards teaching pattern recognition in skin cancer diagnos-
                                                                    11. Monteiro SM, Norman G. Diagnostic reasoning: where we’ve been,
tics. We investigated the learning curves of medical students
                                                                        where we’re going. Teach Learn Med. 2013;25 Suppl 1:S26-S32.
and found a deceleration following approximately 100 train-             DOI: 10.1080/10401334.2013.842911. PMID: 24246103.
ing cases and a continuous and stable superiority among the         12. Norman GR, Rosenthal D, Brooks LR, Allen SW, Muzzin LJ.
upper compared to lower tertile. Access to histopathological            The development of expertise in dermatology. Arch Dermatol.
                                                                        1989;125(8):1063-1068. PMID: 2757402.




8                                                                        Original Article | Dermatol Pract Concept. 2023;13(2):e2023105
13. Norman GR, Coblentz CL, Brooks LR, Babcook CJ. Expertise in                 deliberate practice. Acad Med. 2015;90(11):1471-1486. DOI:
    visual diagnosis: a review of the literature. Acad Med. 1992;67(10          10.1097/ACM.0000000000000939. PMID: 26375267.
    Suppl):S78-S83. DOI: 10.1097/00001888-199210000-00045.                21.   Loerch AG. Learning Curves. In: Gass, S.I., Fu, M.C. (eds) Encyclo-
    PMID: 1388563.                                                              pedia of Operations Research and Management Science. Springer,
14. Yoon JS, Boutis K, Pecaric MR, Fefferman NR, Ericsson KA,                   Boston, MA. 2013 DOI: 10.1007/978-1-4419-1153-7_526
    Pusic MV. A think-aloud study to inform the design of radio-          22.   Schwartz D, Bransford J, Sears D. Efficiency and innovation in
    graph interpretation practice. Adv Health Sci Educ Theory Pract.            transfer. In J. Mestre (Ed.), Transfer of learning from a modern
    2020;25(4):877-903.       DOI:       10.1007/s10459-020-09963-0.            multidisciplinary perspective. Greenwich, CT: Information Age
    PMID: 32140874. PMCID: PMC7471179.                                          Publishing. 2005:1-51.
15. Baghdady MT, Pharoah MJ, Regehr G, Lam EW, Woods NN.                  23.   Davis JM, Leelawong K, Belynne K, Ret al. Intelligent User
    The role of basic sciences in diagnostic oral radiology. J Dent             Interface Design for Teachable Agent Systems. International
    Educ. 2009;73(10):1187-1193. PMID: 19805783.                                Conference on Intelligent User Interfaces. Miami, Florida, The
16. Baghdady MT, Carnahan H, Lam EW, Woods NN. Integration                      Association for Computing Machinery. 2003:26-33.
    of basic sciences and clinical sciences in oral radiology education   24.   Cook DA, Artino AR Jr. Motivation to learn: an overview of
    for dental students. J Dent Educ. 2013;77(6):757-763. PMID:                 contemporary theories. Med Educ. 2016;50(10):997-1014.
    23740912.                                                                   DOI: 10.1111/medu.13074. PMID: 27628718. PMCID:
17. Harvey VM, Patel H, Sandhu S, Wallington SF, Hinds G. Social                PMC5113774.
    determinants of racial and ethnic disparities in cutaneous mel-       25.   Andreasen LA, Tabor A, Nørgaard LN, Ret al. Multicenter
    anoma outcomes. Cancer Control. 2014;21(4):343-349. DOI:                    randomized trial exploring effects of simulation-based ul-
    10.1177/107327481402100411. PMID: 25310216. PMCID:                          trasound training on obstetricians’ diagnostic accuracy:
    PMC4505912.                                                                 value for experienced operators. Ultrasound Obstet Gyne-
18. Ibfelt EH, Steding-Jessen M, Dalton SO, Lundstrøm SL, Osler                 col. 2020;55(4):523-529. DOI: 10.1002/uog.20362. PMID:
    M, Hölmich LR. Influence of socioeconomic factors and re-                   31152560.
    gion of residence on cancer stage of malignant melanoma: a            26. Flavell JH (1979). Metacognition and cognitive monitoring: A new
    Danish nationwide population-based study. Clin Epidemiol.                 area of cognitive–developmental inquiry. American Psychologist.
    2018;10:799-807. DOI: 10.2147/CLEP.S160357. PMID:                         1979;34(10): 906–911. DOI: 10.1037/0003-066X.34.10.906
    30022857. PMCID: PMC6044336.                                          27. Ericsson KA, Simon HA. Protocol Analysis: Verbal Reports as
19. Tschandl P, Rinner C, Apalla Z, et al. Human-computer collab-             Data. 1993. DOI:10.7551/mitpress/5657.001.0001.
    oration for skin cancer recognition. Nat Med. 2020;26(8):1229-        28. Fox MC, Ericsson KA, Best R. Do procedures for verbal re-
    1234. DOI: 10.1038/s41591-020-0942-0. PMID: 32572267.                     porting of thinking have to be reactive? A meta-analysis and
20. Ericsson KA. Acquisition and maintenance of medical exper-                recommendations for best reporting methods. Psychol Bull.
    tise: a perspective from the expert-performance approach with




Original Article | Dermatol Pract Concept. 2023;13(2):e2023105                                                                                  9
     2011;137(2):316-344.   DOI:   10.1037/a0021663.   PMID:
     21090887.




10                                                             Original Article | Dermatol Pract Concept. 2023;13(2):e2023105
