Quality Standards in Medical Education Research
10.14601/TUTOR-13683
Questo articolo riassume la lettura tenuta il 14 novembre 2012 durante il XVI Congresso Nazionale SIPeM a Garda, circa gli standard di qualità per la ricerca in Pedagogia Medica. La ricerca che riguarda i metodi di insegnamento e valutazione deve essere indirizzata a produrre prove di efficacia (evidence) e può essere schematicamente suddivisa in tre fasi: comprensione della metodologia di ricerca, raccolta dei dati, valutazione dei risultati. La ricerca in Pedagogia Medica è importante, perché è indirizzata a produrre un miglioramento continuo della formazione nelle professioni sanitarie e perché mira anche a influenzare le politiche educative locali ed internazionali. I prodotti di ricerca sono riconosciuti in molti Paesi come mezzo di valutazione per la carriera accademica. Si possono distinguere 4 approcci principali: studi esploratori mirati a produrre modelli, studi sperimentali giustificativi, studi osservazionali predittivi, studi translazionali mirati all’implementazione di teorie e modelli. Ogni tipo di studio si può giovare di metodi qualitativi o quantitativi, per la cui selezione è essenziale che sia correttamente posta la “domanda di ricerca”, destinata ad esplicitare gli scopi e le finalità. Le migliori domande di ricerca nascono dal riferimento a contesti teorici e concettuali. L’articolo infine indica una struttura standard per la stesura di un buon progetto di ricerca, i mezzi migliori per la diffusione dei risultati e suggerisce risorse utili, come le Guide AMEE e la web community MedEdWorld.
application/pdf
1971-8551
4874
2
it
John Dent
Tutor
research studies
Text.Serial.Journal
https://ojs.unito.it/index.php/tutor
13
2013-12-04
2019-07-30
2013-12-04
7-12
1
John Dent
2013/12/04
10.14601/TUTOR-13683
7
1971-8551
2
research studies
12
https://ojs.unito.it/index.php/tutor/article/download/4874/4457
13
tutor                                                                                                      Vol. 13, N. 2, 2013: 7-12



        Guest Lecture. XVI Congresso SIPeM, Garda, 14 November 2012



           Quality Standards in Medical
           Education Research
                   Dr John Dent, MMEd,MD, FHEA, FRCS(Ed)
                   International Relations Officer for AMEE, the International Association for Medical Education. Tay Park House, 484
                   Perth Road, Dundee, DD2 1LR Scotland, UK. Tel: +44(0)1382 381953. Email: j.a.dent@dundee.ac.uk




        SUMMARY
        This paper highlights the importance of using the best evidence available when choosing teaching or assess-
        ment methods. It describes three stages to help us get started with research in medical education:
        • understanding methodology,
        • data collection,
        • evaluating results.
        Approaches to presentation or publication are discussed and a note of helpful resources provided.

        Keywords: Medical education, research studies, data collection

        RIASSUNTO
        Questo articolo riassume la lettura tenuta il 14 novembre 2012 durante il XVI Congresso Nazionale SIPeM
        a Garda, circa gli standard di qualità per la ricerca in Pedagogia Medica. La ricerca che riguarda i metodi di
        insegnamento e valutazione deve essere indirizzata a produrre prove di efficacia (evidence) e può essere
        schematicamente suddivisa in tre fasi: comprensione della metodologia di ricerca, raccolta dei dati, valuta-
        zione dei risultati.
        La ricerca in Pedagogia Medica è importante, perché è indirizzata a produrre un miglioramento continuo
        della formazione nelle professioni sanitarie e perché mira anche a influenzare le politiche educative locali
        ed internazionali. I prodotti di ricerca sono riconosciuti in molti Paesi come mezzo di valutazione per la
        carriera accademica.
        Si possono distinguere 4 approcci principali: studi esploratori mirati a produrre modelli, studi sperimentali
        giustificativi, studi osservazionali predittivi, studi translazionali mirati all’implementazione di teorie e mo-
        delli. Ogni tipo di studio si può giovare di metodi qualitativi o quantitativi, per la cui selezione è essenziale
        che sia correttamente posta la “domanda di ricerca”, destinata ad esplicitare gli scopi e le finalità. Le migliori
        domande di ricerca nascono dal riferimento a contesti teorici e concettuali.
        L’articolo infine indica una struttura standard per la stesura di un buon progetto di ricerca, i mezzi mi-
        gliori per la diffusione dei risultati e suggerisce risorse utili, come le Guide AMEE e la web community
        MedEdWorld.

        Parole chiave: Formazione medica, progetti di ricerca, raccolta dati




                                                                                                                                7
           © Firenze University Press     ISSN (print) 1973-638X
           http://www.fupress.com/tutor   ISSN (online) 2037-1861
  tutor Volume 13, Numero 2, settembre 2013

                   I am honoured to have been invited to address the         For example, in BEME Guide 11, “The education-
Lavori originali




                   XVIth Congress of SIPeM in Garda and to contrib-          al effects of portfolios on undergraduate student
                   ute this paper to the Journal. Other conference           learning: a Best Evidence Medical Education (BEME)
                   sessions provided detailed discussion on particular       systematic review” [4] Buckley and colleagues seek
                   issues on the topic so I hope this paper will en-         to produce a collated summary of the evidence for
                   courage junior colleagues now to make a start in          the educational effects of the use of portfolios in
                   medical education research.                               undergraduate education.
                   To do this I thought we might begin by asking our-        The increased interest in medical education and
                   selves five questions:                                    the move to evidence-informed practice in the last
                   1. Why is research in medical education important?        two decades have been associated with a greater
                   2. How can I plan a research study?                       recognition of scholarship in teaching [5]. The num-
                   3. How can I present or publish my work?                  ber of papers published on the subject of medical
                   4. What resources are available to help me?               education has increased at a higher rate than in any
                   5. What should I do next?                                 other field in medicine [6]. Being a good user of
                                                                             educational research demonstrates our scholar-
                                                                             ship of teaching. This is one of the five domains of
                   1. WHY IS RESEARCH IN MEDICAL                             a medical educator as proposed by the Academy
                   EDUCATION IMPORTANT?                                      of Medical Educators in the UK [7]. Their Domain
                   Research in medical education is important because        4, “Educational research and evidence-based prac-
                   it provides an evidence-base for medical education        tice” describes the requirement of a medical educa-
                   essential for good teaching and because it can mea-       tor to “critically evaluate the educational literature
                   sure the effectiveness of one method of teaching          and apply this learning to his or her educational
                   or assessment compared to another. Todres and             practice”.
                   colleagues [1] observed that the practice of medical      So, if we can appreciate the usefulness of medical
                   education appears to be “founded on the basis of          education research then the next question is how
                   nothing more than pragmatism, fashion and whim”.          can we contribute to that research-based knowl-
                   In saying this they highlight a professional discrepan-   edge ourselves. “Research in medical education
                   cy because which of us would prescribe a drug or a        seeks to inform continued improvement of medi-
                   course of treatment based on a hunch, a prejudice,        cal education locally and to influence medical ed-
                   an opinion or a guess? And yet, when it comes to          ucation policy nationally and internationally” [8]. It
                   teaching, assessment or deciding on a programme           may contribute to scholarly debate and promote
                   or curriculum our choices may often be directed by        continuing professional development. Medical edu-
                   these thoughts rather than by considering the best        cation scholarship is therefore grounded in the real
                   evidence available. Best evidence medical education       world of medical education practice and in turn
                   (BEME) “relies on the judicious examination and ap-       medical education practice informs work under the
                   plication of evidence in the published literature” [2].   heading of scholarship. There is a practical value to
                   The BEME collaboration [3] is committed to:               being involved in it.
                   “the dissemination of information which allows
                   teachers and stakeholders in the medical and health
                   professions to make decisions on the basis of the         2. HOW CAN I PLAN A RESEARCH
                   best evidence available;                                  STUDY?
                   the production of systematic reviews which pres-          Four main categories of research approaches are
                   ent the best available evidence and meet the needs        described by Ringsted and colleagues [9]:
                   of the user;                                              • explorative studies aiming at modelling;
                   the creation of a culture of best evidence educa-         • experimental studies aiming at justifying;
                   tion amongst individuals, institutions and national       • observational studies aiming at predicting;
                   bodies.                                                   • translational studies aiming at implementing.



                     8
                                                                                Research in Medical Education



So how can we start the process? Two publications        perience), grounded theory which builds theories
may help us at this point. Firstly Creswell [10] gives   of social phenomena based on experiences, action
a highly acclaimed overview of approaches to re-         research which looks at planning, implementing and
search. Secondly, Ajjawi and colleagues in “Getting      reflecting on a change, and hermeneutics which
started… medical education research” [11] lead us        takes the lived experience of an individual as a
through three steps in planning, executing and re-       means of understanding their political, historical or
viewing our research activity. We can follow this        sociocultural contexts [12].
three stage approach.
                                                         Any research project begins with the construction
                                                         of a good research question. This should make the
Stage 1:                                                 purpose of our study clear and direct how we will
We must begin by thinking of what to do before           accomplish the project. At an early stage we need
we actually start our research project. There are        to consider what theoretical or conceptual frame-
two main methods of research, quantitative and           work we will use to strengthen our conclusions.
qualitative, as well as those with elements of both,     If we can base our work on an established theory
“mixed method approaches” [10]. What does this           then our arguments will be strengthened and more
mean? Put simply it can be said that quantitative        readily generalised. “Theory can help structure and
research stresses objectivity and is concerned with      facilitate all aspects of the research process; it can
absolute truth and the analysis of quantities and        help us generalise our study results beyond their
numbers. Qualitative research on the other hand          immediate settings” [13]. We might then seek out
stresses subjectivity, is concerned with socially con-   colleagues to collaborate with. Collaborative re-
structed truths and analyses qualities and words.        search can contribute to the rigour of the study and
Quantitative approaches are used to test a hy-           facilitate generalizability of the results [14]. Next
pothesis, to answer questions about ‘how much’ or        we must consider the ethical issues of the study and
‘how many’. Qualitative approaches seek answers          seek ethical committee approval for the project.
to the ‘what’, ‘how’ or ‘why’ of a phenomenon” [8].      Now we are ready to write our research proposal
Medical education research may appear to be more         together; this may be a lengthy, detailed document.
focused on these qualitatitve approaches than on         “A research proposal provides a detailed descrip-
the quantitative approaches we are used to in            tion of the proposed research project. It acts as a
our research in the clinical or basic sciences.          plan of action and a contract between the research-
Quantitative research in medical education may           er and the superior, the funding source and the eth-
include correlational research; for example we           ics committee” [15]. It should include: [10]:
might be looking for a relationship between stu-         • the title, which should be informative and
dents’ exam ranking in a knowledge-based assess-             attention-grabbing
ment (MCQ) and in a performance-based assess-            • the introduction, stating the problem, the con-
ment (OSCE) or we may be seeking to compare the              text and the purpose of the research
exam performance of two groups of students who           • a literature review indicating what is already
are exposed to a new topic in two different ways,            known on the topic
for instance one group by lectures and the other         • the specific research question, indicating the sig-
group by directed self-learning (experimental re-            nificance of the work and showing what contri-
search). Other examples of quantitative research             bution it will make to existing knowledge
may include causal comparative research and sim-         • a description of the research method to be
ple or stratified random sampling.                           used – either qualitative or quantitative
Qualitative research methods on the other hand               or a mixed method. It should include the study
may include ethnography (writing about people                design, the methodology, the sampling strategy
and studying people), phenomenology (looking at              and the method of data collection such as focus
the perspectives of different people on a new ex-            groups or questionnaire.



                                                                                                          9
  tutor Volume 13, Numero 2, settembre 2013

                   • something about how quality has to be assured              ence (consistency between the purpose, the meth-
Lavori originali




                        and ethical issues answered                             odology and the methods used), its credibility and
                   • details of the budget required and other logisti-          by its evaluative and procedural rigour. It is import-
                        cal matters                                             ant that the opinions of the researcher do not bias
                   • a list of relevant references                              the wording of the questionnaire or the answers
                   • an appendix which might contain the question-              an interviewee gives (reflexivity). Quantitative
                        naires used, timelines and costings.                    research is judged by the size of the study group
                   It is usually best to write the research proposal ear-       and the rigour of the sampling and the statistics as
                   ly as it helps to develop and organise your thoughts.        well as by the measurement instrument used and
                                                                                the reliability and validity of the work.
                                                                                Finally you should ask yourself about the transfer-
                   Stage 2:                                                     ability / generalizability of your work; how useful
                   Now comes the hard work!                                     will the research findings be to anyone else in a sim-
                   In qualitative research data collection may include          ilar situation?
                   observation, structured or semi-structured inter-
                   views, focus groups or document analysis. For ex-
                   ample a new undergraduate teaching programme in              3. HOW CAN I PRESENT OR PUBLISH
                   the outpatient department was introduced based               MY WORK?
                   on the participation of invited patient volunteers           Decide first whether your work is to be presented
                   and tutors from community practice. To evaluate              or published. Is it to be at a local, national or in-
                   the impact of this innovation on all involved, the           ternational level? Departmental meetings are good
                   opinion of all participant groups was sampled by             venues for your first oral presentation and are less
                   questionnaire and semi-structured interviews.                threatening occasions than presentations else-
                   In quantitative research student rankings in differ-         where. At a larger conference you may be able to
                   ent class examinations might be studied to compare           submit your work for presentation as a short com-
                   their performance in a traditional exam compared to          munication, a poster or as an e-poster. Poster pre-
                   the one newly introduced. If further information was         sentations may also include a short time to present
                   sought then qualitative research approaches may              the key findings orally. Wherever you are going to
                   be added such as questionnaires, structured-inter-           present your work it is important to practice the
                   views or focus groups to collect as much additional          presentation repeatedly. Well-rehearsed, clear de-
                   information as possible (mixed research model) [10].         scriptions and simple tables or illustrations are re-
                   Once you have collected your raw data it is time to          quired, but it is still important to practice the pre-
                   analyse it. Statistical software is available to help with   sentation and have constructive feedback on your
                   the analysis of numerical data from quantitative re-         performance from a colleague. Remember there
                   search. A useful study by Dembe and colleagues [16]          will be a strict time limit for you to speak and that it
                   aims to identify the statistical software applications       is important to leave the time requested at the end
                   most commonly employed for data analysis in health           for questions and discussion. You might rehearse
                   services research studies in the U.S. Results from           the answers to possible questions in advance.
                   qualitative research may be analysed by grouping             Publications may be in a peer-reviewed journal of
                   them into themes or by indexing or charting to con-          medical education, in an e-journal section of an es-
                   struct a framework or map by which they can be               tablished journal or in an entirely web-based jour-
                   linked to similar studies identified in the literature.      nal. Some journals favour papers on theoretical
                                                                                issues while others favour papers with more prac-
                                                                                tical implications so you should carefully consider
                   Stage 3:                                                     where to submit your paper. The impact factor
                   Now it is time to evaluate your research activity.           of a journal may also influence your choice. Take
                   Qualitative research is judged by internal coher-            an objective look at your work, read your target



                     10
                                                                                        Research in Medical Education



journal and read the journal’s advice to author. It         5. WHAT SHOULD I DO NOW?
has to be said though that the majority of medical          “Use your ideas and hunches from the real world
publications are in English so, if this is not your first   of medical education practice (in which you work)
language, find someone to help [17]. When a paper           to inform the work you undertake under the head-
is submitted for peer review it will be sent to three       ing of scholarship” [8]. Choose something that you
or more reviewers who may ask themselves the                would like to know the answer to, check the liter-
basic screening question, “Who cares?” They will            ature, find research colleagues to discuss your idea
decide whether your manuscript should proceed               with and see what other resources are available to
along the editorial process further. Features they          support you. You can now begin to take the first
will be looking out for include: originality, timeli-       steps in exploring your research ideas and contrib-
ness, grounding in current knowledge, relevance             uting your findings for on-going discussion in the
to the journal’s readership, the use of appropriate         medical education literature.
methods and procedures, the standard of presenta-
tion of the paper and adherence to appropriate sci-
entific and professional standards [5]. Journals on         REFERENCES
average are only able to accept <10% of the manu-           [1]    Todres M, Stephenson A, Jones R. Medical education
scripts they receive. As a response to this the new                research remains the poor relation. BMJ, 2007, 335:
MedEdWorld Publish website [6] offers an oppor-                    333-335.
tunity for unpublished, unreviewed material to be           [2]    Mennin SP, McGrew MC. Scholarship in teaching and
presented to a wider audience online where a range                 the best evidence medical education: synergy for tea-
of comments and discussion can then be expected.                   ching and learning. Medical Teacher, 2000, 22: 468-471.
                                                            [3]    www.amee.org. Accessed 23/2/13
                                                            [4]    Buckley S, Coleman J, Davison I, Khan K, Zamora J,
                                                                   Malick S, Morely D, Pollard D, Ashcroft T, Popovic C,
4. WHAT RESOURCES ARE AVAILABLE                                    Sayers J. “The educational effects of portfolios on un-
TO HELP ME?                                                        dergraduate student learning: a Best Evidence Medical
The AMEE organisation [3] can support research-                    Education (BEME) systematic review” BEME Guide 11.
ers in medical education in many ways. The annu-                   Medical Teacher, 2009, 31: 282-298.
al conference spotlights international speakers on          [5]    McGaghie WC Scholarship, publication and career
educational research and facilitates networking or                 advancement in health professions education. AMEE
collaboration with like-minded colleagues. Posters                 Guide No 43.MedicalTeacher, 2009, 31: 574-590.
at the conferences provide a good opportunity for           [6]    MedEdWorldPublish. www.mededworld.org Acces-
presentation and discussion with a group of peo-                   sed 13/2/13
ple interested in the same topic. The AMEE re-              [7]    Academy of Medical Educators. Professional Standards
search committee offers research grants and can                    2012, www.medicaleducators.org. Accessed 13/2/13
                                                            [8]    Cleland J, Roberts T. Medical education research. In:
support researchers by review and feedback on
                                                                   JA Dent, RM Harden (editors). A Practical Guide for Me-
submitted material. The biennial Ottawa confer-
                                                                   dical Teachers,4ed. London: Elsevier, 2013: 399-404.
ence focusses on assessment and forms a specialist          [9]    Ringsted c, Hodges B, Scherpbier A. ‘The research com-
network opportunity [3]. The AMEE Guides cov-                      pass’; an introduction to research in medical education;
er research-related topics; for example in AMEE                    AMEE Guide No. 56. Medical Teacher, 2011, 33: 695-709.
Guide No. 60: Building bridges between theory and           [10]   Creswell JW. Research design: qualitative, quantitative
practice in medical education using a design-based                 and mixed methods approach. Los Angeles, SAGE Publi-
research approach Dolmans and Tigelaar [18]                        cations, 2009.
discuss the role of design-based research.                  [11]   Aijawi R, McAleer, S, Smith J, Rees C. “Getting started
The new website MedEdWorld [6] offers special-in-                  …” with medical education research. In: JA Dent, C
terest groups with online discussion on different                  McRae (editors). “Getting started …” a practical guide
topics and again provides a support network for                    for clinical teachers,4ed. University of Dundee, Centre
                                                                   for Medical Education, 2013: in press.
researchers.



                                                                                                                     11
  tutor Volume 13, Numero 2, settembre 2013
                   [12] Lingard, L. and Kennedy, T. J. Qualitative Research            Research, 2011, 11: 252-257.
Lavori originali




                        Methods in Medical Education. In: T Swanwick (edi-        [17] Bligh J, Brown J. Publishing. In: JA Dent, RM Harden
                        tor). Understanding Medical Education: Evidence, Theory        (editors), A Practical Guide for Medical Teachers, 4ed.
                        and Practice, Oxford, Wiley-Blackwell, UK, 2010.               Edinburgh, Elsevier, 2013: 384-390.
                   [13] Rees CE & Monrouxe LV. Theory in medical educa-           [18] Dolmans DHJM, Tigelaar D. 2012. Building bridges
                        tion research: how do we get there? Medical Education,         between theory and practice in medical education
                        2010, 44: 334-339.                                             using a design-based research approach: AMEE Guide
                   [14] Huggart KN, Gusic ME, Greenberg R, Ketterer JM.                No. 60. Medical Teacher, 2012, 34: 1-10.
                        Twelve tips for conducting collaborative research in
                        medical education. Medical Teacher, 2011, 33:713-718.
                   [15] Illing J. Thinking about research; frameworks, ethics     ACKNOWLEDGEMENTS
                        and scholarship. In: T Swanwick (editor). Understanding   I am grateful to these colleagues in the University
                        Medical Education. Oxford, ASME and Wiley-Blackwell
                                                                                  of Dundee, Rola Ajjawi, Sean McAleer, Julie Smith
                        UK, 2010:283-300.
                                                                                  and Charlotte Rees (cited above) for their helpful
                   [16] Dembe AE, Partridge JS, Geist LC. Statistical software
                        applications used in health services research: analysis
                                                                                  summaries of established texts on the topic of this
                        of published studies in the U.S. BMC Health Services      paper.




                     12
