10.14639/0392-100X-N2212
Objective. To evaluate the reliability and validity of the Italian version of the Brief Questionnaire of Olfactory Disorders (Brief-IT-QOD). Methods. The study consisted of six phases: item generation, reliability analysis (112 dysosmic patients for internal consistency analysis and 61 for test-retest reliability analysis), normative data generation (303 normosmic subjects), validity analysis (comparison of Brief-IT-QOD scores of healthy and dysosmic subjects and scores correlation with psychophysical olfactory testing TDI and SNOT-22 scores), responsiveness analysis (10 dysosmic chronic rhinosinusitis with nasal polyps patients before and after biologic therapy), and cut-off value determination (ROC curve analysis of Brief-IT-QOD sensitivity and specificity). Results. All subjects completed the Brief-IT-QOD. Internal consistency (α&nbsp;&gt;&nbsp;0.70) and test-retest reliability (ICC&nbsp;&gt;&nbsp;0.7) were acceptable and satisfactory for both questionnaire subscales. A significant difference between dysosmic and control subjects was found in both subscales (p&nbsp;&lt;&nbsp;0.05). Significant correlations between subscales scores and TDI and SNOT-22 scores were observed. Brief-IT-QOD scores before treatment were significantly higher than after biological therapy. Conclusions. Brief-IT-QOD is reliable, valid, responsive to changes in QoL, and recommended for clinical practice and outcome research.
text/xml
1827-675X
2212
en
Francesco Mozzanica
http://creativecommons.org/licenses/by-nc-nd/4.0
ACTA Otorhinolaryngologica Italica
quality of life
Italian version of the Brief Questionnaire of Olfactory Disorders (Brief-IT-QOD)
Text.Serial.Journal
https://www.actaitalica.it/
43
Rhinology
2023-05-23
2022-07-11
2023-07-25
2023-07-25
252-261
1
Francesco Mozzanica
2023/05/23
10.14639/0392-100X-N2212
252
Department of Otorhinolaryngology, IRCCS Multimedica, Ospedale San Giuseppe, Milan, Italy; Department of Clinical Sciences and Community Health, University of Milan, Milan, Italy
1827-675X
quality of life
en
261
Italian version of the Brief Questionnaire of Olfactory Disorders (Brief-IT-QOD)
https://www.actaitalica.it/article/download/2212/1004
43
ACTA OTORHINOLARYNGOLOGICA ITALICA 2023;43:252-261; doi: 10.14639/0392-100X-N2212




Rhinology


Italian version of the Brief Questionnaire of Olfactory
Disorders (Brief-IT-QOD)
Versione italiana del brief Questionnaire of Olfactory Disorders (Brief-IT-QOD)
Arianna Cardella1,2, Giuseppe Riva3, Andrea Preti1, Andrea Albera3, Livio Luzi2,4, Roberto Albera3, Davide Cadei1,
Gian Marco Motatto3, Filippo Omenetti1, Giancarlo Pecorari3, Francesco Ottaviani1,5, Francesco Mozzanica1,5
1
 Department of Otorhinolaryngology, IRCCS Multimedica, Ospedale San Giuseppe, Milan, Italy; 2 Department of Biomedical
Sciences for Health, University of Milan, Milan, Italy; 3 Division of Otorhinolaryngology, Department of Surgical Sciences, University of
Turin, Turin, Italy; 4 Department of Endocrinology, Nutrition and Metabolic Diseases, IRCCS Multimedica, Milan, Italy; 5 Department of
Clinical Sciences and Community Health, University of Milan, Milan, Italy


SUMMARY
Objective. To evaluate the reliability and validity of the Italian version of the Brief Ques-
tionnaire of Olfactory Disorders (Brief-IT-QOD).
Methods. The study consisted of six phases: item generation, reliability analysis (112 dys-
osmic patients for internal consistency analysis and 61 for test-retest reliability analysis),
normative data generation (303 normosmic subjects), validity analysis (comparison of
Brief-IT-QOD scores of healthy and dysosmic subjects and scores correlation with psycho-
physical olfactory testing TDI and SNOT-22 scores), responsiveness analysis (10 dysosmic
chronic rhinosinusitis with nasal polyps patients before and after biologic therapy), and cut-
off value determination (ROC curve analysis of Brief-IT-QOD sensitivity and specificity).
Results. All subjects completed the Brief-IT-QOD. Internal consistency (α > 0.70) and
test-retest reliability (ICC > 0.7) were acceptable and satisfactory for both questionnaire
subscales. A significant difference between dysosmic and control subjects was found in
both subscales (p < 0.05). Significant correlations between subscales scores and TDI and
SNOT-22 scores were observed. Brief-IT-QOD scores before treatment were significantly
higher than after biological therapy.
Conclusions. Brief-IT-QOD is reliable, valid, responsive to changes in QoL, and recom-
mended for clinical practice and outcome research.                                                  Received: July 14, 2022
KEY WORDS: olfaction disorders, questionnaire, quality of life                                      Accepted: November 5, 2022

                                                                                                    Correspondence
RIASSUNTO                                                                                           Arianna Cardella
Obiettivo. Valutare affidabilità e validità della versione italiana del Questionario Breve dei      Department of Biomedical Sciences for Health,
Disordini Olfattivi (Brief-IT-QOD).                                                                 University of Milan, Milan, Italy
Metodi. Lo studio è stato composto da sei fasi: generazione del questionario, analisi di affi-      Tel. +39 02 85994471
dabilità (112 pazienti disosmici per la consistenza interna e 61 per l’affidabilità test-retest),   E-mail: arianna.cardella@unimi.it
generazione di dati normativi (303 soggetti normosmici), analisi di validità (comparazione
dei punteggi Brief-IT-QOD di pazienti e controlli e correlazione con valutazione olfattoria           How to cite this article: Cardella A, Riva G,
psicofisica TDI e punteggio SNOT-22), analisi di responsività (10 pazienti disosmici con              Preti A, et al. Italian version of the brief Ques-
rinosinusite cronica poliposica prima e dopo terapia biologica), e determinazione di valori           tionnaire of Olfactory Disorders (brief-IT-QOD).
soglia.                                                                                               Acta Otorhinolaryngol Ital 2023;43:252-261. htt-
Risultati. Tutti i soggetti hanno completato il Brief-IT-QOD. Consistenza interna (α > 0,70)          ps://doi.org/10.14639/0392-100X-N2212
e affidabilità test-retest (ICC > 0,7) erano soddisfacenti per entrambe le sottoscale del que-
stionario. In entrambe le componenti è stata rilevata una differenza significativa (p < 0,05)       © Società Italiana di Otorinolaringoiatria
fra individui disosmici e normosmici. Sono emerse correlazioni significative fra i punteggi         e Chirurgia Cervico-Facciale
delle sottoscale e i punteggi TDI e SNOT-22. I punteggi al Brief-IT-QOD erano significati-
vamente più alti prima della terapia con farmaco biologico.                                                        OPEN ACCESS
Conclusioni. Il Brief-IT-QOD è affidabile, valido, responsivo ai cambiamenti nella qualità          This is an open access article distributed in accordance with
di vita e raccomandato per pratica clinica e ricerca.                                               the CC-BY-NC-ND (Creative Commons Attribution-Non-
                                                                                                    Commercial-NoDerivatives 4.0 International) license. The
PAROLE CHIAVE: disturbi dell’olfatto, questionario, qualità di vita                                 article can be used by giving appropriate credit and mentio-
                                                                                                    ning the license, but only for non-commercial purposes and
                                                                                                    only in the original version. For further information: https://
                                                                                                    creativecommons.org/licenses/by-nc-nd/4.0/deed.en




252
                                                   Reliability and validity of the Italian version of the brief Questionnaire of Olfactory Disorders




Introduction                                                        subjective severity of OD. The availability of a valid and
                                                                    reliable instrument able to assess the QoL in patients with
Olfactory dysfunction (OD) is defined as the reduced or             OD is useful in clinical practice, but an Italian version of
distorted ability to smell during sniffing or eating. It can        this instrument is lacking.
be classified as either quantitative, involving alteration in       The aim of this study was to validate the Italian version
the strength but not in the quality of odour perception (hy-        of the brief-QOD (Brief-IT-QOD) and to use it to evaluate
posmia, anosmia), or qualitative, in which the quality of           the OD-related QoL in a group of Italian individuals. In
odour perception is changed (parosmia, phantosmia) 1. The           particular, the specific aims of the study were to: (1) cultur-
prevalence of OD in the general population is estimated to          ally adapt the brief-QOD into Italian, (2) evaluate the ques-
be 3%-5% for total smell loss (anosmia) and 15%-25% for             tionnaire’s internal consistency and reliability, (3) provide
partial impairment (hyposmia); this increases to about 60%          normative data of the Italian population, (4) evaluate its va-
in individuals older than 65 years 1. OD has profound ef-           lidity and responsiveness, (5) calculate the cut-off score of
fects on quality of life (QoL), physical and social function        this questionnaire.
and even mortality 2,3. In their review on olfactory disor-         The underling hypotheses are: (1) the brief-QOD can be
ders and QoL, Croy et al. 2 showed that loss of the sense of        culturally adapted into Italian; (2) the Italian version of the
smell causes disturbances in important areas of daily life.         questionnaire presents strong internal consistency and re-
The main issues regarded food intake (increased/decreased           liability; (3) the validity and responsiveness of the Italian
quantities, decreased enjoyment, decreased appetite, diffi-         version of the brief-QOD are strong.
culties in cooking), safety (eating spoiled food, failure to        The importance of this study lies in the fact that a validated
perceive fire, smoke, or gas), personal hygiene, social life,       brief-QOD for the Italian language would be useful in clin-
household chores and working life. Patient-reported wor-            ical practice, for example during the assessment of patients
ries about these different aspects and consequent daily-life        suffering from nasal diseases, allowing better knowledge
restrictions negatively affected QoL 2.                             of OD-related QoL. Furthermore, a validated Italian brief-
In order to provide a proper assessment of this frequent dis-       QOD could facilitate both the diagnostic work-up and the
order, it is important to measure and monitor the impact            decision-making process on treatment options.
of OD on patients’ QoL. Questionnaires are the preferred
means for evaluation of this aspect and are often used as
outcome measures. The Questionnaire of Olfactory Disor-             Materials and methods
ders (QOD) 4 was developed by Frasnelli and Hummel as               The study consisted of six different phases: back-transla-
a self-report inventory to assess subjective information on         tion and cross-cultural adaptation into Italian of the brief-
OD. The QOD consists of several items that can be divided           QOD (phase 1); internal consistency and reliability analy-
into three subscales (Parosmia, Quality of Life, Socially-          sis (phase 2); normative data generation (phase 3); validity
desired) and one 5-item visual analogue scale.                      analysis (phase 4); responsiveness analysis (phase 5); cut-
Because of its excellent psychometric properties, the QOD           off value (phase 6). The COnsensus-based Standards for
has been widely used in clinical settings and has been trans-       the selection of health Measurement INstruments (COS-
lated into several languages 5-8. However, the length of the        MIN) checklist was followed for the different phases 12.
questionnaire can be overwhelming for patients, thus limit-
ing its application as an evaluation tool both in the clinic        Participants
and in research. The literature supports the use of shorter         Different groups of patients and controls were recruited for
and simpler versions of questionnaires, which favour great-         each of the six different phases of the study (Tab. I). Inclu-
er response rates and higher quality of data 9.                     sion criteria were: normal cognitive function (Mini Mental
Attempts have been made to address this issue. In 2019,             State Examination score > 24 for subjects older than 65),
Mattos and colleagues 10 developed a brief version of the           preserved reading skills, age > 18 years, no history of neo-
“negative statements” QOD domain (QOD-NS). This                     plastic, airway, neurologic, rheumatologic, hematologic, or
is made up of only 7 items, but maintains consistency in            endocrinologic disorders.
measured patient-reported outcomes of olfactory-specific            Data for phases 2, 3, and 4 were obtained from different
QoL. Later, Zou et al. 11 created a brief version of the QOD        otorhinolaryngologic (ENT) centres in Italy to ensure ap-
(brief-QOD) which included Mattos’ 7 items concerning               plicability of the Brief-IT-QOD in different settings. All
QoL 10 (QOD-NS) plus 4 items regarding parosmia (QOD-               enrolled subjects underwent an objective nasal assessment
P) and 3 visual analogue scales (QOD-VAS). The brief-               which included a full head and neck examination and a na-
QOD showed suitable reliability and validity to assess the          sal endoscopy (with specific assessment of olfactory cleft



                                                                                                                                              253
A. Cardella et al.




Table I. Clinical and demographic characteristics of the samples.
 Phase of the study                          Type of study                        Sample clinical characteristics      Mean age            Sex
                                                                                                                     (range years)    M           F
 1       Item generation                           Item generation                   Patients with OD (n = 20)       53.3 (42-78)    10          10
 2       Internal consistency                    Internal consistency                Patients with OD (n = 112)      57.6 (18-87)    53          59
         Reliability analysis                    Test-retest reliability              Patients with OD (n = 61)      57.3 (20-82)    30          31

 3       Normative data                             Normative data                Asymptomatic subjects (n = 303)    54.9 (18-87)    141         162
         generation
 4       Validity analysis                           Clinical validity            Asymptomatic subjects (n = 303)    54.9 (18-87)    141         163
                                                                                     Patients with OD (n = 112)      57.6 (18-87)     53         59

                                       Concurrent validity (correlation between      Patients with OD (n = 112)      57.6 (18-87)    53          59
                                        Brief-IT-QOD and I-SNOT-22 scores)
     5   Responsiveness                 Comparison pre- and post-biological          Patients with OD (n = 10)       52.2 (41-67)     7           3
         analysis                                    therapy
Age is reported as mean (range). OD: olfactory dysfunction.



patency) using a 30° endoscope, 2.7 mm diameter. In ad-                           This back-translation was compared to the original text by
dition, orthonasal olfactory performance was evaluated                            the professional translator; no items of incongruent transla-
using the extended version of the Sniffing test (Burghart                         tion were noted as every item was semantically identical to
Messtechnik GmbH, Germany), a reliable, and validated                             the original English text. The professionals involved in the
psychophysical test able to evaluate olfactory threshold,                         cross-cultural adaptation also discussed the original ver-
discrimination, and identification (TDI) using pens filled                        sion, the final translation into Italian, and the back-trans-
with odourants 13. The TDI score, which ranges from 1 to                          lation. Finally, the readability of the Brief-IT-QOD was
48, was used to define functional anosmia (TDI ≤ 16), hy-                         checked by a dedicated company. The text was considered
posmia (16 < TDI < 31), or normosmia (TDI ≥ 31) 14.                               readable by a person with the reading competence of five
                                                                                  years of primary education.
Phase 1: back-translation and cross-cultural adaptation
into Italian of brief-QOD                                                         Phase 2: internal consistency and reliability analysis
A cross-cultural adaptation process of translation and back-                      The aim of this phase was to assess the reproducibility
translation was performed. Items of the original brief-QOD                        of Brief-IT-QOD. This was evaluated with two methods:
questionnaire were first translated into Italian by two bi-                       internal consistency and test-retest reliability. The first as-
lingual otorhinolaryngologists experienced in olfactory                           sesses the extent to which each item in a factor measures
disorders management (step 1: forward translation). Dis-                          the same underlying construct, while the latter is obtained
cussion of the translated text with two other otorhinolaryn-                      by administering the same test twice over a period of time
gologists with extensive experience in nasal diseases en-                         to a group of individuals.
sured the unanimity and the interpretation of the translated                      Clinical data were obtained from 112 patients (53 males
text (step 2: synthesis). Twenty patients, 10 males and 10                        and 59 females) evaluated for OD. The mean age of pa-
females, with a median age of 55.3 years (range 42-78),                           tients with OD was 57.6 ± 14.1 years (18-87 years). Eight-
reporting hyposmia were enrolled in a pilot study (step 3:                        een were affected by functional anosmia, while the remain-
pilot study). OD was related to chronic rhinosinusitis with                       ing 94 were hyposmic. The aetiology of OD was CRSwNP
nasal polyps (CRSwNP) in all cases. Each patient autono-                          (60 patients), idiopathic (23 patients), viral infection (20
mously filled in the first translation of the brief-QOD and                       patients), and head trauma (9 patients). Internal consisten-
discussed the wording and meaning of each item of the                             cy of Brief-IT-QOD was assessed using Cronbach’s alpha
questionnaire with the senior clinician. The wording of the                       coefficient. Values between 0.7 and 0.9 were taken to indi-
questionnaire was modified considering the suggestions                            cate acceptable internal consistency 15. For this analysis, the
given by patients (step 4: expert panel). This new and fi-                        Brief-IT-QOD scores obtained in the group of 112 patients
nal version of the Italian brief-QOD (called Brief-IT-QOD,                        were used.
Tab. II) was then translated back into English by a quali-                        Of the 112 patients involved in internal consistency analy-
fied professional translator (step 5: backward translation).                      sis, 61 patients (30 males and 31 females) were randomly



254
                                                                      Reliability and validity of the Italian version of the brief Questionnaire of Olfactory Disorders




Table II. Italian version of the Brief Questionnaire of Olfactory Disorders (Brief-IT-QOD).

Il questionario seguente indaga il ruolo dell’olfatto nella sua vita quotidiana. Per favore, risponda sinceramente alle domande, non ci sono risposte giuste o sba-
gliate.
  P1       A causa di problemi all’olfatto, gli alimenti hanno un sapore diverso da quello che dovrebbero avere.       D’accordo                               o
                                                                                                                       Abbastanza d’accordo                    o
                                                                                                                       Abbastanza in disaccordo                o
                                                                                                                       Totalmente in disaccordo                o
  P2       Sento costantemente un odore spiacevole nel naso, a prescindere dalla prossimità di una fonte               D’accordo                               o
           odorosa.                                                                                                    Abbastanza d’accordo                    o
                                                                                                                       Abbastanza in disaccordo                o
                                                                                                                       Totalmente in disaccordo                o
  P3       Odori che sono piacevoli per gli altri a me sembrano fastidiosi.                                            D’accordo                               o
                                                                                                                       Abbastanza d’accordo                    o
                                                                                                                       Abbastanza in disaccordo                o
                                                                                                                       Totalmente in disaccordo                o
  P5       Il problema più grave per me non è di sentire meno odori o non poter sentirli in assoluto, ma il fatto      D’accordo                               o
           che abbiano un profumo diverso da quello che dovrebbero avere.                                              Abbastanza d’accordo                    o
                                                                                                                       Abbastanza in disaccordo                o
                                                                                                                       Totalmente in disaccordo                o
 QOL1     A causa dei disturbi all’olfatto vado più raramente al ristorante con parenti o amici.                        D’accordo                                 o
                                                                                                                        Abbastanza d’accordo                      o
                                                                                                                        Abbastanza in disaccordo                  o
                                                                                                                        Totalmente in disaccordo                  o
 QOL2     Temo che non riuscirò mai ad abituarmi a questo problema.                                                     D’accordo                                 o
                                                                                                                        Abbastanza d’accordo                      o
                                                                                                                        Abbastanza in disaccordo                  o
                                                                                                                        Totalmente in disaccordo                  o
 QOL3     Faccio fatica a rilassarmi a causa dei disturbi dell’olfatto di cui soffro                                    D’accordo                                 o
                                                                                                                        Abbastanza d’accordo                      o
                                                                                                                        Abbastanza in disaccordo                  o
                                                                                                                        Totalmente in disaccordo                  o
 QOL4     Mi sento isolato/a dalle altre persone per via delle difficoltà con l’olfatto.                                D’accordo                                 o
                                                                                                                        Abbastanza d’accordo                      o
                                                                                                                        Abbastanza in disaccordo                  o
                                                                                                                        Totalmente in disaccordo                  o
 QOL5     A causa dei disturbi dell’olfatto, mangio di più/di meno rispetto a prima.                                    D’accordo                                 o
                                                                                                                        Abbastanza d’accordo                      o
                                                                                                                        Abbastanza in disaccordo                  o
                                                                                                                        Totalmente in disaccordo                  o
 QOL6     I disturbi dell’olfatto mi causano problemi nelle mie attività quotidiane.                                    D’accordo                                 o
                                                                                                                        Abbastanza d’accordo                      o
                                                                                                                        Abbastanza in disaccordo                  o
                                                                                                                        Totalmente in disaccordo                  o
 QOL7     Le difficoltà con l’olfatto mi rendono nervoso/a.                                                             D’accordo                                 o
                                                                                                                        Abbastanza d’accordo                      o
                                                                                                                        Abbastanza in disaccordo                  o
                                                                                                                        Totalmente in disaccordo                  o




                                                                                                                                                                 255
A. Cardella et al.




selected for test-retest reliability analysis. For this purpose,     Phase 5: responsiveness
the Brief-IT-QOD was distributed and filled in twice with            To evaluate the ability of the Brief-IT-QOD to detect im-
a 2-week interval by patients. A variation of two days be-           portant changes over time in the construct to be measured,
fore or after the requested two weeks was accepted in case           a novel cohort of 10 patients with OD due to recalcitrant
of patient’s needs. A 2-week interval period was selected            CRSwNP was recruited. Individuals included in this group
because no substantial change in olfactory abilities was             experienced a significant improvement in nasal condition
expected to take place within this period. No access to an-          and had a measured increase in olfactory abilities (at Ex-
swers given to the first questionnaire was granted to pa-            tended Smell Test) after biological therapy with dupilumab.
tients when filling in the second Brief-IT-QOD. Test-retest          This is a monoclonal antibody that inhibits signalling of
reliability was assessed through Internal Consistency Coef-          both IL-4 and IL-13, which are key cytokines in type-2
ficient (ICC). Correlation strength was considered strong            mediated inflammation. Dupilumab is administered sub-
for values greater than 0.5, moderate for values ranging             cutaneously for the treatment of adults with inadequately
between 0.3 and 0.5 and weak for values less than 0.3 16.            controlled CRSwNP and has been demonstrated to produce
                                                                     a rapid and sustained improvement in the sense of smell 19.
Phase 3: normative data                                              Each patient autonomously filled in the Brief-IT-QOD be-
The aim of this phase was to establish the baseline distribu-        fore and after 2 months of therapy.
tion for Brief-IT-QOD scores in a representative sample of
normosmic subjects with no history or symptoms of OD. In             Phase 6: cut-off value
all, 303 asymptomatic control subjects, 141 males and 162            The cut-off value of the Brief-IT-QOD was determined
females, with a mean TDI score of 34.1 ± 2.6 (31-40) and             based on the sensitivity and specificity indicators of the
no past medical history of sinonasal, neoplastic, airway,            questionnaire using the ‘‘receiver operating characteris-
neurologic, rheumatologic, haematologic, or endocrinolog-            tic’’ (ROC) curve 20. The latter represents the relationship
ic disorders were enrolled. The mean age of the normosmic            between the sensitivity and the specificity of a test by de-
subjects was 54.9 ± 17.6 years (range 18-87). Each subject           termining the real value of these two categories. In other
managed to complete the Brief-IT-QOD without any help.               words, the efficiency of a test is determined by its ability
The data obtained from this group of patients were also              to correctly identify both positive and negative cases. The
used for clinical validity analysis of phase 4 of the study.         maximum value of 1.0 for sensitivity and specificity indi-
Four age categories were considered (18-40 years, 41-60              cates a test of maximum efficiency to evaluate its purpose.
years, 61-80 years, and > 81 years).
                                                                     Statistical analysis
Phase 4: validity                                                    Statistical tests were performed using SPSS 23 statistical
The aim of the 4th phase of the study was to assess the degree       software (SPSS, Inc., Chicago, IL). Kolmogorov-Smirnov
to which the Brief-IT-QOD measures the construct it purports         test was used to test the normality of the distribution of Brief-
to measure (validity) 17. Construct validity was assessed by         IT-QOD scores among patients and healthy subjects. Since
comparing the Brief-IT-QOD scores obtained in patients with          this test demonstrated that the distribution of the scores was
OD and in normosmic subjects. In addition, in order to define        normal in both groups, parametric tests were used. The in-
a clinically relevant difference score for purposes of group         ternal consistency was assessed using Cronbach’s alpha co-
comparisons, Cohen’s effect sizes (ES) were calculated for           efficient. ICC was used to evaluate the test-retest reliability
each of the subscales of the Brief-IT-QOD. A clinically rel-         of the Brief-IT-QOD by comparing baseline and retesting
evant difference score for the Brief-IT-QOD subscales to use         responses. The Anova test was used to evaluate the Brief-IT-
in group comparisons was defined as an effect size of 0.50 or        QOD scores among the different age groups of asymptomat-
greater. Criterion validity evaluates the ability of the Brief-IT-   ic subjects. Student’s t-test was used to compare the results
QOD to adequately reflect olfactory-related QoL. However,            obtained in OD patients and in the control group. The effect
since there is no instrument that is able to assess olfactory-       size was calculated as the difference between the experimen-
related QoL validated into Italian, we decided to collect the        tal group mean minus the control group mean, divided by the
Sino-Nasal Outcome Test-22 (I-SNOT-22) 18 scores (a ques-            standard deviation of the control group 21. The correlation
tionnaire made of 22 CRS-related items which contains also a         between Brief-IT-QOD and I-SNOT-22 scores was assessed
question related to OD) and to analyse their correlations with       using Pearson test. The distribution of Brief-IT-QOD scores
Brief-IT-QOD scores. In addition, the correlations between           obtained in pre- and post-treatment evaluations were com-
the Brief-IT-QOD scores and the results of the extended ver-         pared using the Mann-Whitney test. For all statistical com-
sion of the Sniffing test were also evaluated.                       parisons an a = 0.05 and a power of 0.80 were used.



256
                                                                 Reliability and validity of the Italian version of the brief Questionnaire of Olfactory Disorders




Results                                                                           test revealed a significant difference between the OD group
                                                                                  and the control group for both the QOD-P and QOD-NS
All patients and control subjects included in the study man-                      subscales. ES results are also reported in Table IV, showing
aged to complete the Brief-IT-QOD without needing assis-                          a significant effect size for both subscales.
tance. The time required to fill out the questionnaire never                      The correlation between Brief-IT-QOD and I-SNOT-22
exceeded 7 minutes.                                                               scores obtained in the group of patients with OD was
                                                                                  analysed for concurrent validity. Positive significant cor-
Phase 2: internal consistency and reliability analysis                            relations were found between the I-SNOT-22 and QOD-P
Internal consistency was satisfactory with a Cronbach al-                         scores (r = 0.275, Fig. 1) and QOD-NS scores (r = 0.321,
pha score of a = 0.78 for the QOD-P subscale and a = 0.97                         Fig. 2). In addition, significant correlations were also found
for the QOD-NS subscale. In addition, the test-retest reli-                       between the extended version of the Sniffing test scores
ability was satisfactory for both the subscales with an ICC                       and QOD-P scores (r = -0.258, Fig. 3) and QOD-NS scores
of 0.77 (0.71-0.86) for the QOD-P subscale and 0.91 (0.84-                        (r = -0.403, Fig. 4)
0.95) for the QOD-NS subscale.
                                                                                  Phase 5: responsiveness
Phase 3: normative data                                                           Brief-IT-QOD scores obtained by a group of 10 patients
The mean age of normosmic subjects (n = 303) was 54.9                             with OD due to recalcitrant CRSwNP, who referred a sig-
years (18-87). Forty-seven percent of subjects were males.                        nificant improvement of nasal condition and demonstrated
Mean Brief-IT-QOD scores corresponding to different age                           an improvement in olfactory abilities measured by the Ex-
categories are reported in Table III. No significant differ-                      tended Smell Test after two months of biological therapy
ences among the four age categories of normosmic subjects                         with dupilumab, were compared for responsiveness analy-
were demonstrated on Anova test for either the QOD-P                              sis (7 of 10 patients were anosmic before the beginning of
(p = 0.089) or QOD-NS (p = 0.271) subscales.                                      the therapy and 3 of 10 patients were anosmic after two
                                                                                  months of therapy). The mean QOD-P score in the pre-
Phase 4: validity                                                                 treatment condition was 5.7 ± 1.8, while the post-treatment
For clinical validity analysis, the Brief-IT-QOD scores                           score was 1.8 ± 1.6. Similarly, the mean QOD-NS score in
obtained in patients with OD were compared through Stu-                           the pre-treatment condition was 13.8 ± 1.3, while the post-
dent’s t test to the scores obtained by normosmic subjects.                       treatment score was 4.3 ± 5.1. These differences were sta-
The results of this comparison are reported in Table IV. The                      tistically significant (p = 0.001 and p = 0.001 respectively)



Table III. Mean ± standard deviation of Brief-IT-QOD subscales scores in asymptomatic subjects at different ages. Ranges are reported in parentheses.
                                                                                                   Age
                                                       18-40                  41-60                  61-80                   > 81                  Total
                                                      (n = 73)              (n = 113)               (n = 92)               (n = 25)
 QOD-P                                               1.5 ± 2.3              1.3 ± 1.8              2.4 ± 2.6              2.4 ± 2.1              1.6 ± 2.2
                                                        (0-8)                 (0-6)                   (0-8)                 (0.8)                   (0-8)
 QOD-NS                                              0.6 ± 1.7              0.2 ± 1.1              0.6 ± 1.9              0.6 ± 1.5              0.4 ± 1.6
                                                       (0-10)                 (0-7)                  (0-11)                 (0-7)                  (0-11)




Table IV. Mean ± standard deviation of the Brief-IT-QOD subscales scores in patients with OD and in asymptomatic subjects. Ranges are reported in parenthe-
ses. The results of Student’s t test are reported as well as those of Cohen’s effect size.
                                         Asymptomatic subjects              Patients with OD             P score                       Cohen’s d
                                                 (n = 303)                      (n = 112)
 QOD-P                                       1.6 ± 2.2                      4.6 ± 3.4                       P = 0.001                       D = 2.793
                                                (0-8)                         (0-12)
 QOL-QOD                                     0.4 ± 1.6                      4.3 ± 5.1                       P = 0.001                       D = 2.986
                                               (0-11)                         (0-19)




                                                                                                                                                             257
A. Cardella et al.




Figure 1. Correlation between the Parosmia subscale (QOD-P) scores of the          Figure 4. Correlation between the Negative Statement subscale (QOD-NS)
Italian version of the Brief Questionnaire of Olfactory Disorders (Brief-IT-QOD)   scores of the Italian version of the Brief Questionnaire of Olfactory Disorders
and the Sino-Nasal Outcome Test-22 (SNOT-22). Tendency line with the cor-          (Brief-IT-QOD) and the extended version of the Sniffing test scores (TDI). Ten-
relation coefficient is also reported.                                             dency line with the correlation coefficient is also reported.


                                                                                   using a Mann-Whitney test, suggesting a positive evolution
                                                                                   of OD.

                                                                                   Phase 6: cut-off value
                                                                                   The comparison of the Brief-IT-QOD subscale scores
                                                                                   obtained in patients with OD and in normosmic subjects
                                                                                   demonstrated a significant difference between these two
                                                                                   groups. Consequently, the samples could be submitted to
                                                                                   ROC curve analysis, which allowed for determination of a
                                                                                   cut-off value to discriminate the groups. In order to iden-
                                                                                   tify the Brief-IT-QOD cut-off value, the highest values of
                                                                                   sensitivity and specificity were considered. The values of
                                                                                   sensitivity and specificity for different Brief-IT-QOD sub-
Figure 2. Correlation between the Negative Statement subscale (QOD-NS)
scores of the Italian version of the Brief Questionnaire of Olfactory Disorders
                                                                                   scale scores are reported in Table V, while the ROC graph is
(Brief-IT-QOD) and the Sino-Nasal Outcome Test-22 (SNOT-22). Tendency line         displayed in Figure 5. A QOD-P cut-off value of 3.5 dem-
with the correlation coefficient is also reported.                                 onstrated a sensitivity of 62.5% and specificity of 80.3%,
                                                                                   while a QOD-NS cut-off value of 1.5 demonstrated a sensi-
                                                                                   tivity of 55.4% and specificity of 86.6%.

                                                                                   Discussion
                                                                                   The QOD is a self-reported inventory that assesses QoL
                                                                                   related to OD, initially developed in Germany and then
                                                                                   adapted to different cultural and linguistic contexts, namely
                                                                                   Chinese, Persian, Korean and English 4-8. Mattos’ brief ver-
                                                                                   sion of the QOD-NS 10, on the other hand, was originally
                                                                                   developed in English and has been translated and adapted
                                                                                   into Spanish and French 22,23. Zou’s brief-QOD 11, compris-
                                                                                   ing the 7 items of Mattos’ QOD-NS 10 plus 4 items on pa-
                                                                                   rosmia and 3 visual analog scales, was created in English
                                                                                   and not adapted to any other language. An Italian transla-
Figure 3. Correlation between the Parosmia subscale (QOD-P) scores of the
Italian version of the Brief Questionnaire of Olfactory Disorders (Brief-IT-QOD)
                                                                                   tion of the QOD, in any of its versions, is lacking.
and the extended version of the Sniffing test scores (TDI). Tendency line with     In this study, the psychometric properties of the Brief-IT-
the correlation coefficient is also reported.                                      QOD were analysed in a group of 112 patients with OD and



258
                                                                 Reliability and validity of the Italian version of the brief Questionnaire of Olfactory Disorders




Table V. Coordinates of the ROC Curve for the cut-off. Value of the P-QOD         in a control group of 303 subjects. The results appear prom-
and QOD-NS subscales of the Brief-IT-QOD.                                         ising and seem to indicate that the Brief-IT-QOD can be ap-
                       Cut-off value      Sensitivity       Specificity           plied in Italian patients with OD. In particular, all question-
 P-QOD                       -1,00              1.00               0              naires were completely filled in, suggesting that all subjects
                              0.50             0.813             0.528            understood the questions well and felt comfortable answer-
                              1.50             0.750             0.641            ing them. This allows to speculate that the Brief-IT-QOD
                              2.50             0.670             0.669            is not a burdensome instrument, is easily self-administered,
                              3.50             0.625             0.803            and can be quickly filled out.
                              4.50             0.509             0.859            The internal consistency of the Brief-IT-QOD appeared
                              5.50             0.402             0.908            satisfactory, with the QOD-P subscale having an α of 0.78
                              6.50             0.304             0.972            and the QOD-NS subscale one of 0.97. The original study
                              7.50             0.259             0.972            by Zou et al. 11 had Cronbach’s α coefficients of 0.63 and
                              8.50             0.152             1.000            0.87 for QOD-P and QOD-NS, respectively, indicating an
                              9.50             0.080             1.000
                                                                                  acceptable or excellent reliability. No other translation of
                             10.50             0.027             1.000
                                                                                  this specific questionnaire is available in the literature, but
                                                                                  our results for the QOD-NS subscale can be compared to
                             11.50             0.009             1.000
                                                                                  those from Mattos’ QOD-NS 10 and its adaptations. In fact,
 QOD-NS                      -1,00             1.000               0
                                                                                  the French translation of QOD-NS (Fr-sQOD-NS) 2,3 had
                              0.50             0.571             0.817
                                                                                  α = 0.96, while the internal consistency of Chiesa-Estom-
                              1.50             0.554             0.866
                                                                                  ba’s Spanish version 22 measured with Cronbach α was
                              2.50             0.464             0.894            0.86.
                              3.50             0.429             0.923            As far as the reliability of the Brief-IT-QOD is concerned,
                              4.50             0.375             0.944            the scores obtained in the test-retest analysis support the
                              5.50             0.321             0.951            hypothesis that Brief-IT-QOD has high stability and repro-
                              6.50             0.295             0.965            ducibility over time. In fact, ICC scores were 0.77 for the
                              7.50             0.250             0.986            QOD-P subscale and 0.91 for the QOD-NS subscale. These
                              8.50             0.232             0.986            values can be considered optimal both for group compari-
                              9.50             0.205             0.986            son and individual measurements over time. It is difficult to
                             10.50             0.152             0.993            compare our results with the literature, as previous studies
                             11.50             0.125             1.000            adopted different methods to evaluate reliability. The origi-
                                                                                  nal study 11 used split-half reliability which was 0.60 and
                                                                                  0.87 for QOD-P and QOD-NS, respectively. The test-retest
                                                                                  reliability of Fr-sQOD-NS 23 was high as well (r = 0.88,
                                                                                  p < .001). Finally, the Spanish study 22 used the intraclass
                                                                                  correlation coefficient to evaluate test-retest reliability and
                                                                                  found a value of 0.85, which implies a high correlation.
                                                                                  To the best of our knowledge, no normative data on brief-
                                                                                  QOD are available. In our sample, no difference in mean
                                                                                  Brief-IT-QOD scores among the four age groups was ob-
                                                                                  served. This supports the use of the test in patients with
                                                                                  OD, regardless of their age. Our results of QOD-NS for
                                                                                  normosmic subjects differ from those presented in the Fr-
                                                                                  sQOD-NS study 23, with our cohort having lower average
                                                                                  scores. However, comparison between the two studies is
                                                                                  difficult, as the populations differ in age and numerosity.
                                                                                  Furthermore, Leclercq and colleagues 23 do not specify
                                                                                  how the controls were selected and it is not clear whether
                                                                                  they undertook psychophysical olfactory testing to deter-
Figure 5. ROC curves of the Parosmia subscale (P-QOD) and Negative
                                                                                  mine that their olfactory function was indeed normal. On
Statement subscale (NS-QOD) of the Italian version of the Brief Questionnaire
of Olfactory Disorders (Brief-IT-QOD) and the Sino-Nasal Outcome Test-22          the contrary, in this study we enrolled only subjects who
(SNOT-22).                                                                        scored normal at the extended version of the Sniffing test.



                                                                                                                                                            259
A. Cardella et al.




As for the clinical validity of the Brief-IT-QOD, patients         measures is a five-stage process: translation (stage I), syn-
with OD scored higher than normosmic subjects in both              thesis (stage II), back-translation (stage III), expert com-
the parosmia and QoL subscales. Therefore, it is possible to       mittee review (stage IV), and pretesting (stage V). In our
speculate that Brief-IT-QOD may be a sensitive tool to dis-        study, pretesting was performed as a pilot study (step 3 of
criminate olfactory-related QoL in patients with and with-         Phase 1) before back-translation and expert committee re-
out OD. Similar to our results, the original study 11 found        view; while this should be considered a limitation, we felt
a significantly higher QOD-P score in the hyposmic group           that patient input would be crucial to better define the items
(4.04 ± 3.13) than in the normosmic group (3.03 ± 2.84).           into Italian. This updated version was then subjected to ex-
On the other hand, Zou and colleagues 11 did not obtain            pert review by the professionals involved (step 4 of Phase
significant differences in the QOL-QOD subscale between            1) and eventually underwent backward translation (step 5
patients and controls. Conversely, in their validation of a        of Phase 1). The process we adopted might have impacted
shortened, 25-item version of the QOD, Simopoulos et al. 24        the Brief-QOD final Italian version.
showed a significant difference in QOD-NS scores between           A second limitation lies in the fact that OD and not QoL
patients and normosmic subjects, analogous to our results.         reduction associated with OD was the criterion adopted to
Furthermore, when analysing concurrent validity, we found          include both patients and healthy subjects in the study. This
positive correlations between the results of the extended
                                                                   was due to the fact that a validated measure of OD-related
version of the Sniffing test and Brief-IT-QOD subscales
                                                                   QoL was not available in the Italian language. It is therefore
and between these subscale scores and I-SNOT-22. How-
                                                                   theoretically possible that patients with OD did not present
ever, it should be noted that the two questionnaires do not
                                                                   a reduction in OD-related QoL, and that, on the other hand,
measure the same constructs (QOD is specific for olfactory
                                                                   normosmic subjects might have presented a reduction in
disturbances, while SNOT-22 investigates general sinona-
                                                                   OD-related QoL, in turn. Besides, in the responsiveness
sal complaints but contains a question related to OD).
Comparison with the available literature is again complex.         analysis of the Brief-IT-QOD, it cannot be assumed that an
To evaluate external validity, Leclercq et al. 23 used a visual    improvement in olfactory function would have necessarily
analogue scale related to OD, while they did not perform           resulted in an improvement in QoL measures. However, the
a correlation analysis with SNOT-22. Conversely, neither           patients enrolled in this phase of the study improved their
Zou et al. 11 nor Chiesa-Estomba et al. 22 reported on ex-         TDI scores at the Extended Smell Test by Sniffin’ Sticks
ternal validity. Finally, Simopoulos et al. 24 did not assess      (Burghart Messtechnik GmbH, Germany). We assumed
the external validity of QOD-NS, but only that of QOD-             consequently that the improvement of olfactory abilities
positive statements.                                               could be related to improvement of the Brief-IT-QOD.
The significant decrease recorded in Brief-IT-QOD scores
after successful biologic therapy for recalcitrant CRSwNP          Conclusions
suggests that Brief-IT-QOD is responsive to changes in the
QoL of patients with OD. No previous report on respon-             The Brief-IT-QOD is a reliable and valid self-administered,
siveness of QOD is available. However, the data reported           symptom-specific outcome tool for OD in adult Italian pa-
herein (even if obtained in a small group of patients) sup-        tients. The application of a standardised OD-specific in-
port the clinical applicability of the Brief-IT-QOD in the         strument in everyday clinical practice, such as the Brief-IT-
monitoring of patients’ outcomes.                                  QOD, as well as in epidemiological, efficacy, and outcome
Lastly, cut-off determination was carried out through ROC          research is therefore recommended, as it can facilitate com-
curve analysis, obtaining a QOD-P cut-off value of 3.5 and         parison of results of different studies.
a QOL-QOD cut-off value of 1.5. No literature on this topic
is available to make comparisons with the values we deter-         Acknowledgement
mined. However, the low sensitivity and specificity scores         The authors acknowledge support from the University of
of the two subscales found in this study limit the use of the      Milan through the APC initiative.
Brief-IT-QOD for screening purposes.
                                                                   Conflict of interest statement
Study limitations                                                  The authors declare no conflict of interest.
The present study has several limitations. First of all, the
cross-cultural adaptation from English into Italian did not        Funding
strictly follow international guidelines 12,25. In fact, accord-   This research did not receive any specific grant from funding
ing to these, the cross-cultural adaptation of self-report         agencies in the public, commercial, or not-for-profit sectors.



260
                                                                 Reliability and validity of the Italian version of the brief Questionnaire of Olfactory Disorders




Author contributions                                                                   nosinusitis. Int Forum Allergy Rhinol 2019;9:1144-1150. https://doi.
                                                                                       org/10.1002/alr.22392
FM, LL, RA, FrO: study conception and design; AC, AP,                             11
                                                                                       Zou L, Haehner A, Menzel S, et al. Reliability and validity of a brief
AA, DC, GM, FiO: material preparation and data collec-                                 version of the Questionnaire of Olfactory Disorders (brief QOD) in
tion; GP, FM: conducted the statistical analysis and inter-                            patients with olfactory dysfunction. Rhinology 2022;60:56-62. htt-
pretation of results. The first draft of the manuscript was                            ps://doi.org/10.4193/Rhin21.059
written by AC, FM, GR, GP. All the Authors commented on                           12
                                                                                       Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN study
                                                                                       reached international consensus on taxonomy, terminology, and
previous versions of the manuscript and read and approved                              definitions of measurement properties for health-related patient-
the final manuscript.                                                                  reported outcomes. J Clin Epidemiol 2010;63:737-745. https://doi.
                                                                                       org/10.1016/j.jclinepi.2010.02.006
Ethical consideration
                                                                                  13
                                                                                       Hummel T, Sekinger B, Wolf SR, et al. “Sniffin’ Sticks”: olfac-
                                                                                       tory performance assessed by the combined testing of odor identi-
This study was approved by the Institutional Ethics Com-                               fication, odor discrimination and olfactory threshold. Chem Senses
mittee (Comitato Etico Indipendente IRCCS Multimedica)                                 1997;22:39-52. https://doi.org/10.1093/chemse/22.1.39
(Protocol n. 506.2021).                                                           14
                                                                                       Oleszkiewicz A, Schriever VA, Croy I, et al. Updated Sniffin’ sticks
The research was conducted ethically, with all study pro-                              normative data based on an extended sample of 9139 subjects. Eur
                                                                                       Arch Otorhinolaryngol 2019;276:719-728. https://doi.org/10.1007/
cedures being performed in accordance with the require-                                s00405-018-5248-1
ments of the World Medical Association’s Declaration of                           15
                                                                                       Terwee CB, Bot SDM, de Boer MR, et al. Quality criteria were
Helsinki.                                                                              proposed for measurement properties of health status question-
Written informed consent was obtained from each partici-                               naires. J Clin Epidemiol 2007;60:34-42. https://doi.org/10.1016/j.
                                                                                       jclinepi.2006.03.012
pant/patient for study participation and data publication.                        16
                                                                                       Prieto L, Lamarca R, Casado A. Assessment of the reliability of clini-
                                                                                       cal findings: the intraclass correlation coefficient. Med Clin (Barc)
                                                                                       1998;110:142-145.
References                                                                        17
                                                                                       Vittinghoff E, Glidden DV, Shiboski SC, et al. Regression methods in
1
     Hummel T, Whitcroft KL, Andrews P, et al. Position paper on ol-                   biostatistics. Second edition. New York, NW: Springer; 2012. https://
     factory dysfunction. Rhinol (Suppl.) 2017;54:1-30. https://doi.                   doi.org/10.1007/978-1-4614-1353-0
     org/10.4193/Rhino16.248                                                      18
                                                                                       Mozzanica F, Preti A, Gera R, et al. Cross-cultural adaptation and
2
     Croy I, Nordin S, Hummel T. Olfactory disorders and quality of                    validation of the SNOT-22 into Italian. Eur Arch Otorhinolaryngol
     life – an updated review. Chem Senses 2014;39:185-194. https://doi.               2017;274:887-895. https://doi.org/10.1007/s00405-016-4313-x
     org/10.1093/chemse/bjt072                                                    19
                                                                                       Mullol J, Bachert C, Amin N, et al. Olfactory outcomes with
3
     Wilson RS, Yu L, Bennett DA. Odor identification and mortal-                      dupilumab in chronic rhinosinusitis with nasal polyps. J Allergy Clin
     ity in old age. Chem Senses 2011;36:63-67. https://doi.org/10.1093/               Immunol Pract 2022;10:1086-1095.e5. https://doi.org/10.1016/j.
     chemse/bjq098                                                                     jaip.2021.09.037
4
     Frasnelli J, Hummel T. Olfactory dysfunction and daily life. Eur             20
                                                                                       Fawcett T. An introduction to ROC analysis. Pattern Recognit Lett
     Arch Otorhinolaryngol 2005;262:231-235. https://doi.org/10.1007/                  2006;27:861-874. https://doi.org/10.1016/j.patrec.2005.10.010
     s00405-004-0796-y                                                            21
                                                                                       Rinkel RN, Verdonck-de Leeuw IM, Langendijk JA, et al. The psy-
5
     Yang D, Wang J, Ni D, et al. Reliability and validity of the Chinese              chometric and clinical validity of the SWAL-QOL questionnaire in
     version of the questionnaire of olfactory disorders (QOD) when used               evaluating swallowing problems experienced by patients with oral
     with patients having olfactory dysfunction. Eur Arch Otorhinolaryn-               and oropharyngeal cancer. Oral Oncol 2009;45:e67-71. https://doi.
     gol 2016;273:3255-3261. https://doi.org/10.1007/s00405-015-3869-1                 org/10.1016/j.oraloncology.2009.03.003
6
     Jalessi M, Kamrava SK, Amini E, et al. Is the Persian version of the
                                                                                  22
                                                                                       Chiesa-Estomba CM, Lechien JR, Calvo-Henríquez C, et al. Transla-
     “Olfactory Disorder Questionnaire” reliable and valid? Iran J Otorhi-             tion and validation of the short version of the Questionnaire of Olfac-
     nolaryngol 2017;29:209-213.                                                       tory Disorders – Negative Statements to Spanish. Am J Otolaryngol
                                                                                       2021;42:102775. https://doi.org/10.1016/j.amjoto.2020.102775
7
     Choi WR, Jeong HY, Kim JH. Reliability and validity of the Korean
     version of the Questionnaire of Olfactory Disorders: Korean version
                                                                                  23
                                                                                       Leclercq C, Chiesa-Estomba CM, Horoi M, et al. Valid-
     of QOD. Int Forum Allergy Rhinol 2018;8:1481-1485. https://doi.                   ity and reliability of the French short version of the Question-
                                                                                       naire of Olfactory Disorders-Negative Statements (sQOD-
     org/10.1002/alr.22186
                                                                                       NS). Ear Nose Throat J 2021:014556132110320. https://doi.
8
     Langstaff L, Pradhan N, Clark A, et al. Validation of the olfactory dis-          org/10.1177/01455613211032004
     orders questionnaire for English-speaking patients with olfactory dis-       24
                                                                                       Simopoulos E, Katotomichelakis M, Gouveris H, et al. Olfaction-
     orders. Clin Otolaryngol 2019;44:715-728. https://doi.org/10.1111/
                                                                                       associated quality of life in chronic rhinosinusitis: adaptation and
     coa.13351                                                                         validation of an olfaction-specific questionnaire. Laryngoscope
9
     Harel D, Baron M. Methods for shortening patient-reported outcome                 2012;122:1450-1454. https://doi.org/10.1002/lary.23349
     measures. Stat Methods Med Res 2019;28:1992-3011. https://doi.               25
                                                                                       Beaton DE, Bombardier C, Guillemin F, et al. Guide-
     org/10.1177/0962280218795187                                                      lines for the process of cross-cultural adaptation of self-
10
     Mattos JL, Edwards C, Schlosser RJ, et al. A brief version of the                 report     measures.     Spine     2000;25:3186-3191.       https://doi.
     Questionnaire of Olfactory Disorders in patients with chronic rhi-                org/10.1097/00007632-200012150-00014




                                                                                                                                                            261
