La valutazione della ricerca scientifica sulla base dei parametri di stima in economia
10.6092/issn.1973-9494/4754
Si fa prosieguo alle considerazioni riportate in un Editoriale pubblicato sul fascicolo N. 11 (2011) del Journal storico-tecnico “Conservation Science in Cultural Heritage” dal titolo “La valutazione della ricerca scientifica: risultato di criteri meritocratici o discrezionali?”, in cui si prendevano in considerazione i criteri di valutazione della ricerca scientifica e si esprimevano delle riserve relativamente ai singoli criteri bibliometrici, in riferimento a:  - Impact Factor (IF), lo strumento bibliometrico più noto, di proprietà della Thompson Reuters - Institute for Scientific Information (ISI), che raccoglie i dati di oltre 14 mila riviste presenti nel suo portale web: si tratta di un sistema di valutazione che determina la frequenza con cui un articolo viene citato in un determinato periodo.  - Peer Review, indicatore di qualità tipicamente utilizzato nella selezione degli articoli pubblicabili;  - Open Linking, servizio di referenza offerto dagli aggregatori, che trasforma le citazioni in hyperlink e consente ai ricercatori di navigare online da articolo a articolo, indipendentemente dalla rivista e dalla casa editrice;  - Fattore H o h index che punta a quantificare il contributo scientifico complessivo di un ricercatore.
application/pdf
1973-9494
4754
en
Andrea Natali
Conservation Science in Cultural Heritage
economics
The evaluation of scientific research on the basis of estimation parameters in economics
Text.Serial.Journal
https://conservation-science.unibo.it/
14
Articles
2014-12-20
2015-02-25
2014-12-20
2015-11-02
387-394
CSCH
Andrea Natali
2014/12/20
10.6092/issn.1973-9494/4754
387
1973-9494
economics
en
394
The evaluation of scientific research on the basis of estimation parameters in economics
https://conservation-science.unibo.it/article/download/4754/4244
14
T    HE EVALUATION OF SCIENTIFIC RESEARCH ON THE
     BASIS OF ESTIMATION PARAMETERS IN ECONOMICS
   Salvatore Lorusso•*
   Dipartimento di Beni Culturali
   Alma Mater Studiorum Università di Bologna (sede di Ravenna), Italy

   Vincenzo Barone
   Scuola Normale Superiore di Pisa, Italy

   with the collaboration of Andrea Natali


   Keywords: evaluation, scientific research, economics


   Below are some considerations following on from a past Editorial entitled “The
evaluation of scientific research: the result of merit-based or discretionary criteria?”
published in issue No.11 (2011) of the historical-technical journal “Conservation Sci-
ence in Cultural Heritage”. The Editorial looked at various tools for evaluating scientific
research and expressed some reservations about the bibliometric criteria:
     - Impact Factor (IF), the most well-known bibliometric tool, owned by Thomson
         Reuters – Institute for Scientific Information (ISI), which collects data from
         over 14,000 journals in its web portal: an evaluation system that determines
         the frequency with which an article is cited in a given period;
     - Peer Review, a quality indicator typically used in selecting articles for publica-
         tion;




                                                                                                     CONSERVATION SCIENCE IN CULTURAL HERITAGE
     - Open Linking, a reference service offered by aggregators which transforms
         citations into hyperlinks and allows researchers to browse online from article
         to article regardless of the journal or publisher;
     - H Factor or H Index, which aims to quantify the overall scientific contribution
         of a researcher.
   The Editorial’s objective was to provide a stimulus which was strictly for reflection
only and devoid of any evaluative claims – a simple explanation of the mechanisms
of evaluation used and given openly – in order to apply the said evaluation criteria not
only to disciplines of a technical-experimental-naturalistic nature, but also to historical-
humanistic and economic areas, in particular to research in the field of cultural and
environmental heritage.
   With the intent of connecting the key aspects and relevant tools characterizing this
evaluation with the estimation parameters that measure the ills Europe is struggling
with – a Europe that today is divided by a great many conflicting national interests – the
question arises: “Why not call attention to the fact that conceptual terms, fundamental
in economics and greatly favoured by economists, may be proposed again, with the
necessary adaptations, in certain situations existing in the field of scientific production
and research?”



• Corresponding author: salvatore.lorusso@unibo.it

                                                                                               387
                                                                                                                                  The specific terms are “growth”, “inflation” and “deflation”, in reference to which it
                                                                                                                               is well-known that:

                                                                                                                                  a)    Economic growth in a macroeconomic environment is characterized by the
                                                                                                                                        long and short term increase in the development of society. It corresponds
                                                                                                                                        to a general increase in the level of variables such as wealth, consumption,
S. Lorusso, V. Barone, A. Natali - The Evaluation of Scientific Research on the Basis of Estimation Parameters in Economics




                                                                                                                                        production of goods, provision of services, employment, research and so on.
                                                                                                                                        Zero growth exists in a situation where the economy does not grow and re-
                                                                                                                                        mains stationary or when there is negative growth, as in the case of degrowth.
                                                                                                                                        These situations are also found in scientific contexts relating to trends char-
                                                                                                                                        acterized by: educational activities, scientific production, number of teaching
                                                                                                                                        staff, international mobility of students, research projects, consultancy, part-
                                                                                                                                        nerships, awards, on the basis of meritocratic criteria established over the
                                                                                                                                        years, also at an international level; the trend highlights the growth of one
                                                                                                                                        scientific unit with respect to another that is stationary or has negative growth.
                                                                                                                                  b)    Inflation in economics, on the other hand, indicates a general increase in pric-
                                                                                                                                        es which continues over a period of time. The consumer price index (CPI) is a
                                                                                                                                        statistical tool that measures changes over a period of time in market basket
                                                                                                                                        prices, a sample of goods and services typical of a given country’s household
                                                                                                                                        consumption.
                                                                                                                                        In a scientific context it could more specifically represent one of the param-
                                                                                                                                        eters to be followed, as it is considered significant and emblematic of a grow-
                                                                                                                                        ing reality: one example is the quality of publications in this field based on
                                                                                                                                        common objective criteria at an international level. Just as in economics the
                                                                                                                                        main objective of central banks is to maintain price stability, which is one of
                                                                                                                                        the basic conditions for raising the level of economic activity and employment,
                                                                                                                                        there is no doubt that in a scientific field too, a situation which responds to a
                                                                                                                                        level of significant though stable level of growth is obviously an acquisition of
                                                                                                                                        positive quality.
                                                                                                                                  c)    Deflation corresponds to a general drop in prices, in other words the opposite
                                                                                                                                        of inflation. It is often the result of a recessionary situation in the economy, that
                                                                                                                                        is, of negative growth, in which the demand for goods and services contracts.
                                                                                                                                        This is what was underlined earlier in the case of economic contraction, in
                                                                                                                                        other words, when there is a decline in scientific production or even when it is
                                                                                                                                        non-existent. Nationally and internationally, the resulting classification in the
                                                                                                                                        meritocratic elite of a scientific unit determines a position at a low level and is
                                                                                                                                        therefore negative.

                                                                                                                                  To establish the top universities in Italy, several ‘indicators’ relating to education
                                                                                                                               and research1 are used for assessment. Below is the list compiled by experts from the
                                                                                                                               newspaper “Il Sole 24 Ore”, with input from the Agenzia Nazionale di Valutazione del
                                                                                                                               Sistema Universitario e della Ricerca (ANVUR – the national agency which evaluates
                                                                                                                               the quality of Italian universities and research institutes), following the VQR (Valutazi-
                                                                                                                               one della Qualità della Ricerca) report for evaluating the quality of research: the follow-
                                                                                                                               ing twelve indicators were used, divided into two fields - didactics and research. The
                                                                                                                               final result in the national rankings is given below.




                                                                                                                         388
            • Didactics

        	ATTRACTIVENESS: percentage of student enrolment outside regional
          areas out of the total number of students enrolled;

        	SUSTAINABILITY: average number of teaching staff engaged in basic ac-
          tivities with distinctive characteristics for each degree course;

        	INTERNSHIPS: percentage of credits earned after completion of intern-
          ship out of the total;

        	INTERNATIONAL MOBILITY: percentage of credits earned abroad;

        	GRANTS: percentage of successful candidates who have earned a grant;

        	DISPERSION: percentage of matriculations registered in the second year
          at the same university;

        	EFFICACY: percentage of matriculations registered in the second year at
          the same university;

        	SATISFACTION: post-graduates’ opinion of degree programmes;

        	EMPLOYMENT: number of students seeking employment one year after
          graduation.




                                                                                               CONSERVATION SCIENCE IN CULTURAL HERITAGE
            • Research

        	EXTERNAL FUNDS: ability to attract resources for projects;

        	RESEARCH: opinions obtained by research products in the ANVUR eval-
          uation;

        	HIGHER EDUCATION: opinions obtained by higher education in the AN-
          VUR evaluation.


    The final results obtained using these indicators have allowed the following clas-
sification to be compiled. Table 1 below shows the top 30 in the list.




                                                                                         389
                                                                                                                               Table 1. Ranking of Italian universities for 2014: prepared by Il Sole 24 Ore – ANVUR - VQR
S. Lorusso, V. Barone, A. Natali - The Evaluation of Scientific Research on the Basis of Estimation Parameters in Economics




                                                                                                                                   In the 2014-2015 QS World University Rankings, which classifies more than 800
                                                                                                                               universities by analyzing a total of over 3000, the first Italian university, the University
                                                                                                                               of Bologna is ranked 182, while last year it was188th, followed by Sapienza University
                                                                                                                               of Rome 202nd, the Polytechnic University of Milan 229th, the University of Pisa 245th
                                                                                                                               and the University of Padua 262nd2. Out of the 800, the number of Italian universities
                                                                                                                               in the rankings is 27.



                                                                                                                         390
    The Academic Ranking of World Universities (ARWU) prepared by the Jiao Tong
University of Shanghai, on the other hand, places the top Italian universities among
the first 500 in the world between the 151st and 200th position3. Among them are the
universities of Bologna, Milan, Padua, Pisa and the Sapienza University of Rome.
    Other results are given in the Times Higher Education (THE) classification, in which
there is only one Italian university, the Scuola Normale Superiore of Pisa, number 63
in the top 200 in the world. Successive universities are listed alphabetically in groups
of 25. Other Italian universities in this classification are the University of Trieste, be-
tween 201-225; the University of Milan-Bicocca, between 226-250; University of Pavia,
University of Salento, University of Trento, University of Turin, between 251-275; the
University of Bologna and University of Milan between 276-300. In table 2 is the THE
ranking of the top 30/31 universities in the world.4


Table 2. Times Higher Education 2014-2015 classification of the top 31 universities in the world




                                                                                                         CONSERVATION SCIENCE IN CULTURAL HERITAGE




                                                                                                   391
S. Lorusso, V. Barone, A. Natali - The Evaluation of Scientific Research on the Basis of Estimation Parameters in Economics




                                                                                                                                  These classifications are evidently discordant and demonstrate how the different
                                                                                                                               indexes and criteria used can lead to different evaluations and rankings.
                                                                                                                                  It is interesting to note, that in analyzing the 2014 QS World University Rankings,
                                                                                                                               the top 10 include only British and American universities, while 11 universities out of
                                                                                                                               the top twenty in the world, are American.
                                                                                                                                  However, with the onset of the financial crisis in 2008, this American domination
                                                                                                                               has been curtailed. Of the American universities included in the top 400, many moved
                                                                                                                               down in the classification compared to 2007. A diametrically opposed situation for the
                                                                                                                               Asian universities present among the top 400, which moved up in the classification for
                                                                                                                               the same period. The top Asian university in the 2014-2015 QS rankings is the National
                                                                                                                               University of Singapore, classified 22nd.
                                                                                                                                  The connection between economics and the evaluation of universities thus leads
                                                                                                                               back to what was said earlier in relation to the meaning of the terms ‘growth’, ‘inflation’
                                                                                                                               and ‘deflation’ all of which characterize situations and conditions in any given country.
                                                                                                                               These are all aspects that can influence decisions made by central banks regarding
                                                                                                                               economic policy, but can rightly be considered benchmarks with criteria for evaluating
                                                                                                                               the quality of cultural academic and research units.
                                                                                                                                  The connection between economics and evaluating universities is even more evi-
                                                                                                                               dent when one considers the reform wanted by the Minister of Education, University
                                                                                                                               and Research, Stefania Giannini, based on ‘meritocracy’. Economic rewards for the
                                                                                                                               most deserving universities will carry more weight: the quota of state financing, based
                                                                                                                               on rewards and incentives, of the Fondo Finanziamento Ordinario (FFO), in fact goes
                                                                                                                               up from 13.5% to 18% (i.e. from 819 million to 1.3 billion). ‘Meritocracy’ is a form
                                                                                                                               of government where administrative offices, public offices and any role requiring re-



                                                                                                                         392
sponsibility towards others is assigned on the basis of merit and not membership of a
lobby, family (nepotism/favoritism and cronyism in a broader sense) or economic caste
(oligarchy).
    The term ‘meritocracy’ was first used by the British sociologist Michael Dunlop
Young in the book “Rise of the Meritocracy” (1958). The term was intended to be used
as a disparaging one and his book was the scenario for a dystopic future in which the
social position of an individual was determined by their IQ and their efforts.
    In the book, the existence of such a social system ultimately leads to a revolution;
the masses overthrow the elite which has become arrogant and detached from the
feelings of the people. Despite the originally negative connotation of the term, the
meritocratic system is fairer and more productive than other systems, besides guaran-
teeing the end of discrimination based on arbitrary criteria such as gender, race and
social origins (or membership).
    However, the question arises: how is merit measured?
    The clearest definition of what kind of merit should be rewarded is given in a recent
publication by Roger Abravanel, “Meritocracy”, which restates what Michael Young
claimed “Merit = talent + effort.” Hence the principle of merit based on talent and com-
mitment replaces less efficient allocative criteria (length of service and the like).
    Italy is somewhat behind in matters pertaining to measuring merit: there is no ac-
cepted system of evaluation for measuring the quality of didactic activities. At present
the only certain parameter is related to research, which is a first step in the right direc-
tion towards evaluating universities and consequently for receiving funding, but which
undoubtedly needs to be perfected.
    Experts from the Ministry of Education, University and Research are studying a new
system of calculating the base quota, focusing on so-called ‘standard costs’, so that
the financing each university is entitled to is closely related to the number of degree
programmes, the number of students and teachers, the relationship between them,




                                                                                                     CONSERVATION SCIENCE IN CULTURAL HERITAGE
and to merit.
    “A fair assessment of the university system can only foster meritocracy”. This is
what Marco Mancini, the then President of the Conference of Italian University Rectors
(CRUI), claimed in 2012, at the same time pointing out that: “the stricter the evaluation
criteria are, the easier it will be to prioritize meritocracy and avoid waste”. This merito-
cratic approach would help internationalization to grow and therefore be more competi-
tive on the international market enabling Italian universities to increase the number of
foreign students, researchers and teachers, and so attract private funding.


    Notes

1   Source: http://www.ilsole24ore.com/speciali/classifiche_universita_2014

2   Source: http://www.topuniversities.com/university-rankings/world-university-ran-
    kings/2014

3   After the first 100, successive universities are shown in groups of 50 and simply
    listed in alphabetical order.

4   Source: http://www.timeshighereducation.co.uk/world-university-rankings/2014-15/
    world-ranking

                                                                                               393
                                                                                                                                 Biographical notes

                                                                                                                                  Salvatore Lorusso is Full Professor in “Chemistry of the Environment and Cul-
                                                                                                                               tural Heritage” at the Department of Cultural Heritage of the University of Bologna.
                                                                                                                               He founded and is Director of the Master in “Planning and promotion of artistic and
                                                                                                                               cultural events”. He founded and is Director of the two book series’ “I beni culturali e
S. Lorusso, V. Barone, A. Natali - The Evaluation of Scientific Research on the Basis of Estimation Parameters in Economics




                                                                                                                               l’ambiente” (Pitagora Edizioni, Bologna) and “La formazione e la ricerca nel settore
                                                                                                                               dei beni culturali e ambientali” (Mimesis Edizioni, Milano-Udine) and the historical-
                                                                                                                               technical Journal “Conservation Science in Cultural Heritage”. He is the author of
                                                                                                                               more than 380 publications in national and international journals and of 22 volumes
                                                                                                                               covering various subjects and sectors, such as commodities, technology and environ-
                                                                                                                               ment, focusing in particular on the field of cultural and environmental heritage. He has
                                                                                                                               been Vice-President and actually Councellor of the “Società Italiana per il Progresso
                                                                                                                               delle Scienze (SIPS)”, established in 1839. He is General Director of the Academy
                                                                                                                               of Enogastronomic Culture. He is Foreign Member of Russian Academy of Natural
                                                                                                                               Sciences.

                                                                                                                                  Vincenzo Barone is full professor in “Theoretical and Computational Chemistry” at
                                                                                                                               the Scuola Normale Superiore of Pisa. Author of more than 650 papers in ISI journals
                                                                                                                               and of several book chapters with more than 35000 citations, an h-factor of 75, and 7
                                                                                                                               papers with more than 1000 citations each. The most significant scientific contributions
                                                                                                                               include fundamental developments in Density Functional Theory, solvation theory, and
                                                                                                                               computational spectroscopy, together with state-of-the-art applications in materials
                                                                                                                               chemistry, life sciences, nanosciences and cultural heritage. He is fellow of the Acca-
                                                                                                                               demia dei Lincei, of the International Academy of Quantum Molecular Chemistry, of the
                                                                                                                               European Academy of Sciences and of the Royal Society of Chemistry. He received
                                                                                                                               the Pisani medal in 2014 and the Sacconi medal in 2009. He has been President of
                                                                                                                               the Italian Chemical Society (SCI) and of the Chemistry Panel of the National Agency
                                                                                                                               of Research Evaluation (ANVUR), and is a member of the Advisory Boards of Spec-
                                                                                                                               trochimica Acta A, Open Chemistry, PCCP, Journal of Computational Chemistry and
                                                                                                                               Theoretical Chemistry Accounts. Further details and full reference list are available on
                                                                                                                               the website http://dreams.sns.it.




                                                                                                                         394
