title: Evaluating socio-economic programmes : volume 2, Selection and use of indicators for monitoring and evaluation.

eurovoc_concepts:
EU programme
Structural Funds
economic and social cohesion
economic development
project evaluation
social development
structural policy

formats: pdf print

text: EC Structural Funds Evaluating socio-economic programmes Selection and use of indicat ors iorn jation m MEANS Collection Volume 2 EC Structural Funds Evaluating socio-economic programmes Selection and use of indicators for monitoring and evaluation Volume 2 European Commission MEANS is a programme of the European Commission. It is aimed at improving Methods of Evaluating Structural Policies. The MEANS Programme and Collection was supervised by Phillipe Goybet and Miguel Angel Benito Alonso, Unit for the Coordination of Evaluation in the Directorate for Coordination and Evaluation of Operations of DG XVI Coordination-evaluation@fmb. dg16. cec. be The realisation of the Programme was entrusted to the Centre for European Evaluation Expertise (C3E) under the direction of Eric Monnier and Jacques Toulemonde. Responsibility for the opinions expressed in this book lies solely with the authors of the text: C3E - 13 bis Place Jules Ferry - F - 69006 Lyon - France - c3e@c3e. fr A great deal of additional information on the European Union is available on the Internet. It can be accessed through the Europa server (http://europa. eu. int) and on the Inforegio Website (http://inforegio. cec. eu. int). Cataloguing data can be found at the end of this publication. Luxembourg: Office for Official Publications of the European Communities, 1999 ISBN 92-828-6626-2 © European Communities, 1999 Reproduction is authorized, except for commercial purposes, provided the source is acknowledged. PRINTED ON WHITE CHLORINE-FREE PAPER Printed in Italy Foreword Developing good evaluation practices has become a priority area for the relevant Commission and the effective use of Community to is directly t resources. Indeed, the sound financial management principles which the Commission seeks to promote via its SEM2000 (Sound and Efficient Management) Programme rely, specifically, on the regular evaluation of the Community activities. The Directorates General responsible for structural policy have considerable expe rience in this area. Since 1988 they have made evaluation an in the process of integral part of each stage Community Structural intervention. Thus, the programme development stage includes a prospective evaluation; the implementation stage a strengthening of the monitoring systems and, at the end of the intervention, evaluation establishes what has been achieved in terms of programme impacts. However, the capacity of evaluation to contribute to the effectiveness and transparency of structural interventions rests ultimately on the integrity of the methods adopted to verify the credibility of results. This is why, four years ago, on the initiative for the Directorate General for Regional Policy and Cohesion (DG XVI), the MEANS programme was initiated as a response to the demand for developing evaluation methodo logy. This also include practices combining both general scientific requirements and the specific requirements of Community interventions whilst, at the same time, remaining accessible to a large audience. One of the more tangible results of the MEANS programme has been the publication of this six volume set by a group of independent experts, focu sing on the principal elements of socio-economic programme evaluation. A short summary of this work can be found at the beginning of the present volume. This collection covers the various methodological approaches and innovations that have taken place in the context of structural funds eva luation. It will enable those responsible for evaluation to develop their approach in a flexible and independent manner and for the various programme partners to exercise greater control over the evaluation pro cess. Attending the conferences of evaluation specialists and "clients" convened in Brussels (1995), Berlin (1996) and Seville (1998), I gained some impression of the interest that new evaluation techniques can gene rate. I would stress that the development of evaluation needs to progress in tandem with efforts to incorporate evaluation findings in the decision making process. This implies evaluation as a means of establishing whe ther policies are equitable, effective and relevant to the identified needs. In satisfying these requirements, evaluation should be seen as a valuable instrument affording both Member State and Community authorities the necessary guarantees that Community funds are being deployed in the best interests of the EU citizen. This is a cost worth paying when the goals of Economic and Social Cohesion and the European Union enlargement are at stake. Finally, I wish to extend my thanks to the members of the Committee of Independent Experts established at the launch of the programme. Their expertise and standing have contributed greatly to the quality and relevan ce of the MEANS programme. I therefore thank: E. Chelimsky (USA). ex- President of the "American Evaluation Association"; K. Kennedy (IRL), Director of the "Economic and Social Research Institute" (ESRI); J. R. Cuadrado Roura (SP), President of the "European Regional Science Association"; C. Seibel (F), Director of the "Statistiques démographiques et sociales de l'Institut National de la Statistique et des Etudes Economiques" (INSEE); L. Senn (I), Director of the Department of Economie Regional of the University Bocconi; E. Stern (U. K. ), Director of the unit of evaluation of the Tavistock Institute; A. Sorber (t) (NL), Head of the "Policy Analysis Department of the Ministry of Finance"; L. I. Strom (SV), Statens Institut för Regionalforsking (SIR); L. Tsoukalis (GR. ), professor at the University of Athens; H. Wollmann (D), professor at the University of Humboldt - Berlin and founder member of the "European Evaluation Society". E. Landaburu The MEANS Collection: "Evaluating socio-economic programmes" of European Union The evaluation structural interventions has entered into a phase of maturity, attested by the extension of work carried out by both the Member States and the Commission. Regulations based on decisions by the Council have largely contributed to the development of evaluation practice and have stimulated methodological research. initiated 'by Over the past ten years, the Commission has striven to promote and improve methods for evaluating Structural Funds. The main methodological conclusions have been debated and disseminated as part of the MEANS in 1991. programme Achievements to date provide enough material for a complete and accessible publication intended for a broad public: a six volume Collection covering all aspects of the evaluation of socio-economic programmes. These Volumes are written with a perspective that is broader than the current administrative framework, for this is likely to evolve more rapidly than professional standards. the Commission information on In general, the Collection is aimed at a readership of non- specialists seeking essential the evaluation of socio-economic development programmes. These programmes have multiple objectives, a partnership dimension and a multi-sectoral content - all characteristics which call for suitable evaluation methods and solutions not yet provided by the international technical literature. The complete Collection consists of 6 volumes designed to be complementary and to provide essential for professionals wanting to independently develop their own evaluation practices. Each volume can, however, be read and understood without having to refer to the other volumes in the Collection. recommendations information and Indicators for monitoring and evaluation The first Volume in the MEANS Collection entitled "Evaluation design and management" serves as an introduction both to the Collection and to the evaluation of socio-economic programmes. It enables readers to understand the implications of evaluation and to know what they can and cannot expect from an evaluation at a given time in a particular context. It covers all the elements needed to steer an evaluation project from A to Z: sharing of responsibilities, defining of the aims and method of an evaluation, establishing a schedule and budget, assessing the quality of the work performed and, finally, managing the use of conclusions. The second Volume focuses on the "Selection and use of indicators for monitoring and evaluation". It deals with the subject of indicators used for monitoring and evaluation. It shows how indicators can be used to support policy decisions, improve management and measure the overall performance of programmes. The volume proposes a methodological framework and a practical guide for the use of indicators in these different situations. The third Volume presents a selection of "Principal evaluation techniques and tools" applicable to the evaluation of socio-economic programmes. These tools are taken from relevant scientific disciplines: economics, sociology, management, geography, etc. Each one of the 23 tools is described in the form of an information sheet featuring a general description, the principles of its implementation, a list of its strengths and weaknesses for the evaluation of Structural Funds, an example of its application and a list of references. A number of methods adapted to the evaluation of socio economic programmes are the object of a more detailed description in the fourth Volume entitled "Technical solutions for evaluation within a partnership framework". Several pilot evaluations were implemented within the MEANS programme to design and test the following four innovations, the application of which is described in 777e MEANS Collection detail in this Volume: impact mapping, cross-impact matrix, score cards and multi-criteria analysis. The fifth Volume deals with "Transversal evaluation of impacts on the environment, employment and other intervention priorities" of the European Union and Member States. Among the questions addressed are impacts in terms of equal opportunities between men and women and the competitiveness of SMEs. The Volume provides the elements required for an analysis of those impacts which are not necessarily part of the explicit objectives of all the actions evaluated. Finally, the last Volume is a "Glossary of concepts and technical terms". The compilation of the Collection has necessitated the rigorous standardisation and definition of the terms used. Each of the first five volumes has an index of terms which, along with hundreds of definitions, are brought together in the glossary. Contents Introduction I Terminology 15 The notion of an indicator and its different versions 17 The system of indicators of a socio-economic programme 23 Application to different types of structural interventions 39 II Fields of application How to read the information sheets Indicators for seven domains ■ Transport infrastructure ■ Energy infrastructure ■ Training ■ Tourism ■ Fisheries ■ Research and Technological Development ■ Agriculture and rural development Indicators for five priorities ■ Equal opportunities ■ Environment ■ Information society ■ Competitiveness of SMEs and enterprises in general ■ Urban development Indicators for three major aims ■ Economic development ■ Employment ■ Quality of life III Methods of producing and using indicators Conclusion 9 45 47 49 49 61 69 77 85 93 103 111 111 115 125 135 145 155 155 165 173 179 209 Appendixes Appendix 1 - General bibliography Appendix 2 - Lessons from international experience Appendix 3 - Programme indicators for the 1994-99 period: a state of the art Appendix 4 - Grid for assessing the quality of a system of indicators Appendix 5 - List of examples Index 213 215 219 229 233 237 241 Introduction Introduction International experience Extensive international experience in the use of indicators has existed for a long time. Its origins lie in the success of major macro-economic indicators that currently play a key part in political decision-making and in the media. The role of indicators in the measurement of government deficits in candidate countries for the euro is a clear indication of their importance. In the 1970s, because of the success of economic indicators, many efforts were made to develop social indicators. At present, there is not a single political field in which efforts are not being made to create and quantify indicators at international level. This dynamism mainly concerns fields such as education, science, technology, the environment, health, the quality of life and the information society. Indicators have, for a long time, been associated with statistics since their purpose has been to describe the context of a policy in relation to the context of other regions or countries. In this perspective, the OECD and the OSCE produce interesting series of statistical publications covering many fields. In recent years, 'efforts to construct indicators have also turned to the measurement and management of the performance of public activities. The UK is the most advanced European country in this sense, going as far as to establish direct links between performance and the allocation of public funds. France has also been making significant efforts in this direction. Apart from the famous example of macro-economic convergence criteria, the European Union uses indicators in many fields. Most of these quantified data are available to the public through Eurostat. Science and technology is the main area in which efforts are being made, mobilising a team in Brussels as well as a network of organisations in the Member States. Every two years the full list of Science and Technology (S&T) indicators is published in the form of a report. • Community indicators and practices In the field of Structural Funds, indicators are used in two types of situation: A large number of global indicators are used on a European scale. Most of these have been rigorously defined and the definitions harmonised (e. g. unemployment rate, population density, and disparities in infrastructure). For many years a lot of work has also gone into regionalising these Indicators for monitoring and evaluation statistical indicators. They are used when major political decisions have to be taken, such as the earmarking of budgets and the definition of eligible areas. They are also used to account for the use of Structural Funds in reports to the European institutions (e. g. the triennial "Cohesion Report"). Since the 1988 reform, the Commission has encouraged national and indicators for quantifying programme regional authorities to develop objectives, monitoring programme implementation and estimating the effects obtained. Lists of indicators have been proposed to certain countries by the services of the EC. However, programme managers have generally constructed their own decentralised information systems adapted to their programmes. This freedom of choice, combined with the multiplicity and heterogeneity of programmes, has spawned a multitude of indicators. Many of these are specific to a particular domain of intervention (e. g. number of visits by tourists, number of trainee hours, number of homes with access to a sewerage system). It seems that substantial progress still needs to be made in the use of indicators for programme management. In monitoring and evaluation, efforts to develop quantification must be continued, particularly for indicators related to programme impacts. For the third Structural Fund programming period (2000-2006), stricter methodological supervision, by the Commission, of systems of indicators in fields of intervention is expected. This second Volume of the MEANS Collection thus proposes some ideas for reflection and a practical guide for the use of indicators. Like the other five Volumes, it is written for professionals involved in the evaluation of European Structural Funds, i. e. designers and managers of socio-economic programmes and evaluation teams. Its purpose is to show that it is possible to define, choose and quantify useful indicators in a pragmatical way. Suggestions in this Volume are consistent with the decentralised approach followed until now by the Commission. In other words, the idea is to clarify possible options and not to impose a restrictive system. The autonomy of programme managers should not, however, be the source of wide disparities in concepts and approaches, nor hinder the harmonisation of the main indicators. For this reason, without claiming to impose standard indicators, this manual aims to provide a sound methodological framework that can be shared by all programme managers and facilite interaction between all actors involved in European economic and social cohesion policy. It therefore: Introduction • proposes a coherent and rigorous conceptual framework; • demonstrates the operationality of this framework for the monitoring and evaluation of programmes; • proposes criteria for assessing the quality of systems of indicators; • shows the advantages of comparability and makes suggestions in this respect; • suggests ways of using indicators for managing programme performance; • finally, suggests possible links between the 'micro' level (outputs, results and impacts peculiar to measures) and the 'macro' level (global impacts in terms of cohesion and Community priorities). A large place has been reserved for examples of indicators relating to different fields of intervention of socio-economic programmes. These examples are relevant to programme designers and managers. They have been developed by qualified experts in the relevant fields. They provide the reader with a pragmatic set of indicators suitable for monitoring outputs and results, and for evaluating impacts. They also suggest new routes for more effectively linking information systems to the aim of economic and social cohesion. The last part of this Volume is entirely devoted to the measurement of programme performance. It contains numerous suggestions for developing the use of indicators for this purpose. In particular, it proposes a list of key indicators for developing comparisons and facilitating communication on a European scale. Box 1 - How to use this guide For the non-specialist who is interested in having a broad overview of indicators Part I (Terminology) provides a wealth of information about different types of Indicators and their use within the framework of the Structural Funds. It also provides a standard framework of definitions that should reduce the risk of misunderstandings about II (Fields of application) provides concrete examples of the use of indicators. A quick reading might concentrate on transport a infrastructure, equal opportunities and economic development. Part III (Methods of producing and using indicators) and the appendices can be read by those who wish to supplement the information provided in the earlier parts of the Volume. limited number of relevant information sheets, for indicators. Part instance: For the specialist and non-specialist with an interest in particular fields of application The relevant information sheet(s) in Part II is the first port of call. The numerous examples resulting from the contribution of specialists provide concrete illustrations of the use of indicators in different fields of application. Reference to Part I will then allow readers to develop their knowledge of indicators from a general perspective and establish a standard terminology. Part III and the appendices should be read by those who wish to supplement the information provided in the earlier parts of the Volume. For decision-makers, programme managers, etc. Part III provides an essential starting point for those who are concerned with managing programmes, commissioning evaluations, etc. In the case of thematic evaluations, the relevant information sheet in Part II is also essential reading. For a more in-depth appreciation of indicators and associated issues, a number of appendices are provided, along with a bibliography. Part I has the essential role of facilitating communication by providing a standardised terminology relating to indicators I Terminology Indicators for monitoring and evaluation The notion of an indicator and its different versions The notion of an indicator An indicator can be defined as the measurement of an objective to be met, a resource mobilised, an effect obtained, a gauge of quality or a context variable. An indicator produces quantified information with a view to helping actors of public interventions to communicate, negotiate or make decisions. Within the framework of evaluation, the most important indicators are linked to the success criteria of public interventions. An indicator can be used to measure a fact or an opinion. The resulting measurement is usually approximate. An indicator may be constructed specifically for an evaluation by the evaluation team ('ad hoc' indicator) or taken from a monitoring system. What constitutes a good indicator? An indicator quantifies an element considered to be relevant to the monitoring or evaluation of a programme. For example: "1200 long-term unemployed received training financed by the programme" or "75% of the participants in training courses claim to be satisfied or highly satisfied". A good indicator must provide simple information that both the supplier and the user can easily communicate and understand. This is, however, a necessary but not sufficient quality. The following are examples of indicators that are readily understood: rate of budget absorption; percentage of regional firms assisted; number of net jobs created; and number of jobless in the eligible area. An indicator may have several values, depending on the state of the quantitative variable to be measured. Thus, the unemployment rate indicator may have an initial value, a value for the intermediate objective, a value taken mid-way through the implementation of a programme, etc. Indicators and quantitative data Not all statistical data or other quantitative information are automatically indicators. In this Volume, the term "indicator" applies only to information that programme managers consider as relevant and necessary for them to make decisions, negotiate and communicate. Within the framework of monitoring and evaluation, the most important indicators are related to the Indicators for monitoring and evaluation success criteria of the programme, such as: the state of progress of projects; objectives to be met; beneficial effects to be obtained; quality factors; progress to be made; etc. The measurement given by an indicator may be complete or partial. For example, the indicator "rate of budget absorption" measures all the financial resources used by a programme (this indicator may be calculated for public expenditure at the Community, national and local levels, as well as for private expenditure). In contrast, the indicator "number of kilometres of motorway built" is a partial measurement of output. To obtain a complete measurement, not only would the length of the section built have to be taken into account but also other important elements such as the number of lanes on the motorway or its reference speed. Some indicators measure a well-defined variable such as the "number of SMEs assisted by the programme". In contrast, other indicators apply to abstract concepts and help to define and measure them. That is the case, for example, of the indicator "proportion of businesses assisted, developing new activities in sectors with growing demand". This indicator provides both a definition of sectoral redeployment and a measure of the impact of a programme on it. Indicators constitute an essential part of the information system of a programme, along with other more qualitative forms of information. The information system of a programme produces, notably, operating reports, reviews, progress reports and evaluation reports. Indicators and qualitative data regional authorities This Volume is devoted entirely to indicators, that is to say, to quantitative measurement. It does not cover qualitative measurement, for which the is used. The following statements are examples of term "descriptor" descriptors: their mode of "the management when the programme was implemented" or "the motorway did not spoil the countryside". Nor does this Volume cover scoring. The following statement is an example of scoring: "the environmental impact of the project is -2 on a scale ranging from -3 to +3, representing all possible levels of impact, from the most negative to the most positive". In this Volume, a score is not considered to be an indicator. transformed in Volume 4 of the MEANS Scores and descriptors are presented Collection. These techniques are often used in combination with indicators. 777e notion of an indicator For example, the World Bank uses a scoring system that functions in parallel with indicators, to monitor and evaluate its projects. The British government manages its research grants by means of expert panels that use scoring systems and indicators. This Volume presents several typologies of indicators which are summarised in Box 2. These typologies are consistent with those in the document "Common Guidelines for Monitoring and Evaluation" (European Commission, 1995, Luxembourg: OPOCE). They were refined during discussions with about thirty experts, mainly during a seminar organised in Brussels on 23 January 1997. Box 2 - Main typologies of indicators >■ in relation to the processing of information: Elementary, derived and compound indicators (see below) >- in relation to the comparability of information: Specific or generic indicators, key indicators (see page 20) >- In relation to the scope of information: Context and programme indicators (see page 24) >- in relation to the phases of completion of the programme: Resource, output, result and impact indicators (see page 28) >- in relation to evaluation criteria: Relevance, efficiency, effectiveness and performance indicators (see page 34) >- in relation to the mode of quantification and use of the information: Monitoring and evaluation indicators (see page 35) Elementary, derived and c o m p o u nd indicators This first distinction separates indicators according to the extent to which the information provided has to be processed. An elementary indicator provides basic information on which other indicators can be built. For example, the number of jobless is an elementary indicator which can be used to calculate the unemployment rate (number of unemployed / working population), changes in the unemployment rate, etc. The number of kilometres of roads built, the number of businesses assisted, and the number of beaches complying with standards are all examples of elementary indicators. Indicators for monitoring and evaluation A derived indicator (ratio or rate) is based on the calculation of the ratio between two elementary indicators. For example, an effectiveness indicator can be obtained by dividing two values of the same impact indicator (estimated impact / expected impact). An efficiency indicator can be obtained by dividing the values of two elementary indicators (observed result / budget absorption). A compound indicator is the weighted sum of several elementary or derived indicators. A weighting is attributed to each indicator to reflect its importance in relation to a set of indicators. The indicator considered to be the most important is therefore given the most weight, and the one considered to be least important is given the least weight. The value of the compound indicator is the sum of all the adjusted values of these indicators. For example, the United Nations "quality of life" index combines three elementary indicators: income (GDP), life expectancy and percentage of children in school. In 1986, an index of global infrastructure endowment was calculated by DG XVI for all European regions. This index is the weighted sum of 41 ratios of regional infrastructure endowment. The infrastructure considered ranged from roads (expressed in km/km2) to nature reserves (expressed in ha/inhabitant). Each ratio relates the regional endowment for a given infrastructure to that of the best-endowed European region. Breaking down an indicator All derived or compound indicators can be broken down into different dimensions of the variable measured, as in the following examples: annual budget absorption broken down by category of intervention; turnover of assisted businesses broken down by sector of activity; number of jobless broken down by age-group. The term "breakdown" is used when the indicator is divided into different categories, e. g. : agriculture - industry - service; workers - jobless; men - women. In contrast, the term "disaggregation" is used when the indicator is divided into similar categories, e. g. : distribution of budgets between zones within an assisted region; distribution of the number of trainees between training organisations, etc. Indicators are often quantified at a lower level and then summed to produce an aggregated indicator. Specific and generic indicators, key indicators One characteristic of socio-economic programmes is the diversity of the interventions that comprise them. A single programme often consists of The notion of an indicator several dozen different interventions and two programmes rarely contain the same combination of interventions. This diversity tends to multiply the number of indicators used and make them difficult to compare. It is therefore particularly useful to distinguish indicators in terms of the comparability of information. A specific indicator is used in the case of an intervention and is not intended to be used for comparison. For example, the proportion of trainees belonging to the Catholic minority is a good indicator for a training programme in Northern Ireland, but is of little use in another region or for another intervention. A generic indicator serves to make comparable measurements of sever al different kinds of intervention within the same programme. The com parison is internal and allows the aggregation of data within the pro gramme, in the form of a sum or average. Examples of generic indicators applied to an entire programme are the rate of budget absorption and the completion rate. Some generic indicators apply only to a part of the pro gramme: number of innovations (for all the components which favour technological development); proportion of SMEs among the addressees or recipients (for all the components which benefit businesses); cost per job created (for all the components which directly or indirectly create jobs); percentage of the surface areas of protected zones in the total surface area of zones meriting protection (a generic indicator for actions related to biodiversity, water resources and soil quality). Generic indicators are rela tively difficult to define and to quantify when the components of a pro gramme are heterogeneous. Methods for facilitating this task are set out in the last chapter of this Volume. Key indicators are those which lend themselves to internal comparison between different interventions and to external comparison with other programmes. They can be used to establish points of reference such as average European performance or cases of excellent performance to be emulated. The main challenge in the use of key indicators is to ensure that their quantification in several different programmes adheres to har monised definitions, so that the information produced can be pooled. Key indicators can then be used for aggregation on a European scale. Examples of key indicators proposed in this Volume include the rate of budget absorption, the completion rate, and the cost per job created. Key indicators are treated in more detail in the last chapter. The system of indicators of a socio-economic programme Structural intervention programmes Indicators are used from the beginning of the programme cycle to help in defining territories eligible for European assistance, in analysing the regional context, in diagnosing economic and social problems to be solved, and in assessing the needs which the programme has to meet. At this stage, indicators such as the unemployment rate or disparities between infrastructures play a decisive role. The choice and validation of the intervention strategy constitute the second step in the programming cycle. At this stage the programme designers must define the objectives precisely and quantify them. Indicators are inseparable from quantification and are also very useful for clarifying objectives. For example, strategies for supporting the competitiveness of SMEs often include consultancy services. The objectives of these measures are defined in terms of the number of businesses to receive particular types of consultancy services. The indicator serves not only to quantify the objective, but also to define the expected service. For example, the number of SMEs to be assisted should only take into account large consultancy missions (e. g. 5 days of consulting). Once defined and adopted, the programme is implemented. It is monitored and evaluated at mid-term. Here again, indicators are indispensable for circulating, in a simple and condensed form, information required by programme managers. For example, indicators serve to monitor the pace at which budgets are spent, the extent to which the schedule is adhered to, the proportion of the eligible population reached, the rate of satisfaction of addressees or recipients, the number of jobs created, etc. The programming cycle ends with an ex post evaluation, of which one of. the main functions is to report on the programme results and on the extent to which aims have been achieved. The use of indicators is strongly recommended at this stage in so far as it allows the communication of simple information that is immediately understood by a wide public, e. g. cost per job created or rate of placement of jobless people assisted. Most socio-economic programmes adopt integrated strategies, in other words, they try to solve all the problems affecting a given territory and they Indicators for monitoring and evaluation for instruments territory. This use all available characteristic necessarily entails a multiplication of needs for indicators, which would if the programmes were not highly structured. Programmes financed by the European Structural Funds are usually structured on three levels: lead to confusion intervening in that • the overall programme level to which the global objective is related, e. g. economic development or employment. This level consists of a small number of priority axes (less than ten) which break down the global objective into its main strategic dimensions; • the measures level (one or several dozen), corresponding to the basic unit of programme management. Each measure has its own specific management apparatus. (Actions, which correspond to the smallest homogeneous component in the programme - since each action groups together similar projects - are considered on the same level as measures); • the project level (often a few hundred), which is the implementation unit of the programme, since each project is a point of articulation between the programme and its addressees or recipients. Programme and context indicators Context indicators apply to an entire territory, population or category of population. An example of a context indicator is the level of connection to digital phone lines in an eligible territory. Within the framework of ex ante evaluation, programme intervention may be justified by the backwardness of the region in terms of level of connection. The same indicator may then show, within the framework of an intermediate evaluation, that the region has caught up with others in this respect and that the intervention is no longer justified. In this case, the context indicator does not provide information on the effectiveness of the programme since regional backwardness has been remedied by market mechanisms. A context indicator does not apply to the implementation of a programme and to its effects. It always applies to the entire eligible territory or targeted public, without distinguishing between those that have been reached by the programme and those that have not. In contrast, programme indicators concern only the part or category of the public or the part of the territory that has effectively been reached. Programme indicators try to monitor, as far as possible, the direct or indirect effects of the programme. For example, they measure the extent to The system of indicators which a target population has been reached or the extent to which a lasting advantage has been obtained by direct addressees or recipients. Within the framework of monitoring and evaluation, a programme indicator is able to show that a particular intervention is a success or failure. On the other hand, a context indicator may show that a particular intervention remains relevant or is no longer justified. Programme and context indicators are always related to each other and this relationship can be expressed in the following way: Initial state of the context + All the programme effects + Change in the context due to other factors = Final state of the context Box 3 - Theoretical links between context and programme indicators Exogenous (confounding) factors •fr. Global change in the context (Present situation - Baseline) Context indicators '' /. -' '-—1 Indirect effects of I the proqramme X / / Direct effects of the programme / Programme implementation Programme indicators -k Difficulties in matching programme and context indicators This relationship is easier to describe in theory than to use in practice. A programme always has numerous consequences: positive and negative, direct and indirect, immediate and long-term. Programme indicators are rarely able to measure all these consequences. Moreover, the link between Indicators for monitoring and evaluation context and programme indicators is also problematic because a part of the observed changes in the context are due to confounding factors (also factors"). Techniques for analysing causality are called "exogenous intended to separate from those of confounding factors. In Volume 3 of the MEANS Collection in which these techniques are presented, it can be seen that they are able to provide an estimation but never a complete measurement of impacts. Box 3 shows the theoretical relationship between context and programme indicators, and the complexities of this relationship in practice. the effects of the programme The overall change in the context is measured by means of context indicators. The information provided by a context indicator usually includes a baseline value, also called the initial state, and a current value. Examples of context indicators include: the number of inhabitants of the region connected to a sewerage system (infrastructure endowment); the regional rate of connection to the Internet, as a percentage of the European average (disparity); and the percentage of company directors who consider that accessibility by air is a major problem (needs). Changes which affect the territory or the eligible population are not only explained by the programme, but also by exogenous (confounding) fac tors. Indicators measuring the evolution of these factors help to interpret or to qualify the information given by context indicators, when these show success or failure that may be more apparent than real. The following examples illustrate confounding factor indicators and their use: the natu ral growth of the working population may explain an increase in regional unemployment even though the programme has created jobs; the drop in world prices may explain the decline in traditional local production even though the programme supported marketing and promotion efforts. Even if the link between programme and context indicators is problematic, it is often useful to link them, as the example in Box 4 shows. In the context of an intervention providing consultancy services to businesses to help them develop their exports, it is relatively easy to construct the initial indicators of the intervention. its direct and immediate effects is defined: number of recipient firms, number of hours of consultancy services received, number of export projects concerned by the provision of advice. It is a little more difficult to estimate intermediate effects such as: number of new international contacts generated by the advice that have been concretised given, percentage of commercially. The following effects may be estimated, although with Information concerning these contacts 777e system of indicators difficulty, by means of a survey among company managers: the proportion of annual turnover resulting from customers contacted on the basis of advice given (after two years). Beyond a certain limit, the monitoring of effects by means of programme indicators is no longer credible. For example, it would be futile to ask company managers how many jobs had been created in the months after they benefited from a few hours of advisory services. It is at this stage that it becomes essential to use context indicators. For example, one may know the turnover and number of jobs in all the industrial firms in the region. These indicators may serve to calculate a ratio of average turnover per job and, by extrapolation, to estimate the number of jobs generated by new exports. This example illustrates the fact that programme indicators are most useful for analysing the implementation of the programme and its first effects, and then that the priority progressively shifts to context indicators as the focus moves to longer term effects. The distinction between context and programme indicators will system atically be applied in about fifty examples, throughout this Volume (see page 237 for the list of examples). As suggested in Box 4, the two types of indicator are linked at all stages of the programme, from its implemen tation to its most long-term effects. This distinction therefore corre sponds to the typology presented below and its categories of outputs, results and impacts. For example, there is a point of articulation between the indicator of the capacity of purification plants built with Community funds (programme indicator / category of outputs) and the indicator of regional infrastructure endowment (context indicator / category of out puts). In cases where it is difficult to classify an indicator in terms of the "context / programme" typology, it should be remembered that a pro gramme indicator concerns a closed list of the persons or organisations that participated in the programme or were affected by the programme, whereas a context indicator concerns the entire targeted public, the entire regional economy, and the entire natural environment, including the people or economic agents not affected by the programme. Indicators for monitoring and evaluation Box 4 - Example of links betwe en cont ext and programme indicators Programme indicators (related to the intervention and to its effects) Context indicators (related to the whole eligible area) Number of entreprises having received export advice Number of entreprises in the eligible area ! Number of new international contacts % of exporting entreprises J Exports generated Total exports 1 r Jobs resulting from generated exports Total jobs Average turnover per job Resource, output, result and impact indicators A programme mobilises resources (financial, human and institutional) with a view to achieving a global objective (socio-economic cohesion). To monitor and evaluate the programme, it is necessary to break down its global logic and distinguish a series of related objectives, as shown in Box 5. This typology is inspired by the logical framework technique used for the past few decades in the field of development aid. The indicators are classified into five categories: resource, output, result, specific impact and global impact. This typology will be used frequently throughout this Volume. This box shows that the different levels of objective are related to different categories of people (or organisations) concerned by the programme. It is important to clearly define these categories. 777e system of indicators Box 5 - Definition of indicators b) ι level of objective Level of objective Type of indicator Resource {input) Definition Key actors Financing authorities and operators Means made available by financing authorities and used by operators for their activities Operational objective Output Product of the operators' activity Operators Immediate specific objective Result (immediate outcome) Immediate effect for direct addressees or recipients Direct addressees or recipients Sustainable specific objective Specific impact (sustainable outcome) Strategic objective Aim Global impact (outreach) Sustainable effect for direct addressees or recipients Global effect for the entire population concerned (direct and indirect addressees or recipients) Direct addressees or recipients Direct and indirect addressees or recipients The programme funders are people or organisations which decide on its implementation and allocate the required resources. The operators are those who use the resources closest to the field. They are the last link in the chain of implementation of the programme. They are at the source of information which feeds the monitoring system. Examples of operators are: a local authority which distributes direct support to businesses; a group of firms which distributes aid among its members; a private com- ' pany which constructs and manages a telephone network; a private com pany which builds a road for a regional authority; a training institute which trains people funded by the programme. The financing authorities and operators make up the category of programme actors. Irrespective of the level on which they intervene, the programme actors must account for their activity in a regular and complete way in the form of progress reports. Indicators for monitoring and evaluation Broadly speaking, the programme addressees or recipients consist of all the people or organisations that receive benefits or are affected directly or indirectly by the disadvantages of the programme. The reason d'être of an action is to produce a positive change for addressees or recipients1. Some individuals or organisations are direct addressees or recipients in so far as they receive or personally use the services or support provided by the programme. Examples of direct addressees or recipients are: the motorist who uses a road built during the programme; the family that uses a telephone network improved by the programme; the firm that received an investment subsidy; the trainee who benefited from subsidised training. Indirect addressees or recipients are individuals or organisations that are affected by the programme without having participated directly in its activities. Indirect addressees or recipients may be affected through a direct addressee or recipient, e. g. the firm that employs a person trained during the programme. Indirect addressees or recipients may also be affected without an intermediary, e. g. inhabitants of a neighbourhood next to a rundown area that was renovated during the programme, or people sensitive to the deterioration of their natural environment, who live near a motorway. The strategic objective of a programme is to provide its direct and indirect addressees or recipients with an overall advantage. In some cases certain people or organisations may, however, be disadvantaged. The negative effects which these addressees or recipients suffer are usually unintended (e. g. bankruptcy of a local firm due to the distortion of competition in favour of an assisted firm). The disadvantages may also be related to the objectives of the programme (e. g. substitution of a long-term unemployed person for another unemployed person). 1 Because programme addressees or recipients do not always gain from the programme, the term "beneficiaries" has not been used to identify them. Moreover, the Commission has chosen to use the term "beneficiary" in another sense. The term "final recipient" is used in the regulatory texts (and measures for financial execution) of the Structural Funds to denote the public or private operators responsible for commissioning projects or for the distribution of public aid (see article 9§1-J of the proposed regulation on general measures for Structural Funds, Official Journal, 9/6/1998). The final recipient has to submit regular and systematic reports, and is therefore the programme operator in the sense defined in this Volume. In contrast, certain evaluation manuals define beneficiaries as the individuals, groups or organisations which receive programme services or assistance, that is to say, direct addressees or recipients in the sense used in the present Volume. irte system of indicators Box 6 - Implementation of the programme and its effects Implementation of the programme Social, Natural and Economic Environment Actors Addressees/recipients Funding authorities J Operators Direct addressees /recipients Among the people and organisations concerned by the programme, there is a basic difference between the actors and the addressees or recipients. The actors are within the operational system of programme implementation, whereas addressees or recipients are outside this system. As shown in Box 6, the boundary is that which separates the programme from its economic, social and natural environment. This boundary separates the operators from the direct addressees or recipients (it may, however, occasionally happen that operators are also simultaneously addressees or recipients - e. g. in the case of investments in roads). These two categories play a decisive role in the definition of resource, output, result and impact indicators. Resource indicators provide information on the financial, human, material, organisational or regulatory means used by operators for implementing the financing programme. Resources are the joint responsibility of the authorities, which allocate them, and the operators who implement them. Most resource indicators are regularly quantified by the monitoring system. Examples of resource indicators include: the total budget (quantity of rate); resources); annual budget absorption percentage of expected overspending; percentage of European financing in (resource absorption Indicators for monitoring and evaluation the total public financing; number of people working on the implementation of the programme; number of organisations involved in the implementation. Output indicators represent the product of the operators' activity. More precisely, an output is considered to be everything that is obtained in exchange for public expenditure. Two examples in the field of SME consultancy services can be used to demonstrate the principle of an output, and help to distinguish an output from a result. Firstly, an operator might receive a fixed sum of money to finance the setting up of a consultancy service for SMEs. In this instance, the expenditure has "bought" the establishment of a consultancy service, which is considered as the output. On the other hand, an operator might be allocated a budget of 400,000 euros for a SME consultancy project planning to supply 5,000 hours of consultancy services. However, if the project were to deliver only half of the planned hours of services, the operator would only be paid 200,000 euros. In other words, if an output is not realised, the support is withheld. The output in this case would therefore be the 2,500 hours of consultancy services effectively delivered. Outputs are under the entire responsibility of operators who have constantly to report on them through the monitoring system. Examples of output indicators include: kilometres of roads built; progress rate of the building of a road; hectares of urban wasteland rehabilitated; capacity of purification plants built; number of trainees whose training was paid by the programme; and percentage of this training of which the quality is certified. Result indicators represent the immediate advantages of the programme (or, exceptionally, the immediate disadvantages) for the direct addressees or recipients. An advantage is immediate if it appears while the addressee or recipient is directly in contact with the programme. The full results may be observed when the operator has concluded the action and closed off the payments. Since result indicators are easily known to the operators, they are generally quantified exhaustively during monitoring. Result indicators provide information on changes which occur for direct addressees or recipients, e. g. time saved by users of a road; reduced rates for telephone calls; qualifications earned by trainees; new tourist activity generated by a farmer; use of new productive capacity created by a firm; and the satisfaction of businesses which have received consultancy services. It is at the time that addressees or recipients receive support or programme services that results can be quantified. Either direct measurements are made 777e system of indicators (e. g. by counting the number of trainees recruited during their training) or the direct addressees or recipients are asked to state the advantages they have obtained (e. g. by means of a questionnaire at the end of a consultancy mission). Impact indicators represent the consequences of the programme beyond its direct and immediate interaction with the addressees or recipients. An initial category of impacts group together the consequences for direct addressees or recipients of the programme, which appear or which last into the medium term (specific impacts), e. g. traffic on a road one year after it is opened; the placement rate of trainees after twelve months; sustainable jobs created in an industrial plant built with programme support; and the survival rate of businesses created with programme support. Some impacts are unexpected (spin-offs) but indicators are rarely created for unexpected impacts. A second category of impacts consists of all the consequences that affect, in the short or medium term, people or organisations that are not direct addressees or recipients. These impacts may be similar (e. g. improvement industrial of the quality of life for people living near a rehabilitated wasteland; improvement in the quality of beaches near a new purification plant). They may, in contrast, spillover to affect people or organisations far from the programme, as in the case of macro-economic impacts. The mechanisms of impact propagation are often separated into two categories: market effects (e. g. impact on suppliers or sub-contractors of the assisted firms) and non-market effects (e. g. positive impact of the improved image of the region or negative impact of a deterioration in the environment). Because non-market effects or externalities are not reflected in the price system on which individual socio-economic actors largely base their private decisions, and because these decisions have economic consequences for other actors, it is particularly useful to take these effects into account in the context of a public programme. Because of the time lag or their indirect nature, impacts cannot easily be ' known to operators during their daily management of the programme. Impact indicators are therefore quantified from time to time only, usually during evaluations. One way of establishing impacts is to carry out a survey of direct addressees or recipients, for example a year after they have left the programme. The questions asked might concern facts (e. g. how many new jobs have been created since the support was obtained?) or opinions (e. g. how many jobs would have been lost in the absence of support?). Indicators for monitoring and evaluation In this Volume, an "effect" is taken to mean any change caused by the implementation of the programme, whether this change is direct or indirect, immediate or long-term. Thus, effects include results and impacts. In all cases, surveys can be used to observe the apparent effects (gross effects) and not the real effects of the programme (net effects). The gross effect is almost always over-estimated, either because observations are partly affected by confounding factors, or because opinions gathered are over- optimistic. In particular, people who answer questionnaires tend to forget or the overlook what they would have done without programme. Sometimes an addressee or recipient has already taken a final decision (e. g. to invest, go on a training course, start a business) when he or she learns that the programme can be helpful. In this case, the addressee or recipient takes advantage of the support (deadweight effect), but the real effect of the programme is nil. Deadweight effects must, if possible, be estimated and subtracted from gross effects. the support of To obtain an estimation of net effects, it is also important to take into account the substitution effect (in the case of interventions aimed at individuals or groups of individuals) or the displacement effect (in the case of interventions aimed at geographical areas). Substitution effects occur when a result is obtained in favour of a direct addressee or recipient at the expense of a person or organisation that is not eligible for the public intervention. When this effect is unwanted, it is considered as negative and subtracted from gross effects. Displacement effects occur when a result obtained in favour of addressees or recipients is accompanied by effects on non-addressees or non- recipients within the programme area. In general, displacement effects are not desired and are subtracted from gross effects. Relevance, e f f e c t i v e n e s s, e f f i c i e n cy and performance indicators The most useful indicators for evaluation are those associated with a criterion or judgement such as relevance, effectiveness or efficiency. Relevance indicators relate the programme objectives to the needs that have to be met. For example, the number of places for trainees that the programme can provide, in relation to the number of long-term unemployed in the region; the number of planned consultancy missions, in relation to the number of regional firms that have never exported. The system of indicators Effectiveness indicators relate what is obtained to what was expected. An effectiveness indicator can therefore be calculated by dividing two values of the same output, result or impact indicator, that is to say, the observed value at a given date and the objective initially set. When talking of effectiveness, it is preferable, for the sake of clarity, to specify whether the reference is to the effectiveness of outputs, results or impacts. Examples of effectiveness indicators are: outputs exceed the objective by 5%; the number of businesses created amounts to 85% of the objective; the placement rate of trainees after a year is 10% higher than expected. indicator is therefore the ratio of two Efficiency indicators relate what was obtained to the resources mobilised. An efficiency indicators: the measurement of what was obtained / the measurement of resources mobilised to obtain it. The calculation of efficiency can be based on an output, result or impact indicator. When talking of efficiency, it is preferable, for the sake of clarity, to specify whether the reference is to the efficiency of outputs, results or impacts. The efficiency of outputs is often called the unit cost. The efficiency of impacts is often called the cost / efficiency ratio. Examples of efficiency indicators are: the average cost of a kilometre of motorway built was 5 MEuro; the average cost of training a person who had been out of work for an extended period of time was 2 KEuro; the average cost of a net job created was 30 KEuro. Performance indicators, according to the definition proposed in this Collection, encompass the effectiveness and efficiency of outputs, results and impacts. In fact, the word "performance" is used in many different ways. In this context, there is a large overlap between the notion of a programme indicator and that of a performance indicator. In some contexts, the term performance is used within the framework of a "reorientation of the administration towards results, service quality and user satisfaction" (USA - Government Performance and Result Act). In other contexts, the term performance applies mainly to resource management and outputs. That is the case, for example, at the World Bank (World Bank, 1997 Operation Evaluation Department, Lessons and Practices, 1997 / 10). Here again, because the vocabulary is not stable, performance needs to be redefined each time reference is made to performance indicators. Monitoring and evaluation indicators Monitoring indicators are quantified by operators when they implement the programme or when they are in contact with the addressees or recipients. Indicators for monitoring and evaluation By means of new communication technologies, it is now possible to construct a monitoring system that provides information on resource consumption, output rates and numbers of addressees or recipients (trainees, assisted firms, etc. ). It is even possible in many cases to maintain a data base with the identity of the direct addressees or recipients (e. g. name of trainees, telephone number of the contact person in the assisted firms). It is also easy to obtain statements by the addressees or recipients when they are in contact with the programme (e. g. investments made by the assisted firms; trainee satisfaction at the end of their training courses). The operators may also be asked to quantify the impact indicators during monitoring, particularly with respect to sustainable impacts on direct addressees or recipients. It is technically possible to systematically telephone or write to all direct addressees or recipients, for example a year after their participation in the programme. In practice, it may happen that monitoring reinforced to this extent provokes rejection by the addressees or recipients who see it as being bureaucratic, or by the operators who consider that the workload involved is excessive. By wanting to obtain too much information through the monitoring system, there is the risk that the quality and reliability of this information will be reduced. The example in the box below shows the limits of the collection of exhaustive information among addressees or recipients. Box 7 - Collection of information from addressees or recipients: monitoring or evaluation? Ireland - Objective 1 Human Resources Operational Programme, 1994-99 A Human Resources Operational Programme was set up in the Irish Republic for the period 1994-1999. Many private and public training organisations took part in the programme. Initially, each organisation was required to indicate the number of people that had found jobs after the training course. To fulfil this obligation, the organisations concerned conducted exhaustive mail or telephone surveys among their trainees. After this system had functioned for a while, the programme evaluation unit noted that the data were neither homogeneous nor reliable. In particular, jobs were measured at any time between 8 and 18 months after the training course. Moreover, the system was cumbersome and bureaucratic. It was decided to abandon the exhaustive collection of data and to quantify the placement rate indicator by means of surveys organised on the scale of the programme. training organisations complained that the 777e system of indicators Impact indicators are essentially evaluation indicators. Evaluation obtains information on impacts by means of surveys or in-depth studies. Collection techniques use sampling, which makes it possible to limit the number of people questioned and to avoid the impression of bureaucracy. Moreover, evaluation is rarely carried out by the operators and so does not add substantially to their workload. Monitoring and evaluation indicators can also be distinguished by their implications on the sharing of responsibilities. Monitoring indicators and, in particular, resource and output indicators, enable operators to report on the use of resources allocated to them and on the activities for which they are fully responsible, e. g. building facilities without overspending or exceeding deadlines. Result indicators are used either for monitoring or evaluation, depending on the degree of decentralisation adopted in the programme management. If the programme is highly decentralised (management by results), the operators can and must constantly adjust their activity in relation to the results obtained. For example, they may notice that the advice given to firms is not satisfactory, and decide to make extensive amendments to the system of support for consultancy services. In this case, the operators attentively follow result indicators in order to be able to adjust the programme as necessary and fulfil their management responsibilities. In contrast, if the operators are not free to adjust the rules of implementation, they have no personal interest in following result indicators which will then be used for evaluation. In practice, a spectrum of intermediate situations co-exist between these two models. Information provided by evaluation indicators corresponds to factors for which the operators are not fully responsible. For example, the operators cannot be expected to react in real time to impact indicators. Evaluation indicators are intended for a broad audience, including financing bodies' and operators. They are used relatively infrequently to make decisions which lead either to changes in the programme so that objectives can be achieved more satisfactorily, or to changes in objectives so that experience can be taken into account. Box 8 specifies the role of the different types of indicator from the point of view of the exercise of operators' responsibility. Indicators for monitoring and evaluation Box 8 - Indicators and operators' responsibilities Resources —*■ Outputs —► Results —*· Impacts Operators. Are directly responsible for outputs (monitoring). are indirectly responsible for results since they can be observed. do not control the impacts (evaluation) Application to different types of structural interventions Application to infrastructure, services and subsidies When the programme produces physical outputs such as infrastructure, the four- category typology can easily be applied. Thus, in the first example in Box 9, the operator develops an industrial site and the firms that establish themselves on this site are the addressees or recipients. The outputs are clearly limited to what is under the operator's responsibility (the surface area developed). The results represent the advantages of the development, from the addressees or recipients' point of view (the attractiveness of the industrial site as compared to other competing sites). The impacts appear at the same time as the addressees or recipients' demand (progressive occupation of the site by firms). Box 9 - Exar nples of indicato -s in the case of infrastructure Resources Outputs Results Impacts Industrial site Road between A & B Rehabilitation of an urban wasteland Cost of the development Surface area developed Attractiveness of the site compared to neighbouring sites Surface area bought by businesses after 1 year Evolution of employment in the businesses Cost of construction Cost of the project Surface area rehabilitated Evolution of the number of inhabitants (and their socio- professional composition), within a radius of 1 km Percentage of inhabitants of the town who want to continue living there (context) Length built Gain in accessibility Number of vehicles using the road after 1 year Percentage of regional businesses satisfied with their accessibility (context) Indicators for monitoring and evaluation In the second example, the operator who has built a road (output) can immediately know what the advantage is for the addressees or recipients in terms of accessibility and time saved (result). It will, however, be necessary to wait a year or more before counting the number of vehicles using the road. An expensive survey will then have to be launched to establish how many passengers travel in the vehicles using the road, the origin and destination of their journeys and the changes caused by use of the road (impacts). The third example is the case of a municipality which was helped to rehabilitate an industrial wasteland (output). This example is more difficult to treat because there are no direct addressees or recipients. The inhabitants of the town are indirect addressees or recipients and the operator has no personal contact with those who benefit from the project (e. g. inhabitants of neighbouring areas, passers-by). It is therefore difficult to construct result indicators. By convention, it is proposed that inhabitants situated within a radius of 1 km are considered as direct addressees or recipients. Beyond this radius, the project improves the overall quality of life of all the inhabitants of the town who will be more satisfied with life there and less tempted to move away (impact). Box 10 - Examples of indicators in the case of services Resources Outputs Results Impacts Training of the long-term unemployed Budget spent on training Number of days of training financed Number of trainees who have gained a qualification Number of trainees who found work within a year Creation of new training courses in an emergency industry Budget spent Number of places offered on the new training courses Percentage of places offered on training courses in growth sectors Number of trainees recruited into the new industry within a year Consultancy services to SMEs Budget spent on consultancy services Number of hours of consultancy services financed Rate of satisfaction of SMEs who received consultancy services Percentage of SMEs who claim to have increased their activity within the first year, owing to the consultancy services The four categories become more difficult to distinguish when the programme provides immaterial services (e. g. training, consultancy). Not only does the Application to different types of structural interventions term "physical outputs" become inappropriate because the service is immaterial, the very notion of an output becomes problematical. It is not possible to "produce" the service independently of those who receive it. The production of the service (output or supply) therefore becomes practically synonymous with the fact that the addressee or recipient benefits from it (result or demand). In the case of a programme providing services, it is logical that outputs and results merge. In this Volume, however, the distinction has been retained, on the basis of the following convention: the service financed by the operator is considered to be an output. The category "results" is limited to other immediate advantages gained by the addressee or recipient. This definition applies as indicated in Box 10. Box 11 - Examples of indicators in the case of subsidies Setting up of a farm Laying up of a fishing vessel Creation of new businesses Resources Amount of subsidy Amount of subsidy Amount of subsidy Outputs Results Impacts Amount of investments by farmers who set up farms Tonnage of fishing vessels laid up Number of businesses created Percentage of farmers who set up farms and comply with a regional quality label Percentage of people who formerly worked on fishing vessels and who are now unemployed Number of jobs generated within a year on the farms created Reduced catches of threatened species of fish (T/year) in the fishing area concerned Gross value added created after one year on the new farms Percentage of businesses created in high tech sectors Number of jobs created within one year in the new businesses Gross value added in the new businesses within one year The typology in four categories becomes even more difficult to apply when the programme offers subsidies. The reference here is not to a subsidy attributed to a programme operator (e. g. funds allocated to a local authority for creating a purification plant or to a training organisation for providing courses), but to Indicators for monitoring and evaluation subsidies attributed to direct addressees or recipients. In this case, there is complete simultaneity between the resource (the programme absorbs the amount of the subsidy), the output (the operator grants the subsidy) and the result (in exchange for the subsidy, the addressee or recipient invests in a project and obtains immediate advantages). In the case of a programme delivering subsidies, it is logical that the concepts of resources, outputs and results tend to converge. In this Volume, the following language has been adopted: resources are subsidies granted; outputs represent key elements in the subsidised project (those which must absolutely be obtained if the subsidy is to be paid, e. g. the actual creation of a business or the maintenance of an existing business); the results are all other immediate effects on the addressee or recipient. These conventions apply as in Box 11. ■ Application to d o m a i n s, priorities and aims Part II presents a series of systems of indicators in fifteen "fields" covered by the Community Structural Funds. The fifteen fields do not correspond to the classification of Structural Fund interventions determined by the Commission2. If this classification had been adhered to, the Volume would have had the following structure: Productive environment Agriculture, forestry and rural development Fishing Support for large businesses Support for SMEs and the crafts Tourism Research and Technological Development (RTD) Human resources Support for people Support for organisations and systems Basic infrastructure Transport infrastructure Telecommunications and information society infrastructure Energy infrastructure Environmental infrastructure Development and rehabilitation Health infrastructure 2 This typology is taken from an internal memorandum of the Commission presenting a project to codify fields of Structural Funds interventions. Application to different types of structural interventions The European Commission's classification creates mutually exclusive sectors of intervention. In contrast, the fields studied in this Volume overlap each other, the notion of field used here being similar to that of a "theme" used in Volume 5 of the MEANS Collection. Certain fields correspond to sectors of intervention similar to the categories of the Commission, whereas others correspond to aims of the Structural Funds. Other fields have an intermediate status, some of them corresponding with the Commission's policy priorities. The first series of fields focus on the outputs of Structural Fund interventions in seven domains: transport, energy, training, tourism, fisheries, research and technological development, and agriculture and rural development. Domains of intervention are situated upstream in the chain of causality in relation to the socio-economic impacts they are intended to produce. The relevant indicators in this category tend therefore to focus on the outputs and results of interventions. As for indicators at the impact level, these concentrate on specific impacts, i. e. those impacts arising earliest in the chain of causality and which can be attributed to the results of interventions. The second series of fields is devoted to five themes, most of them being Community priorities: environment, information society, competitiveness of in general, equal opportunities and urban SMEs and enterprises development. Themes tend to be situated further along the causal chain than domains. In many instances a theme is linked to a mix of interventions, some of (i. e. telecommunication networks relate to the theme "information society") and some related to the theme at a further stage in the causal chain (i. e. support for new high-tech enterprises). the output stage theme at related them the to Finally, three fields reflect the overarching rationale of Structural Funds interventions of promoting economic development, employment and the quality of life. Indicators in this category therefore provide information on global impacts. As shown in Box 12, this Volume attempts to establish backward linkages between global impact indicators, specific impact indicators, results and output indicators, and even resource indicators. These attempts provide a fresh and innovative way of dealing with the micro-macro problem, i. e. in linking the daily management of operations to the global aim of the policy. Indicators for monitoring and evaluation Box 12 - Structure in fourteen fields: domains, priorities and aims transport, energy, training, tourism, fisheries, RTD, agriculture and rural development environment, competitiveness of enterprises and SMEs, equal opportunities, urban development, information society economic development, employment, quality of life i / / Domains A , outputs results specific impacts global impacts Fields of application How to read the information sheets Each information sheet presented in this section is constructed to a standardised format: Introduction An introduction briefly discusses the field of application within the framework of the Structural Funds. Interventions within the field The discussion of indicators for interventions within the field of application follows the chain of causality from resource indicators through to impact indicators. This is followed by a number of concrete examples of interventions presenting output, result and impact indicators. Programme indicators are systematically supplied at all levels. At the impact level they cover both impacts arising directly from the results of interventions and intermediate impacts that link interventions to global impacts arising at the macro level, which are reflected by the appropriate context indicators. While programme and context indicators are linked at all stages along the causal chain, the key role of context indicators is to provide information about the evolution of the overall macro-economic situation. However, given the usefulness of the information provided by context indicators, whenever possible this type of indicator has been included at the output and result level. Other interventions having an impact in the field An intervention in one field often has an impact in another. This section examines the possibility of interactions between fields, giving examples where relevant. Availability and collection of data This section tackles issues associated with data collection within the specific context of the field concerned. Key indicators A number of key indicators are provided that aggregation and comparison. lend themselves to Indicators for monitoring and evaluation Bibliography For readers who wish to deepen their knowledge of a particular field. Indicators for seven domains Transport infrastructure in the field of transport infrastructure Structural Fund interventions encompass roads, bridges and motorways, railways lines and stations (either inter-city or suburban), subways, airports, ports, intermodal centres, etc. Interventions involve construction as well as upgrading. Eligible projects are largely limited to infrastructure and do not usually include mobile equipment like trains, ferries and aeroplanes. Major funding of transport infrastructures is channelled through the ERDF in Objective 1 areas, often in parallel with EIB loans. Interventions take various forms: large projects, operational programmes (transport specific) or transport measures within regional programmes. Transport infrastructure has a number of specific characteristics, which shape the systems of indicators used to evaluate interventions in the field. Projects are usually large and relatively few in number, and are implemented over several years. Recipients directly use some projects (roads), whereas others (railways) need an operator and complementary investments before they reach their end users. Finally, a transport infrastructure project is part of a network; only the network provides benefits for the users, not the project by itself. Interventions within the field Resource indicators While the need to monitor the cost of programme and project implementation is universal, the risk of overruns in terms of budget and schedule of large infrastructure projects makes the following-up of project execution particularly important. At the programme level, resources can be efficiently monitored with one single indicator: "compliance with project cost" which can be defined as the "latest estimate of the total cost of the project in' proportion to the cost that appeared in the programming document". The progress of project execution can be followed by comparing planned and actual project duration, and planned and actual project costs (See Box 13). Output indicators Without prejudice to more detailed levels of information needed in lower levels of management, the information needed for monitoring purposes Indicators for monitoring and evaluation only needs to include the "rate of completion". In order to compute the "rate of completion", large projects must be broken down into many small tasks, every task being weighted with its planned cost. The "rate of completion" is the weighted percentage of completed tasks. It is easy to compute a weighted average "rate of completion" of all infrastructure projects within a programme. A drawback with using the rate of completion as an indicator is that its meaning comes very close to that of an indicator of budget absorption, at least while there is no cost overrun. In order to better reflect outputs, it is worthwhile computing an indicator of "compliance with project duration". To facilitate judgement and comparisons, it is essential that project duration be presented with both initial and current values. And it is also very important that time and money be presented side by side, since a project may involve overspending on cost overruns and underspending due to delays, the cost balance being close to nil. In this case, cost dimension alone would falsely suggest that all is running well (see Box 13). Box 13 - Follow-up of project execution Duration-Plan Duration-Current Cost-Current Cost-Plan <d " ιΛ ■ ■ s ■ S \J \; υ V ^l \) -rJ l> 3 10 20 30 40 50 60 % of Realisation Π Year Begin Now In this example the project, which was well behind schedule at the start of the year, has almost caught up with the planned level of realisation (upper two groups of histograms). The cost overrun observed at the start of the year has also been slightly reduced as the project progressed through the year (lower two groups of histograms). Transport infrastructure indicators can be usefully related to the stock of regional Output infrastructure. New roads and railways are likely to increase regional endowment measured in terms of km/km2 or km/inhabitant. • Result indicators On the road, the majority of travellers use the infrastructure with their own equipment (cars). As for other modes, the addressees or recipients need the services of transport operators, and are thus dependent not only on the quality and capacity of the infrastructure, but also on the array of services made available by those operators and on how well their requirements are met by them. This fundamental difference has consequences not only for the indicators that have to be selected for evaluation of investments in the different modes, but also for the speed and geographical diffusion of the benefits from any particular project. This is because new road infrastructure is available from the first day it is open to service and is easily integrated into the remaining road network. On the other hand, other modes frequently require additional investments to be made by the operators who may have to buy additional and improved equipment, reorganise their services and market them. Impact indicators The initial specific impacts of interventions in this field are associated with the traffic flow on the new infrastructure. Actual users of the new infrastructure can be counted at the earliest one year after opening. By this date, the end-users can also be the subject of studies in terms of travel purpose, travel time saving, travel cost, etc. Other impacts that have to be considered are related to the balance of inter-modal competition. The opening of new transport infrastructure is likely to change the market shares of the various modes. But there are also many instances in which the case of complementarity arises, through which additional flows by one mode in a section of the network will be transferred to another mode to cover another part of the journey within the same logistical chain. The improvement of regional dynamism and competitiveness remains the infrastructure foundation of the existence and dimension of transport projects within the Structural Funds. Many studies have shown that these impacts are always very difficult to assess. Sometimes, it makes sense to apply economic impact indicators to the transport system as a whole. On the contrary, it is often meaningless to apply economic impact indicators to a single project. Indicators for monitoring and evaluation The increasing sophistication of some types of transport infrastructure has lead to a growing concentration of costs, which can result in stronger discontinuities of provision and inequity in terms of accessibility. Evaluation must therefore consider not only the gains for those who benefit from the use of improved services, but also issues of distribution arising from direct and from indirect effects on persons and regions not benefiting interventions. Another area of significant external impacts of transport investments is the environment. Unfortunately, a lot of attention is given to the impacts of infrastructure construction but much less to the impacts of the functioning of transport systems and of policy decisions (or inaction) that may also have a very strong environmental impact. A specific problem arises about how short term construction jobs should be counted. Since they are part of the human resources invested in the implementation of the programme, they should be theoretically counted at the resource level. This may, however, lead to problems of double counting since the creation of construction jobs is, in practice, estimated at impact level by most evaluation teams. Transport infrastructure Example 1: New section of motorway connecting A and Β Programme Indicators (Related to the Intervention and its effects) Context indicators (Related to the assisted area) Compliance with project duration Rate of completion km of new motorway km of new lanes km of motorways per million inhabitants In the area (endowment) O u t p ut i n d i c a t o rs Progress Quantity Result indicators Speed E. S. S. (1)betweenAandB Average E. S. S. to and from all relevant urban centres in the area Impact indicators Traffic flow Time Saved (3) Safety Transport system Indirect economic effect Traffic flow(2) of vehicles using the new infrastructure after one year Traffic flow in the area (vehicle χ km / year / inhabitant) Total journey time saved by users (hours χ vehicles χ average number of passengers per vehicle) after one year Number of traffic accidents(4) on the motorway after one year Traffic accidents in the area (number / year / Mio inhabitants) % traffic between A and Β using the new infrastructure Environment Number of houses suffering from traffic noise (6) Hectares of natural sites disturbed % of managers in the area who declare that road accessibility is a major constraint for their flrm{5) Number of dwellings in the area Hectares of natural sites in the area (1) E. S. S. stands for Equivalent Straightline Speed and measures the ease of access from one point to another, irrespective of the distance between those points and of the path taken. It Is computed by dividing the straightline distance between the two points by the duration of the quickest possible trip between them. (2) Traffic flows are expressed in vehicles per average day. When possible, one truck is counted as two cars. (3) Time saved should be measured In relation to both Interurban and intraurban journeys. (4) Accidents are expressed as the number of accidents involving injuries per year. When possible, accidents involving deaths are counted as double. (5) Accessibility constraint is quantified through a survey among regional managers. It is expressed as the percentage of respondents who agree with the sentence 'road (rail, air, sea. ) accessibility ¡s a major constraint for the development of my organisation". (6) Houses suffering from traffic noise must be counted in accordance to national levels of acceptable noise. The indicator applies to the vicinity of both the motorway and the other roads joining A and B. Indicators for monitoring and evaluation Example 2: Improvement of a railway line between A and B, bringing it up to the high speed range Programme indicators (Related to the intervention and its effects) Context indicators (Related to the assisted area) Output indicators Progress Quantity Result indicators See Example 1 km of high speed railway Speed E. I. S. S. (1 ) between A and Β km of high speed railway per million inhabitants Average E. I. S. S. between all relevant urban centres in the area Impact indicators Traffic flow Time saved Transport system Indirect economic effect Traffic flow(2) of passengers between A and Β after one year Traffic flow to and from area (passengers χ km/year) Total journey time saved by users (hours χ passengers) after one year % traffic between A and Β fully exploiting the new infrastructure % of managers in the area who declare that rail accessibility is a major constraint Environment See Example 1 See Example 1 (1) E. I. S. S. stands for Equivalent Interval Straight-line Speed, an easily transferable indicator which is intended to measure the ease of access from one point to another, in a discontinuous service, irrespective of the distance between those points and the path taken. It is computed by dividing the day into intervals of a width judged adequate for the distance in question (e. g. 2 hour intervals), then computing the best E. I. S. S. (or departures within each interval, and then averaging those E. I. S. S. results over the relevant period of the day (e. g. from 7:00 to 22:00). If there is no departure in a certain interval, the corresponding E. I. S. S. is taken as zero and Is thus entered into the average computing process. The computation of E. I. S. S. values must take into account not only the actual departure times but also transfer possibilities and timetables, whenever we consider the connection between two points not enjoying a direct connection (at least with a departure in the interval being considered). (2) Traffic flow is expressed in passengers / year. Transport infrastructure Example 3: Expansion of a regional airport Output indicators Progress Capacity Result indicators Destinations Speed Programme indicators (Related to the intervention and its effects) Context indicators (Related to the assisted area) See Example 1 Maximum number of passengers per year Maximum number of passengers per year per million inhabitants Number of destinations with at least ten direct flights a week(l) E. I. S. S. by air between the closest urban centre and the most relevant destinations (direct + indirect connections) Average E. I. S. S. by any means between all urban centres in the area and the most relevant destinations (direct + indirect connections) Cost for users Changes in airport charges Impact indicators Traffic flow Indirect economic effect Air traffic flow(2) in the regional airport after one year Air traffic flow to and from area % of managers In the area who declare that air accessibility is a major constraint Environment See Example 1 See Example 1 (1) The number of destinations served by a regular weekday service of at least two flights per day. (2) Air traffic flow is expressed in number of passengers and tonnes of freight per year. Indicators for monitoring and evaluation Example 4: Improvement of a container terminal in a sea port Programme indicators (Related to the intervention and its effects) Context indicators (Related to the assisted area) Output indicators Progress Capacity Result indicators Waiting and turnaround time Punctuality Destinations Handling time I m p a ct i n d i c a t o rs Traffic flow Indirect economic effect See Example 1 Maximum number of containers lifted by cranes per hour and per annum Waiting time of ships before docking (hours) Turnaround time of vessels % of scheduled services arriving on time Number of shipping lines calling in lhe port Average import dwell time(1) Turnaround time of road vehicles Traffic flows through the container terminal (TEU (2) / year) after one year ¡ (TEU / year) ' Traffic flow to and from area % of managers in the area who declare that shipping accessibility is a major constraint (1) Average time spent by import units in the port. (2) TEU - Twenty-foot Equivalent Units. Transport infrastructure Example 5: Upgrading of a railway line for freight operations Programme indicators (Related to the intervention and its effects) Context indicators (Related to the assisted area) Output indicators Progress Quantity Quality Result indicators Speed Impact indicators Traffic flow Transport system Indirect economic effect See Example 1 Number of km upgraded % of line allowing speeds of at least a desired threshold value (i. e. 100 km / h) % of line length allowing trains of heaviest standard load Typical commercial speed of freight trains on the line Average speed on ralways lines in the area Freight traffic flows on the line (Tonnes / year) after one year Traffic flow to and from area (Tonnes / year) Traffic flow of freight using the new infrastructure and withdrawn from road (Tonnes / year) % of area managers who declare that rail freight accessibility is a major constraint Example 6: Buik ling an intermodal (rail / road) centre Programme indicators (Related to the intervention and its effects) Context indicators (Related to the assisted area) Output indicators Progress | See Example 1 Result indicators Speed Impact indicators Traffic flow Typical commercial speed of goods transported through the centre Freight traffic flows through the centre (Tonnes / year) Freight traffic flows within the area (Tonnes / year) Transport system See Example 5 Indicators for monitoring and evaluation Other interventions having an impact in the field Whilst interventions in the field of transport infrastructure create important and widespread external impacts in numerous non-transport fields, impacts arising from non-transport interventions on transport infrastructures are generally quite limited. When these impacts do occur, they are most likely to be generated by interventions aimed at supporting economic development and enterprises. Interventions in these fields are likely to affect transport system indicators via increased economic activity. Similarly, interventions targeted at the environment or tourism may have an impact on the transport system (i. e. decreased or increased private vehicle use). impact • Availability and collection of data Work site managers should be required to provide synthetic output data (rate of completion and compliance with project duration) every six months. As part of their normal management activity, the operators should quantify simple result and impact indicators (e. g. traffic flow, average speed, etc. ). Result indicators are typically quantified every year. Quantifying complex impact indicators (i. e. regional component of the traffic flow) generally requires that a special study or survey be launched. For instance, these studies may involve surveys of firms or individuals using the improved infrastructure. These studies are costly and cannot be repeated every year. A good method involves an initial study just before work is finished and a second study three years after completion. Context indicators relate to the transport system and the region as a whole, and many can be derived from statistical sources. Some European statistics are available at regional level as shown in Box 14. Box 14 - Eurostat: Transport infrastructure and road safety Regional statistics (NUTS II) Roads: total (km) of which Motorways (km) Rail: total (km) of which Double track or more (km) of which Electrified lines (km) Roads: network density (km / km2) Rail: network density (km / km2) Deaths caused by road accidents (nb / Mio inhab / year) Transport infrastructure There may be other available data of several types of impact (environment, market share by transport modes, industrial productivity) if regular collection of longitudinal data is made by national statistical offices or by regional organisations. Key indicators The key indicators presented below lend themselves to comparisons both between different interventions of the same programme and between those of different programmes, and can in certain cases be aggregated at a European level. Level Key indicator Resource Output(2) Result Impact Rate of consumption of budget (% of allocated funds) % of budget devoted to environmental mitigation measures (1) Rate of completion of project (% of objective) Compliance with project duration Average speed between principal economic centres(3) % of regional managers declaring that accessibility is a major constraint for their firm (1) Mitigation measures reduce or eliminate the environmental impact of supported projects. (2) In order to accurately measure outputs and reflect the dimension of the project they should be refined with additional details. (3) This gives an indication of the overall accessibility afforded by transport infrastructure investments. The basis of calculation is the E. S. S. (Equivalent-Straight-llne Speed) between the ten most Important economic centres of the region, using the most efficient mode of transport. For more information s e e- European Commission (1997), EU transport in figures. Luxembourg: Office for Official Publications of the European Communities. Burmeister, Α. , Colletis-Wahl, Κ. (1996), 'T. G. V. et fonctions tertiaires: grande vitesse et entreprises de services à Lille et à Valenciennes' in: Revue Transports Urbains, Volume 93. Button, Κ. (1995), 'What can Meta-Analysis tell us about the implications of Transport' in: Regional Studies, Volume 29(6), 507-517. Rephann, T. J. (1993), 'Highway Investment and Regional Economic Development: Decision Methods and Empirical Foundations' in: Urban Studies, Volume 30(2), 437-450. Energy infrastructure Structural interventions in the energy sector are aimed either at energy production (electric power production plants) or at energy distribution (electric power distribution networks, methane pipelines or other gas pipelines). They involve the construction of new infrastructure or the enlarging and reinforcing of existing facilities. Structural interventions may also aim at saving energy. Examples are the implementation of systems for the recovery of the heat generated within various production processes, or the production of electric power derived from renewable sources or biomass. The main objective of interventions is to supply energy to the population and to industry. In the first instance, the supply of energy for final consumption by families has to be able to guarantee that the quality of living conditions is acceptable. In the second case, the supply of energy to industry and public institutions has to be competitive with regard to quantity, quality and costs. This implies that two other crucial objectives, strictly related to economic development and quality of life, are attained: (1) reduced energy dependency on overseas sources through the exploitation of national and local sources of energy and (2) reduced environmental impact from energy production and distribution through the exploitation of alternative, and possibly renewable, resources. In the field of energy, it is particularly relevant to distinguish between national interventions and local ones. In the first case, intervention programmes usually consist of large infrastructure projects that provide energy to addressees or recipients through complex distribution networks; in other words, addressees or recipients do not directly gain from projects, but from the distribution networks to which they are connected. On the other hand, local interventions are smaller in size, more numerous and often aim at producing or saving energy for the producer's own benefit. Projects in the field of energy generally require several years to be implemented. They often receive loans from the EIB and national banks, in addition to Structural Fund support. Interventions within the field Resource indicators At the programme level, resources can be monitored by using an indicator of "compliance with project cost": i. e. the most recently available Indicators for monitoring and evaluation information on the total cost of the projects in proportion to the cost which appeared in the programming document. Output indicators The progress of a project is often verified by simply monitoring expenditure statements. In cases of large projects (e. g. building or upgrading of electric power production plants) the verification of in-progress implementation can be estimated by monitoring the planned completion date (final testing of the works). In case of smaller projects (e. g. building or upgrading of electric power distribution networks) it is also possible to quantify the physical progress of the works in terms of "percentage of km of network completed by a certain date in respect to the total km of network to be completed". As regards the final output of the intervention, the main indicator is the power installed for production plants (kW or MW is applicable to electricity). Power installed in kW / MW is a measure that can apply to almost all energy infrastructure, but the rule is to provide a breakdown by each source of energy (e. g. coal, gas or oil generation plants; hydro-electric system or wind system; etc. ). Result indicators The main results of structural interventions in the field of energy principally concern the cost and reliability of the new service supplied as well as the typical savings generated for end-users. Impact indicators Specific impacts are related to energy consumption (electricity, gas) after one year. Programme and context indicators are the following: amount of electricity and gas consumed as a result of interventions, number of users connected to the completed networks, and degree of customer satisfaction (to be monitored and evaluated mainly through specific surveys among users; alternatively, this could be captured through the operators' records related to the reliability of the service supplied). Temporary employment is also created during the investment phase (measured in jobs χ years). The following impacts to be considered are: value added (in practice, measured in terms of wages and salaries plus profits) generated by the functioning of new infrastructure (production and distribution of electricity and gas), employment created by the operation of infrastructure and the reduction in energy dependency on overseas sources. Since energy production and the Energy infrastructure environment are closely linked, indicators which inform on the depletion of natural non-renewable energy sources, (e. g. quantity of renewable energy produced) or on air pollution (e. g. quantity of fossil fuels burnt and emissions produced) should be employed whenever relevant. Most of these impacts can be either positive or negative, depending on the purpose of the projects (energy production or energy saving). Example 7: New electricity plant using fossil fuels Programme indicators (Related to the Intervention and its effects) Context indicators (Related to the assisted area) Output Indicators Progress Compliance with project duration Rate of completion Capacity kW of installed power Overall national electric power production (MW), broken down by energy sources Investment efficiency Investment unit cost per kW of installed power Unit cost per kW of installed power for similar investments Result Indicators Population served Quality of service Cost for final users Impact indicators Value added Estimated number of users on the basis of yearly average coefficients of energy consumption for domestic use and the annual production of the plant (kWH) Number and duration of interruptions of electric power distribution for the average user connected to the network to which the new plant is linked Estimated reduction in unit cost of energy purchase expressed in euros / kWH Average energy consumption per head per year. Number and duration of interruptions of electric power distribution for the average national user National average unit cost of energy purchase expressed in euros / kWH Value added generated by the plant (euros / year) Yearly value added in the electric power production and distribution sector (1) at national level Permanent employment Permanent employment created by the operation of the completed factory Indirect economic effect Depletion of natural fossil fuel resources Yearly average amount of fossil fuels necessary to operate the plant (Tonnes of Petroleum Equivalent) % of managers in the area who declare that cost or availability of energy is a major problem for their firm (1) class 40. 10 of NACE (Nomenclature statistique des Activités économiques dans la Communauté Européenne). Indicators for monitoring and evaluation Example 8: Constructing or upgrading an electric power distribution network Programme indicators (Related to the intervention and its effects) Context indicators (Related to the assisted area) Output Indicators Progress Quantity See Example 7 km of new / upgraded lines, broken down by kW categories Overall network length in the area broken down by kW categories Investment efficiency Investment unit cost per km of new line Unit cost per km of new line for similar investments R e s u lt I n d i c a t o rs Population served Quality of service Impact indicators Number of users connected to the new network (or new network portion), broken down in domestic / economic units Number and duration of interruptions of electric power distribution for the average user connected to the new network or part of network which is new Number and duration of interruptions of electric power distribution for the average national user Value added See Example 7 See Example 7 Permanent employment See Example 7 Consumption Overall yearly consumption of electricity by users connected to the new network (kWH) after one year Average yearly consumption of electricity at national level (kWH / connected resident, kWH / connected job). Indirect economic effect 1 See Example 7 Energy infrastructure Example 9: Upgrading a gas distribution network Programme indicators (Related to the intervention and its effects) Context indicators (Related to the assisted area) Output Indicators Progress Quantity See Example 7 km of new / upgraded pipes, broken down by diameter categories Overall network length In the area broken down by diameter categories Investment efficiency Investment unit cost per km of new pipe Unit cost per km of new pipe for similar investments Result Indicators Population served See Example 7 Consumption Quality of service Overall yearly consumption of gas by users connected to the new network (m3) Average yearly consumption of gas at national level (m3 / connected resident, m3 / connected job). Number and duration of interruptions of gas supply for the average user connected to the new network Number and duration of interruptions of gas supply for the average national user Impact indicators Value added See Example 7 Permanent employment See Example 7 Indirect economic effect (1) class 40. 20 of NACE Yearly value added in the gas production and distribution sector (1) at national level See Example 7 See Example 7 Other interventions having an impact in the field All structural interventions aim at increasing economic activity, either directly or indirectly. Consequently they play a role in determining higher in terms of electricity) and therefore the energy demand (especially these expansion of consequences are an intended effect of Structural Funds, they do not need to be monitored or evaluated through specific indicators. the energy sector. However, since none of Availability and collection of data The initial providers of information for the quantification of indicators are the operators of the new infrastructure (during the construction period, at the moment of project completion, during the management of the new infrastructure). Indicators for monitoring and evaluation Specific surveys among the programme addressees or recipients are not frequent in the energy sector since it is difficult to track consumers benefiting from new investments in cases of complex interconnected networks. Interesting statistics can be obtained from national and local bodies m a n a g i ng electric p o w er p r o d u c t i on plants and electricity and gas distribution networks, as well as from statistical offices. As Box 15 shows, the available statistics provide data for the calculation of most of the context indicators previously identified. Box 15 - E x a m p le of statistics available f r om Eurostat at regional level (NUTS II) Total electricity production capacity Electricity production capacity by source Total gross installed capacity Gross installed capacity by source Energy production by source Electricity production by source Gas production Production of coke, lignite, oil Total electricity consumption Electricity consumption by sector Accounts of companies in the energy sector CORINAIR statistics at national level and, in some countries, at sub-national level Key indicators The key indicators presented below lend themselves to comparisons, both between different interventions of the same programme and between those of different programmes, and can in certain cases be aggregated at a European level. Energy infrastructure Level Key indicator Resource Rate of consumption of budget (% of allocated % of budget devoted to environmental mitigation measures funds) Output Result Impact Rate of completion of project (% of objective) Compliance with project duration Number of users connected to the new infrastructures, broken down in domestic / economic units % of managers who declare that cost or availability of energy is a major problem for their firms Net employment created or maintained (FTEs of which held by women) For more information see. EPA (1996), Joint EMEP I CORINAIR Atmospheric Emission Inventory Guidebook. Copenhagen: EPA. Eurostat (various years), Energy - yearly statistics. Luxembourg: Eurostat. Eurostat, Renewable energy sources statistics 1989-94. Luxembourg: Eurostat. Eurostat, Energy balance sheets 1993-94. Luxembourg: Eurostat. Training The main objective of structural interventions in this domain is to promote the employability of individuals across the entire European Union, by adapting and modernising the Member States' systems of education, training and employment. the obsolescence of their qualifications. It has been observed that workers who have been in a particular area of employment over a long period have difficulty adapting to the new job market due to Industrial restructuring and, more generally, massive redundancies have created a large number of long-term unemployed. The need to link training and the search for new employment has thus become very clear. Young people looking for their first jobs also experience difficulties, especially those with few skills or recognised qualifications. Their rate of unemployment is higher than the average rate for adults. Training takes many forms: basic training at the outset of a career, the renewal or acquisition of skills throughout working life, academic or professional training, alternated or not with periods of employment. It may also be dispensed within the framework of assistance to young people or the unemployed to support their social and professional integration. In all cases, it is aimed at providing skills, qualifications, knowledge and know- how to individuals so that they can use them as a basis for negotiation in the job market. All training activities, irrespective of their nature and public, may be measured by means of the same indicators. On the other hand, the aim of the programme determines which indicators should be selected. Interventions within the field Resource indicators Resource indicators must highlight very clearly the distribution of European, · national, regional and local co-financing. Training programmes very often straddle two calendar years, creating a situation that requires very precise financial statements. Output indicators Output indicators are peculiar to the types of training delivered: surface area of the training centres, number of teachers, number of courses offered, number of Indicators for monitoring and evaluation trainees (to monitor the rate of attendance of the programmes), volume of training provided (if the funding of the intervention is directly related to the number of hours of training and the number of participants). These indicators are relatively easy to quantify as programme indicators. The number of participants can be expressed as a proportion of eligible persons (coverage rate). Context indicators are difficult to obtain because the supply of training courses is not always transparent. It may be delivered by operators with low visibility such as small companies, associations or institutions which do not take advantage of public funds and do not systematically declare all their training activities. In contrast, suppliers of training courses who depend directly on the major public actors occupy most of the field when it comes to making their activities known. Result indicators The addressees or recipients of assistance for training are individuals who attend training courses and who improve their qualifications, skills and immediate employability. The most suitable indicator is the "number of participants on training courses who obtained the required qualification". Although few analyses have focused on the conclusion of training periods, the qualification rate provides a robust indicator of the number of people who attended the training course right up to the end, even though someone who drops out mid-way may have done so to be recruited. Finally, the "rate of participation during the training" is proposed as an indicator that can be used to evaluate absenteeism. Employability would be a good result indicator if it could be quantified. In theory, employability is the ability of an individual to find a job or to make a success of a professional position. This ability depends on individual aptitude and particularly on capacities acquired or enhanced during the training. It also depends on the probability of a job offer occurring that corresponds with the capacities acquired. The latter dimension of employability is very difficult to define and measure. To date, no recognised method exists for doing so and no harmonised definition of employability has been established. Impact indicators The indicators which have received the most attention in many countries are the "rate of employment after a period of training", the "placement rate" or the "average period of unemployment after the training". Yet these Training indicators are not sufficient for evaluating an impact in terms of a return to employment because the best trainees often qualify for the most effective training (skimming-off effect). It is therefore difficult to say whether recruitment after training is really an impact of the training itself. It may just as easily be a simple effect of the intrinsic quality of trainees who have completed the course. The quantification of the rate of placement in employment must be carried out at a sufficiently distant time-horizon (6, 12 or 18 months after the end of the training) and must take into account the sustainability of jobs. Theoretically, it can never be guaranteed that the training was the direct and unique cause for occupational insertion. Conversely, in a situation of job scarcity, the fact of not finding work after completing a training course should not be interpreted as a failure of the programme. A training course which has a low placement rate may nevertheless have a significant intermediate impact in terms of placement of trainees into more advanced training. This impact may be monitored by a "rate of transition" indicator which records all the trainees who have improved their situation (placement in terms of work or education) after the training course. Many studies focus only on the effect on the placement of trainees but there are other domains to consider such as socialisation at work or the ability to cope with the everyday requirements of the workplace (e. g. timekeeping, attitude, etc. ). Some effects may be far off target compared to initial objectives. For example, when an effort is made to help the unemployed to find jobs, the result may be an influx of previously inactive people into the job market. The effect is an increase in the unemployment rate, but the effect on employment may, in fact, be positive. An important impact occurs for those firms or other organisations (indirect addressees or recipients) which employ individuals who have qualified on the training courses, and thus avoid paying part or all of the costs involved in recruiting and training an unskilled person. The ultimate impacts for firms' are difficult to measure. They can be conceived theoretically: better research and development, better quality procedures, better performance, greater value added, etc. It is, however, impossible to trace the effects of training so far downstream. In the context of training, a good indicator can be analysed by means of graphs and comparisons. A breakdown of individuals in the programmes - Indicators for monitoring and evaluation by gender, qualification or professional situation  can provide a lot of information about the success of training courses and the interest shown in them by different categories of people. The evolution of the "proportion of unemployed in the cohort of qualifying trainees" is a good indicator that is too often forgotten when more readily available indicators are chosen. Example 10: Financing of training for new trades for the longterm unemployed Programme indicators Context indicators (in relation to the intervention and (in relation to the entire target public) its effects) Output indicators Supply Volume of training Number of trainees enrolled Rate of coverage Result indicators Rate of success Number of training places offered (¡nel. number intended for women) Number of training courses (occupations) offered Number of hours of training delivered (hours χ trainees) Number of trainees enrolled on the course (incl. number of women) Number of trainees enrolled, as a % of the number of longterm unemployed in the region Percentage of trainees who qualified with the required capacities (incl. % of women) Immediate placement rate Rate ot recruitment during training (incl. number of women) Impact indicators Transition rate % of trainees who completed a more advanced course, after 12 months Number or longterm unemployed Placement rate 12 months % of trainees employed after 12 months Annual rate of recruitment after a period of longterm unemployment (1) Sustainable placement rate % of trainees employed after 12 months and who are still employed after 24 months (incl. % of women) (1) Unemployment is considered to be longterm if it has lasted for over 18 months. Training Example 11: Skills improvement programme for young people with few qualifications^) Programme indicators (in relation to the intervention and its effects) Context indicators (in relation to the entire target public) Output indicators Supply Result indicators Adaptation of training Success rate Impact indicators Number of trainees qualifying Salaries of the trainees recruited Number of training places proposed by the programme % of places offered corresponding with growing sectors % of young people who are trained for growing sectors Number of trainees qualifying / number of trainees enrolled on the training course (incl. number of women) Number of trainees trained Average monthly salary of trainees employed after 12 months (average for women / men) Number of young people with a low level of skills Average monthly salary of young people (1) A "young" person is someone who is under 25. He or she has a "low" level of skills if he or she has international that can normally be obtained at the age of 18. The none of the qualifications classification of qualifications used in "workforce" surveys can also be referred to. Indicators for monitoring and evaluation Example 12: Assistance to businesses for adapting their employees' skills Output indicators Activity of the operator Number of assisted businesses Number of training courses funded Result indicators Qualifications obtained Impact indicators Maintenance of employment Professional progress Programme indicators (in relation to the intervention and its effects) Context indicators (in relation to the assisted area) Number of projects prepared with the candidate businesses Number of businesses with a training programme in the area Number of businesses that have received support for training Number of employees who have received more than 10 days of training Annual % of employees who benefit from in-house training in the area Number of employees who have obtained a qualification in trades in growth areas Number or % of individuals who attended the training course and were not made redundant within 12 months (incl. number or % of women) Number or % of individuals who attended the training course and who were promoted (incl. % of women) Number of redundancies in the area / year Other interventions having an impact in the field The effects of some interventions merge with those of training, particularly at the national level (e. g. reduction in social charges for employers, tax reforms, incentives for recruitment or any combination of these three financial measures favouring businesses). Training may also be closely linked to the growth of the information society. For example, a very peripheral area might have access to occupational training by means of electronic communication. Availability and collection of data Only (temporal) monitoring data used to observe individuals over time can allow for the construction of coherent indicators. No training action can be indicators during or evaluated solely by means of easily available these data varies training. The accessibility of immediately after considerably and they cover periods of unequal length in different the Training countries. However, from the point of view of pragmatism, the cost of collecting them is proportionally very low compared to the cost of the training itself. Depending on the complexity and heterogeneity of the programme, the authorities responsible for it have to choose between exhaustive quantification by the operators and quantification based on surveys. Box 16 illustrates a very specific case in which it is possible to collect most of the information through the operators. Box 16 - Training and survey: proposed schema for data collection A scheme for collecting information consists of asking trainees to fill in a questionnaire at the beginning of the course. This questionnaire could even be attached to the registration procedure when there is one. It might include all the relevant questions concerning the person's experience, qualifications and, above all, motivations and aspirations. These variables are essential for judging the training course and enabling the trainees to achieve their goals. It is essential that the trainees also answer another questionnaire when they complete the training course. In this way questions can be asked about the course itself, even if it seems that the information thus obtained will be of little value since it may be biased by enthusiasm if the course went well or by resentment if it did not. If a qualification is obtained at the end of the course, these questionnaires can be handed out before the final exam. The main purpose will then be to see whether the addressees or recipients' aspirations have changed. During the third phase of the survey, 6 to 24 months after the end of the training course, the impact indicators can be quantified, possibly on the basis of a sample of former participants. While this survey is longitudinal by construction, only the third phase involves substantial costs. The best possible use is made of the presence of addressees or recipients at the launch and conclusion of the training course. Moreover, all questions about opinions are meaningful because they are asked in the present and the people questioned have not had time to rationalise their choice of answers. • Key indicators The key indicators presented below lend themselves to comparisons both between different interventions of the same programme and between those of different programmes, and can in certain cases be aggregated at a European level. Indicators for monitoring and evaluation Level Key indicators Resources Rate of real spending of available funds (% of budget allocated) Outputs Results Impacts Number of training courses financed directly (1) (incl. no. of women) Success rate in reaching the eligible public (2), Hours of services and training received by the addressees or recipients (incl. number for women)(3) % of trainees who belong to a priority public (e. g. jobless young people) Sustainable placement rate (4) (% of addressees or recipients who are employed after 12 months, incl. % of women) Rate of transition (% of addressees or recipients whose social situation has improved after 12 months, incl. % of women) (1) Number of trainees who started or completed the training course. (2) This indicator, also called the rate of coverage, is used to show whether the training effort was significant or not in relation to the problem requiring to be solved. For example, the percentage of individuals who were longterm unemployed at the start of the year and who were trained during the year will be measured. (3) The volume of training received (number of hours χ trainees) is used to compare the real importance of the training effort that was financed. (4) The sustainable placement rate is the most frequent, albeit highly Imperfect, measurement of the success of the training delivered. In order to facilitate comparisons, the date of quantification must be homogeneous. For more information. European Commission (1997), Key Figures on Vocational Training in the European Union. Luxembourg: Eurostat and CEDEFOR Grubb W. N. and P. Ryan, (1997), Plain Talk on the Field of Dreams: The Roles of Evaluation for Vocational Education and Training, Geneva: International Labour Office, Training Policies Branch. 1RES (La revue de), 1995. 'Comparaison européenne des dispositifs d'insertion professionnelle des jeunes' in: La revue de l'IRES, numéro 17, hiver. Schmidt G. , O'Reilly J. , Schömann, Eds (1996) International Handbook of Labour Market Policy and Evaluation. Edward Elgard Publishing Ltd, Cheltenham Werquin P. (1997), 'Dix ans d'intervention sur le marché du travail des jeunes en France, 19861996' in: Économie et Statistique, Volume 4 / 5, 121306. Tourism The three Structural Funds intervene in the field of tourism in the form of infrastructure provision, support for the creation or improvement of tourist accommodation, training in tourism-related occupations, as well as agri- tourism in rural areas. Most regional programmes and certain Community Initiatives (e. g. LEADER) involve tourism. Moreover, roads and airports add to the attractiveness of a territory for tourists. Other Community interventions, particularly those contributing to the improvement of the environment or to sustainable development, contribute to dynamism of tourist activities. Many territories affected by rural or industrial decline have developed investment strategies in tourism. The economic impact of these strategies is sometimes uncertain and is worth evaluating carefully. In particular, the real weight of the different components of investments in tourism must be taken into account. Accommodation accounts for by far the largest contribution to regional economies (an average of 75% of investments and 90% of jobs created in tourism). The intervention of Structural Funds in tourism has peculiarities that need to be recalled. Assisted operations (accommodation, infrastructure, employment) are usually small, which means that actions tend to be highly scattered. Eligible zones are often non-urban areas where the number of assisted sites is high. The operators may have a variety of very different status: private individuals, firms, local authorities or semi-public organisations. Interventions in the field Resource indicators The standard indicator of "actual expenditure of funds" is applicable to tourism. Where relevant, the heading "tourism" should appear in sectoral breakdowns of resource indicators. This is also valid for the other categories of indicators discussed below. This relative weight of investment in tourism, proposed in the programme, may be compared to the relative weight of tourism in the activities of the territory. Output indicators A programme for improving the supply of rooms or tourist lodges may be carried out at the initiative of private individuals or a municipality. In the first Indicators for monitoring and evaluation case, the private person is the addressee or recipient. He or she receives public support through the intermediary of a programme operator. The construction of the lodge is a decision of the addressee or recipient. In the second case, the municipality can be considered either as an operator - if it allocates support to other organisations or to private individuals - or an addressee or recipient - if it is directly involved as the constructor. Whatever the case, the construction of the lodge is financed by the programme. Consequently, it is an output. An output indicator is the "number of subsidies granted". This indicator may be broken down into private individuals of various activities and professionals in the tourist trade, themselves broken down into individuals, small and large enterprises. The term enterprise should be taken to mean an "economic unit", irrespective of its status. Other output indicators include the "amount of investments made by the addressees or recipients" as well as the capacity of these investments (e. g. the maximum number of visitors / day). These indicators are important since in terms of the time required to recoup the cost of investments - over 15 years - tourism is very similar to heavy industry. Because support is always attributed by operators, output indicators include the activity of these operators (e. g. number of market studies carried out; number of potential addressees or recipients contacted; rate of selection of projects). Result indicators Result indicators relate to the increase in the quality, owing to the intervention, of what is on offer to tourists. Quality can be measured, for example, by compliance with established standards and by the certification of tourist lodges. In farming areas, the "number of farmers whose share of turnover generated by tourism amounts to at least 30%", reflects changes in supply structures through the diversification of agricultural activities. Impact indicators Specific impact indicators measure the number of tourist visits to assisted facilities or accommodation (number of nights accommodation sold per year, number of visits per year, rate of occupation for different periods of the year). Tourist visits must be measured after an adequate period of time (12 months seems to be a minimum). Indicators of the number of visits must be interpreted with caution because custom in the tourist market tends to fluctuate. The general economic context, the weather, the development of Tourism sporting activities, and competition from foreign destinations at low prices, all add up to make the market unstable. It may be interesting to break down the number of visits in terms of the origin of the visitors (nationals of the country or foreigners) in order to assess the programme's contribution to the attractiveness of the region for tourists. An indicator of value added (in practice the value added is the sum of wages, salaries and profits) can be applied to economic units supported in the domain of tourism. It generally consists of the gross value added generated by projects in euros / year. Value added is an interesting indictor because it specifically measures economic development. It allows the factors that are likely to cause a territory to move from an area of tourist activity towards a tourist economy to be highlighted. The "number of jobs created / maintained" applies to tourism, provided the seasonal character of the activity is taken into account (e. g. estimated in terms of job-days). It is also worthwhile evaluating the skills requirements of jobs created because the tourist economy requires trained and adaptable personnel. The natural environment and the tourist sites are no longer enough; the gap with competing territories will widen in direct relation to the levels of training of personnel in tourist businesses. Value added and employment indicators can be linked to context indicators such as average value added per job in the tourist sector. Indicators for monitoring and evaluation Example 13: Support for the creation and improvement of accommodation Programme indicators (in relation to the intervention and its effects) Context indicators (in relation to the assisted region) Output indicators Activity of operators Number of addressees or recipients Number of contacts with potential addressees or recipients Number of economic units assisted Capacity Number of beds created or improved % of beds created or improved which have a quality label Number of farmers whose share of turnover attributable to tourism is at least equal to 30% Total number of units of tourist accommodation % of beds which have a quality label Number of nights sold per year and per bed in assisted accommodation, after one year Evolution of consumption of electricity in the territory during the tourist period % of nights sold to foreign tourists Number of charter flights in the regional airports Number of nights in an average stay in assisted accommodation Number of nights in an average stay in the territory Value added generated in euros / year / bed, after one year Net number of jobs created (incl. % occupied by women) Average value added in euros / year / bed Number of jobs in the tourist sector Result indicators Quality Diversification Impact indicators Number of visits Attractiveness for foreign tourists Length of tourist visits Value added Jobs created Tourism Example 14: Support for the creation of tourist facilities (museums, amusement parks, etc. ) Programme indicators (in relation to the intervention and its effects) Context indicators (in relation to the assisted region) Output indicators Activity of Operators See Example 13 Number of addressees or recipients See Example 13 Total number of tourist facilities Capacity Maximum number of visitors / day Result indicators Length of the visit Cost of the visit Impact indicators Number of visits Attractiveness for foreign tourists Normal length of visit to the facility (in hours) Average cost of a visit to the facility (in euros per person) Number of visits per year to assisted facilities % of visits by foreign tourists Value added Value added generated in euros / year Value added generated in the tourism sector In euros / year Jobs created See Example 13 See Example 13 Other interventions which have an impact in the domain Examples of the multiple interventions which may have an impact on tourism include: efforts to reduce pollution; cleaning up of rivers and beaches; improvement of the water and air quality; rearing of livestock in mountain areas for maintaining uphill grasslands: upkeep of roads; extensification of livestock rearing as opposed to more polluting intensive techniques; improvement of farm buildings; maintenance of wetlands, cleaning up of industrial wastelands; revegetation of former mines and foundries, restoration of the urban and rural heritage; opening up of isolated areas; improvement of links with peripheral regions and of roads, railways, sea links, airports, ports, local transport networks and, lastly, cultural interventions such as RAPHAEL and KALEIDOSCOPE. The following example concerns training. Indicators for monitoring and evaluation Example 15: Funding for language training Programme indicators (in relation to the intervention and its effects) Context indicators (In relation to the assisted region) Output indicators (in the field of training) Quantity Number of places offered Number of jobs in tourism Number of trainees enrolled in the course Number of trainees enrolled (incl. number of women) Volume of training Number of hours of training delivered (hours χ trainees) Result indicators (in the field of training) Number of trainees trained Rate of satisfaction I m p a ct indicators Use Number of trainees qualifying with the required level (incl. number of women) % of trainees satisfied or highly satisfied Proportion of trainees who use the skills acquired in a tourist activity in the region, after one year Availability and collection of data on training Statistics for context indicators have to be collected from the relevant organisations and local, regional and national authorities. In the future, the main effort will have to focus on searching for elements to measure the economic dimensions of tourism more accurately, e. g. by observing the evolution of turnover of activities directly or indirectly related to this activity. This is currently the case of hoteliers who have grouped together to create management centres. Tourism is a sector in which many statistics are not comparable without analysing the conditions in which the data were collected. For example, as far as the "number of tourist visits" is concerned, accommodation uses the notion of "number of visits sold" or of "nights sold", tourist bureaux count the number of persons who visit their offices, and nature reserves count the number of vehicles in the park multiplied by the average number of people per car. For the above indicator, an analytical framework and criteria should be suggested to all the territories concerned defining exactly how the number of visits should be calculated. Tourism Key indicators A number of key indicators lend themselves to comparisons between different interventions in the same programme and with other programmes. They can therefore easily be aggregated on a European scale. Level Key indicator Resources Rate of real consumption of available funds (% of budget allocated) Outputs Number of economic units which have received direct support or a service supported by the programme (including the size of the unit: large, medium, small, individual) Number of new economic units (less than a year old) which have the received direct support or a service supported by programme (including size: large, medium, small, individual) Impacts Value added generated (euros / year / employee) Net jobs created or maintained (in full-time equivalent, including % occupied by women) Bibliographic references Guesnier B. (1996), Le patrimoine, une valorisation touristique à haut risque. Paris: Institut de Prospective du Tourisme et Fondation Auguste Escoffier. Ministère du Tourisme (1996), 'Prospective et Tourisme' in: Les cahiers espaces du Ministère du Tourisme France, Décembre. Fisheries The aim of Structural Fund interventions in the area of fisheries is to strengthen the sector which is going through a prolonged period of restructuring resulting from the increasing need to reduce overfishing of limited natural stocks. The overall objective is to accelerate the structural adaptation of the fishing industry and allow the emergence of a sustainable European fishing industry. Interventions are financed by the Financial Instrument for Fisheries Guidance (FIFG), introduced in 1993. This instrument was conceived with the aim of bringing together all fisheries oriented structural measures, in particular those concerning the promotion and marketing of fisheries and aquacultural products. Interventions in this field include financial support for the restructuring of the fishing fleet, such as subsidies for the withdrawal from service of fishing vessels and payments to fishermen wishing to retrain or to take early retirement. Other interventions directly related to fishing include the modernisation of fleets and harbour facilities. Another source of funding is the PESCA Community Initiative, which aims to cushion the social and economic consequences of restructuring by supporting the diversification of economic activities. Diversification is promoted by grants to support productive investment, the provision of services for SMEs (management, technology transfer, marketing, quality control, etc. ) and subsidies to promote tourism (tourist attractions, yachting and other leisure activities). Interventions within the field Resource indicators Standard resource indicators can be broken down in order to provide information on interventions aimed at: reducing fishing capacity, improving processing and trading conditions, developing aquaculture, modernising port facilities and measures supporting retraining, new businesses, etc. Output indicators In the case of subsidies targeted at reducing fleet sizes, standard output indicators focus, firstly, on the activities of operators (e. g. number of briefing sessions with trawler owners, number of counselling sessions with fishermen, etc. ). Secondly, they estimate the physical manifestation of the Indicators for monitoring and evaluation subsidy, in particular, the tonnage of fishing vessels withdrawn from service. The number of ex-fishermen receiving income substitution payments is also an indicator of the reduction of fishing activities. Where direct support for investments is concerned, outputs can be estimated by using physical measures (e. g. surface area of low temperature storage facilities built or improved, additional productive capacity in the aquaculture sector, etc. ). Contextual indicators provide information on national or regional fishing capacity: the size of the fishing fleet and the rate at which it is being retired. Result indicators The results of interventions designed to reduce fishing capacity and manpower can be estimated by the catch of withdrawn vessels during their last year of service and the retraining of ex-fishermen (e. g. the number of ex-fishermen working in the aquaculture and tourist sectors). In the case of the modernisation of port infrastructure indicators should focus on factors that increase both the potential of shipside activity (e. g. improved handling, turnaround, etc. ) and portside activity (e. g. increased supply and of quality warehousing, cold storage, etc. ). Impact indicators The type of impact indicator used generally depends on the objective of the intervention. Measures that aim at diversifying the local economic base, the retraining of ex-fishermen or the modernisation of infrastructure and facilities require the use of socio-economic indicators which cover jobs created or maintained (FTEs), enterprises created, employment of retrained ex-fishermen, sales turnover, value added (i. e. wages and salaries plus profits), etc. Those interventions aimed directly at fishing activities and resource conservation will focus on the size of catches, broken down into species of fish. In the case of interventions to modernise port impact indicators should capture the effects of improvements on traffic levels, catches and portside conditions (e. g. better sanitation, fewer accidents, etc. ) infrastructure, According to the objective of the intervention, the relevant area for context indicators may either be the port, the eligible area or the region (for socio like jobs or value added) or fishing fields (for economic environment indicators like sustainability of fish stocks). indicators Fisheries Example 16: Grants for aquaculture farms Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Number of addressees or recipients Productive capacity Result indicators Leverage effect Reconversion I m p a ct i n d i c a t o rs Production Value added Labour productivity Direct employment Number of farms receiving grants Additional capacity of supported farms (tonnes / year) Total output in the region (tonnes / year) Accompanying Investment by recipient farms Number of ex-fishermen investing in aquaculture Additional production of the recipient farms after 12 months (tonnes / year) Catch of similar species in the relevant fishing fields (tonnes / year) Value added in the recipient farms after 12 months Value added in the region / in the sector Value added per employee of the farms after 12 months Average value added by employee In the region / in the sector Number of net jobs maintained or created in the recipient farms after 12 months Number of jobs In the region / In the sector Indicators for monitoring and evaluation Example 17: Subsidies for laying up fishing vessels Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the relevant fishing area) Output indicators Activity of the operator Number of vessel owners briefed Fleet tonnage Tonnage of fishing vessels laid up Fleet tonnage in the related fishing area Result indicators Reduction of fishing capacity Impact indicators Fish stocks Sustainability Catch of laid up vessels during their latest year of activity (tonnes / year / species) Total catch by species In the related fishing fields (tonnes / year) Estimated stock in the relevant fishing field(s) (thousands of tonnes by speciosi % of stocks fished which are above the MBAL(1) of the related fishing area (1) MBAL - Minimum Biological Acceptable Level is an estimate of the mature stock level of a species. Below this level there is an increasing risk that the reproductive potential of the stock will collapse. High percentages of stocks fished that are below the MBAL indicate an increased risk of resource depletion. Fisheries Example 18: Modernisation of a fishing port Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Progress Capacity Result indicators Compliance with project duration Rate of completion Port capacity built or rehabilitated (number of vessels and tonnage) Turnaround | Average turnaround time of vessels Impact indicators Traffic Catch Business activity Improved hygiene Tonnage of vessels using the port after one year Catch landed (tonnes / year / species) Surface area (m2) of port-side units bought or rented by traders, wholesalers, etc. Reduction in the number of incidents in which hygiene regulations have been breached Value added Value added generated in the port area Employment Number of jobs created or maintained in the port area (FTEs of which held by women) Value added related to the fishing industry in the town / region Number of jobs In the fishing industry (FTEs of which held by women) Other interventions having an impact in the field Numerous interventions in other fields are likely to produce effects in the field of fisheries and aquaculture. For example, a treatment plant that brings an end to direct discharges of human waste in coastal waters will contribute to the quality of beaches and therefore facilitate efforts to diversify the local economy towards tourism. It may also contribute to the diversification of economic activities by allowing the development of aquaculture (mussel farms, etc. ). Another case could be the setting up of a data exchange system linking up trawler owners, processing plants, and wholesalers allowing the local market for fish to function more efficiently. Similarly, the construction of a new road link to the national motorway system could have an impact on the transport costs of local firms which process locally landed fish, or improve tourist access to a previously poorly connected region. The example presented below is the case of financial support for the retraining Indicators for monitoring and evaluation of fishermen. These courses could be targeted at providing skills suitable for becoming self-employed, (e. g. business plans, book keeping, taxation and grants, etc). Example 19: Support for retraining of fishermen Programme Indicators (Related to the Intervention and its effects) Context Indicators (Related to the assisted area) Output Indicators (in the field of training) Quality Number of recognised qualifications available Volume of training Amount of training received by trainees supplied Take up (hours χ trainees) Number of ex-fishermen attending Total number of potential trainees Result indicators (in Ui e field of training) training courses Success rate % of trainees successfully completing the course Satisfaction rate % of trainees who are satisfied or very satisfied with the training received Impact indicators Retraining % of participants who are employed outside the fishing sector after 12 months Unemployment amongst ex-fishermen in region Availability and collection of data Operators through their monitoring systems, as with other fields examined in this Volume, should systematically collect output and results data. When the co-financing of investment projects is concerned, operators should strive to collect data on the amount invested by private sector partners in order to measure any leverage effects that may occur. Impact related data have to be collected through surveys involving addressees or recipients and the wider population. Whilst the tracking of addressees or recipients who undergo retraining and consequently leave the region may pose some problems, the impacts of other interventions will generally be geographically localised. In the case of context indicators relating to fishery resources, estimates of fish stocks and MBAL are made by the International Council for the Exploration of the Sea (ICES). These are based on data on international Fisheries catches, fishing effort and estimates from research vessels on the abundance of stocks. Government Ministries for agriculture and fisheries provide national data on fishing catches. A statistics annual devoted to fisheries is published by Eurostat. Key indicators The key indicators presented below lend themselves to comparisons both between different interventions of the same programme and between those of different programmes, and can in certain cases be aggregated at a European level. Level Key indicator Resource Rate of consumption of budget (% of allocated funds) Output Result Impact Number of hours of services (training, consultancy, etc. ) received by addressees or recipients Number of individuals receiving direct assistance or services as a result of the programme (of whom men / women) Number of economic units (firms, ship owners, fish farms, etc. ) receiving direct assistance or services as a result of the programme Number of new economic units receiving direct assistance or services as a result of the programme Satisfaction Rate (% of addressees or recipients satisfied / very satisfied by services provided) Immediate success rate (% of addressees or recipients no longer requiring services at the end of their participation in the programme) % of addressees or recipients employed after 12 months % of assisted new businesses that are still active after 12 and 36 months Gross value added (after 12 months in terms of euros / year /employee) For more information see. Commission européenne (1994), Efude comparative et intégration des données socio-économiques dans le secteur de la pêche et de la Communauté. Luxembourg: Office des l'aquaculture dans publications officielles des Communautés européennes. Indicators for monitoring and evaluation European Commission (1994), Employment in the industries of UK, Denmark and Portugal. Luxembourg: fishing Official Publications Office of the European Commission. impact assessment Eurostat (1997), Pêche - Statistiques annuelles. Luxembourg: Eurostat. Department of the Environment (1996), Indicators of Sustainable Development for the United Kingdom. London: HMSO. OECD (1997), Review of Fisheries in OECD Countries 1995. Paris: OECD. Research and Technological Development Structural Fund interventions in the field of Research and Technological Development (RTD) include support for RTD activities in enterprises, especially in SMEs, through the setting up of science and technology parks, subsidies and loans for the development of new products and processes, provision of services such as technology audits, expert advice and the creation of shared data sources (e. g. to give enterprises the possibility of finding suitable partners in other European countries). They equally concern the creation or strengthening of research institutes, university faculties and public laboratories, support for research projects, the financing of individual researchers and training or retraining of workers. More recently, in order to support the transfer of technologies to industry, interventions have become research increasing oriented organisations and enterprises, and developing the capacity of SMEs to acquire and profitably use new technologies. towards encouraging links between through The aim of interventions in this field is to promote economic growth and the development of new activities based on employment in this sense means sustainable technological progress. Sustainable creating a positive and self-reinforcing dynamic between innovatory capacity, competitiveness and economic growth. In order to do this, interventions generally aim to assist the enterprises and workforce of a region to adapt to, and profit from, the opportunities that technological change presents. This involves creating and adapting necessary infrastructures, supporting training and education, promoting collaborative research where this is relevant, and encouraging a region's economic actors to move towards the production of high technology, high value added products and services. Whilst the ERDF is the major source of funding, all the Structural Funds are involved in funding RTD interventions. These are also supported by most of the Community Initiatives, since technological development represents a major precondition for reaching their various aims. RTD has a number of particularities that shape the systems of indicators used to evaluate interventions. Firstly, RTD capacity is highly concentrated in a small number of enterprises, laboratories and universities that tend to be clustered in and around capital cities and a few economically dynamic regional poles. Furthermore, RTD is polarised in international terms with the majority of activity taking place in a small number of industrially advanced Indicators for monitoring and evaluation institute-enterprise, nations. The success of RTD projects depends to a large extent on inter- enterprise, inter-institute and cross-border co operation. The diffusion of results and technology transfer must occur if RTD is to benefit not only the institutes and firms who perform it, but the region in which they are located. The results of RTD activity include not only products and processes, but also knowledge and know-how, which is largely embedded in individual researchers and research teams. These "intangibles" are also inputs into the RTD process. Finally, a substantial amount of RTD is performed by public agencies (mainly university research centres and public laboratories) which is generally an input into industrial RTD performed by firms. Interventions within the field Resource indicators The distribution of financial resources between private and public sector projects gives an indication of the capacity of a region's enterprises to absorb RTD funding. When private sector uptake is low, this may mean that the capacity of enterprises to perform their own RTD or to profit from RTD performed laboratories and related institutions is limited. The share of financial resources devoted to innovatory projects as opposed to institute-enterprise or international / inter-regional technology transfer projects gives an indication of the capacity of enterprises to acquire outside technologies. Resource indicators should be designed to reflect these issues. in university research centres, Output indicators Outputs of RTD interventions can be in the form of infrastructure, services to business and support for research. In the first instance, monitoring focuses on physical indicators providing information on the number and surface area of high-tech development parks opened or the surface area of specialised facilities, usually provided for small high-tech firms. In the public sector indicators relate to laboratory and research centre floor space, the amount of equipment provided (e. g. number of computers, etc. ). In the case of services, indicators relate both to the physical element of the service (e. g. number of certified experts and consultancies providing technology audits, diagnosis and other services to firms, etc. ), to the level of service activity (e. g. the number of contacts occurring between the Research and technological development operators and addressees or recipients of a programme) and to the volume of services provided (e. g. the "number of hours of technology audits undertaken", etc. ). Result indicators Most benefits arising from RTD occur after a period of many months, even years. For this reason, it is not easy to design result indicators that measure immediate effects. In the case of RDT infrastructure like science parks, result indicators could provide information on the quality / cost ratio of the equipment, from the standpoint of the potential users. In the case of services, result indicators basically measure addressees' or recipients' satisfaction. It is also useful to measure to what extent the supplied service matches the needs of local firms. One indicator of local firms' interest is the amount of private money invested in the supported projects (leverage effect). Another interesting indicator is the proportion of addressees or recipients launching their first RTD activity with the support of the programme. One of the problems of Structural interventions in the field of RTD is that resources tend to be concentrated in public research bodies. This is because initial low levels of industrial RTD performed by firms restrict their ability to launch RTD projects to absorb available funds. This exacerbates any mismatch that may exist between technology supply and demand. The BERD / GERD3 ratio is a contextual indicator of industrial RTD intensity reflecting the capacity of private sector to undertake RTD or absorb funding. The "ratio of researchers (total, private, public) per 1,000 workers (FTEs)" gives similar contextual information. Other context indicators include RTD as a percentage of exports and as a percentage of GDP. In the case of technology transfer support, the "number of assisted firms purchasing patents, or licences, or entering into collaborative R&D ventures with foreign firms" is an indicator of formal cross-border technology' transfers. The percentage of projects approaching the marketing stage (i. e. one year or less away) can give an indication of the proportion of the likely economic benefits of the programme. 3 Business Expenditure on Research and Development, Gross Domestic Expenditure on Research and Development. Indicators for monitoring and evaluation Where public research is concerned, results of interventions to support individual researchers can be measured by the number of supported post graduate researchers successfully completing their programme of research and / or obtaining a PhD. Impact indicators to rather to measure In the field of RTD evaluation of public institutions it is customary to focus on such indicators as the number of PhDs and bibliometric measures, the production of notably publications and citations, knowledge. However, Structural Fund interventions in the field of RTD now increasingly emphasise the development of the RTD capacity of regional industry rather than supporting more upstream institute-based research. Furthermore, the knowledge produced by universities and research the worldwide stock of publicly available institutes contributes to regional economic knowledge than contributing directly development and social cohesion. It is in these circumstances that indicators reflecting an academic rather than an industrial logic are of limited use. Impact indicators for Structural interventions which do support upstream activities should therefore concentrate on such areas as human resource development, industry-university collaboration and the production of potentially marketable products and services. The "number of co operative R&D arrangements between institutions and enterprises" and the "number of graduate placements in firms" are indicators of collaborative effort. Impacts arising from supported public and joint research projects can be measured by the number of prototypes or demonstration units produced and the number of patents and licences sold. In the case of support for industrial RTD, interventions are oriented towards innovation and technology acquisition. This can be measured using the the acquired number of new products or processes technology, which are marketed by the recipient firm. incorporating Direct employment impacts can be estimated by using the "number of R&D posts created or maintained within the firm" and the "proportion of R&D personnel employed in relation to the total workforce (FTEs)". Because of the long lead times between research, development and the marketing of an the impact expectations of addressees or recipients. indicators often have to be based on innovation, Research and technological development Example 20: Sup sort for a science and technology park for SMEs Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Quantity Result indicators Cost Scientific attractiveness I m p a ct i n d i c a t o rs Occupation Networking Direct employment Surface area (ha. ) of S&T(1) park Floor space available (m2) in the park Total surface area (ha. ) of S&T parks in the area Total floor space availablelm2) In S&T parks Cost of establishing a small high-tech firm(2) in the park (euros / year /m2) Number of researchers working in the vicinlty(3) of the park Number of small high-tech firms establishing themselves in the park after one / three years Number of research institutes in the park (originating from outside the region) Number of collaborative projects involving two or more occupants of the park after one/three years Number of R&D posts created by park occupants after one/three years (FTEs, including number held by women) Number of small high-tech firms In the area Number of RTD posts in the area per 1,000 workers (1) Science and Technology. (2) Whilst there is no agreed single definition of what constitutes a high-tech firm, they can be generally defined as enterprises that are active in what is considered a high tech sector. One way of Identifying a high tech sector is by their significantly higher than average expenditure on R & D as a proportion of sales or significantly greater than average proportional employment of qualified scientists and engineers. The OECD has also established a typology of sectors using the ISIC classification that categorises sectors as high tech, medium tech, medium -low-tech and low -tech. (3) Vicinity can be defined by "a distance of less than 5 km". Moreover, this definition must be adapted to the practical situation. Indicators for monitoring and evaluation Example 21: Financial and informational support for technology acquisition by SMEs Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) O u t p ut indicators Number of addressees or recipients Number of SMEs receiving support Number of SMEs Volume of technical services provided Hours of consultancy and training services dispensed Result indicators Leverage effect I m p a ct i n d i c a t o rs Internal technology transfer(2) External technology transfer(3) Technology acquisition Economic impact Investment In acquisition oriented RTD by firms benefiting from financial support Ratio of BERD to GERD(1 ) Number of collaborative arrangements between research institutions and assisted enterprises after one year Number of assisted firms purchasing patents, licences or entering into R&D collaboration with foreign firms after one/three years Number of new products or processes incorporating the acquired technology marketed by assisted firms after one / three years % of managers of recipient firms who declare that the Intervention has relaxed constraints in terms of access to new technologies % of managers in the area who declare that access to new technologies is a major constraint Value added Value added generated after one/three years In euros / year Value added generated in the area by SMEs (1) Business Expenditure on Research and Development, Gross Domestic Expenditure on Research and Development. (2) Internal technology transfer refers to transfers taking place within a region or country, generally between research institutes and enterprises. (3) External technology transfer concerns the acquisition of technologies resulting from RTD activities by enterprises outside the region or country. Research and technological development Example 22: Support for post-graduate research Output indicators Research activity Result indicators Qualifications Networking Impact indicators Potential innovatlons(l) Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Number of supported research students (of which women) Number of research projects employing supported researchers Number of researchers employed In the area per 1,000 workers Number of supported researchers completing post-graduate research programmes and obtaining a PhD Number of contacts and collaboration with regional firms involving supported researchers % of supported postgraduates hired by regional firms Number of patents taken out for potential innovations being developed with private sector partners resulting from research by supported researchers - after one/three years Annual number of doctoral students In the area Number of patents taken out by firms in the area (1) Potential innovations include prototypes, demonstration units and services under development. Other interventions having an impact in the field The central objective of Structural Fund interventions in the field of RTD is to encourage greater RTD activity in regions where it is low and hence to close the technology gap between these and more dynamic regions. However, the dynamics of RTD are such that the disparities in capacity have proved to be highly persistent and RTD activity remains clustered and geographically polarised. Interventions supporting the development of information networks and improved telecommunications systems may help to counteract this effect, as they permit research institutes and firms to' enter into effective collaborative ventures over larger distances and generally improve networking possibilities. Availability and collection of data Output and result data from the monitoring of projects may be complemented by data on the satisfaction of addressees or recipients with Indicators for monitoring and evaluation services provided by operators. Information on enterprises should include their size in terms of employees or turnover, since enhancing RTD investment and technology acquisition capacity of SMEs is a major objective of interventions. Data should also include information on the accompanying expenditure of firms on RTD, so that the leverage effects of interventions can be estimated. As in the case of more general interventions aimed at enterprises, impact related data for firms have to be collected through sample surveys and extrapolated to allow general conclusions to be drawn. Given the extended time scale over which RTD impacts occur, some reliance has to be placed on projections made by firms on the likely impacts of interventions. In such cases, the replies of the respondent are more likely to overestimate than underestimate the potential impacts of an intervention. Context indicators relate to the state of the science and technology system of a region or regions (e. g. number of PhDs, post PhD researchers, BERD, GERD, number of R&D employees, etc. ). Data for context indicators are generally available at a national level and the up-to-date statistics for R&D expenditure and patent applications at the national and regional level in EU and the European Economic Area (EEA) are also available. Key indicators The key indicators presented below lend themselves to comparisons both between different interventions of the same programme and between those of different programmes, and can in certain cases be aggregated at a European level. Research and technological development Level Key indicator Resource Rate of consumption of budget (% of allocated funds) Output Result Impact Selection rate (1) (% of projects accepted in financial terms) Number of hours of expert advice received by addressees or recipients Satisfaction rate (% of addressees or recipients satisfied / very satisfied by services provided) Leverage effect(2) (private sector spending occurring as a consequence of the programme in relation to financial support received) Value added / sales generated (atteri 2 / 36 months in terms of euros / year / employee)(3) (5) Net employment^) created (FTEs of which held by women) after 1 2 / 36 months (1)The project selection rate provides information about the quality of applications for assistance, allowing inferences to be drawn on the absorptive capacity of regions weak in RTD. (2) The stimulation of private sector investment is often an explicit objective of certain types of interventions, particularly RTD. (3) Given the long lead times for RTD projects, changes in value added and sales after 36 months are likely to be limited. (4) Employment created by RTD interventions is more readily measured in terms of scientists and technicians. Employment impacts in areas not directly associated with performing RTD are generally likely to occur in the medium to long-term, and are therefore more difficult to measure. (5) Given the long lead times for RTD projects, the number of innovations resulting from an intervention are liable to be limited, even after 36 months. Indicators for monitoring and evaluation For more information see. Archibugi D. and Pianta M. (1996), 'Innovation surveys and patents as technology indicators: the state of the art' in: Innovation, Patents and Technological Strategies. Paris: OECD. Commission européenne (1994J, Rapport européen sur les indicateurs technologiques 1994, Luxembourg: Office des scientifiques et publications officielles des Communautés européennes. Eurostat (1997), R&D Annual Statistics 1997. Luxembourg: Eurostat. Lundvall B. -A. (ed. ) (1992), National Systems of Innovation: Towards a Theory of Innovation and Interactive Learning. London: Pinter. OECD (1994), The Measurement of Scientific and Technological Activities: for Surveys of Research and Proposed Standard Practice Experimental Development - Frascati Manual 1993. Paris: OECD. OST (1996), Science & Technologie: Indicateurs 1996. Paris: Economica. Agriculture and rural development Structural Fund interventions in the field of agriculture and rural development are very heterogeneous. They are mainly financed by the EAGGF-Guidance and ERDF. Many measures in this field intersect with those in the other fields mentioned in this Volume. Nevertheless, four types of interventions are specific to this field. Firstly, rural infrastructure (e. g. village renewal, etc. ). Secondly, the diversification of farms (e. g. agro-tourism, farm shops, new farm products, labelling, education and advice). Thirdly, the rationalisation of production (e. g. investments in farms, assistance to young farmers, retirement schemes, processing and marketing of agricultural products). Finally, support for low intensity agricultural production (e. g. afforestation, environmentally favourable land use). The addressees or recipients of interventions are farmers, former farmers and other economic units in rural areas. The rationale for assisting the agricultural sector and rural areas has several dimensions. The agricultural sector is in need of structural adjustment because the demand for food is stable, while high rates of technological development have resulted in an increase in supply, a surplus of production, and a decrease in rural population and density. Income and employment alternatives for farmers as well as for firms and their employees in related crafts and trades have to be established and developed. Single projects are usually very small in terms of investment volume and large in number. Interventions are often complex measures constructed as local programmes, containing various types of public and private projects. Interventions within the field Resource indicators For programme driven interventions, resources may be monitored through the measurement of project subsidies and compared with the funds in the programming document. Sometimes the national contribution is much higher than planned in the programming document. If this is the case, additional monitoring of European Structural Fund financial resources is necessary. Output indicators Outputs of interventions in the area of infrastructure can be measured in physical units: number of processing and packaging plants constructed, Indicators for monitoring and evaluation surface area of renewed villages, etc. Due to the high degree of heterogeneity of projects, it is difficult to consolidate indicators on a level higher than measures, except by using an aggregated rate of completion. In the general case of subsidies and grants, the main indicators are the "number of economic units assisted" and the "number of private households assisted". These indicators might be combined with financial resources, and broken down into categories of farms or households, especially into income groups. Result indicators the limited Because of rural infrastructure, addressees or recipients can usually be identified without too much difficulty, e. g. inhabitants connected to a new sewage system, residents served by an improved road. interventions concerning range of The "proportion of assisted farms and households in the defined area or group" indicates the degree of penetration of the programme among the total number of farms or households in the defined area or group. A further indicator is the "amount of additional private expenses incurred by farms and rural households in relation to the support received". This indicator measures the "leverage effect". Due to a ceiling on support, the investments actual investments of farmers might be higher than the indicated on application forms. Indicators providing information on changes in the structure of the local economy focus, for example, on the diversification of farmers' incomes, the composition of the addressees or recipients in terms of age and gender. Impact indicators Unlike larger projects in other fields, impacts of rural development activities are more concentrated in the immediate region. Agricultural and rural development activities have impacts that depend on their various target groups. The diversification of activities has a direct impact on fanners, helping to provide new employment and income opportunities adjacent to those in the agricultural sector. Investment support to improve processing and marketing of agricultural products has an indirect impact on farms. However, due to the high degree of competition in the food sector it will not necessarily result in higher producer prices or larger market shares. In conjunction with this support, supply contracts can help to reduce farmers' income risk. Agriculture and rural development Rural infrastructure improvement measures have impacts on the residential attractiveness of the eligible area, helping to avoid people moving away from the region. Assistance has possible additional impacts on rural tourism as well. The time scale of impacts varies between one year (for an impact on agro- tourism) and ten years or even more (for an impact on population density). Besides the impacts of Structural Funds programmes, rural areas are thoroughly dependent on other policies (mainly EAGGF-Guarantee). Natural and climatic conditions also have a substantial influence on farmers' incomes. Example 23: Fina of y ncial support to assist the sung farmers setting up Programme Indicators (Related to tbe Intervention and its effects) Context Indicators (Related to the assisted area) Output Indicators Number of addressees or recipients Number of assisted young farmers (incl. % of women) Total number of young farmers Result indicators Leverage effect Restructuring I m p a ct i n d i c a t o rs Survival rate Total investments made by assisted young farmers (broken down into farm type (1) Number of assisted young farmers(2) who replace retiring farmers Average stock of capital per farm Number of farmers retiring per year Age distribution of farming population Ratio of farmers starting out to farmers terminating their activity Survival rate(2) of young farmers' businesses after two years Survival rate of businesses in the agricultural sector in the region Jobs created Number of FTE jobs on the farm after Farm income two years Income growth in % two years after investment Number of FTE jobs in the agricultural sector in the region Average income per farmer (1) Farm type is the category of the main farm production, e. g. field crops, dairy cattle, horticulture. (2) A young farmer Is defined as a person under the age of 40 (35 in some countries) taking over or establishing a farm for the first time. Indicators for monitoring and evaluation Example 24: Supported investment in agricultural product processing plants Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Processing capacity Result indicators Leverage effect Local supply structure Impact indicators Direct employment Tonnes / m2 of capacity (broken down into different products) Complementary Investment made by addressees or recipients Number of supply contracts fixed with local farmers Number of net jobs created in the recipient firms after two years (FTEs incl. % held by women) Number of jobs in the agri-food industry Value added Increase In producer prices Increase in producer prices for agricultural products in the area Knock-on effect % of the region's eligible agricultural products bought by recipient firms after 2 years Example 25: Village renewal - public and private projects Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Quantity R e s u lt i n d i c a t o rs Population Satisfaction Impact indicators M2 of village squares renovated km of village roads improved Number of houses renovated Number of inhabitants living in the vicinity of rehabilitated areas (1) Number of inhabitants living in low density areas Number of inhabitants that are satisfied or very satisfied with the works Residential attractiveness % of inhabitants who answer "No" to Total migration from the assisted area the question "Do you wish to stay in the area in the next 5 years?" Average cost of a house in the rehabilitated village Average cost of a house in the assisted area (1) Vicinity should be defined as "less than 1 k m ". However, that definition must be adapted to practical situations Agriculture and rural development Example 26: Support for establishing tourist accommodation on farms Programme Indicators (Related to the Intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Number of farms Number of projects granted assistance Capacity Number of tourist beds created on farms Total number of tourist beds provided Result i n d i c a t o rs Diversification Impact indicators Number of overnight visitors Income Value added % of addressee or recipient who enter in the agro-tourism business as a consequence of the programme Number of overnight stays on addressees' or recipients' farms per year and after two years Overnight stays of tourists Increase in recipient farmers' income Average income per farm Gross value added generated by the programme (euros / year) after one year Value added generated within the agricultural sector Other interventions having an impact in the field Farmers, and even more so rural households, are often part of wider target groups in other fields. Support for businesses has an impact on job opportunities for rural households and provides alternative or additional income possibilities for farmers, contributing therefore to structural change in the agricultural sector. Similarly, interventions of the field of transport have impacts on transport costs for agricultural inputs and products. Additionally, activities in this field improve the access of rural households to job opportunities outside their area. Structural interventions in the field of tourism are in some cases related to tourist attractions in rural areas. Visitors may be interested in visiting other small rural tourist attractions. They may stay in these areas for longer periods for their holiday activities. The supply of regional agricultural specialities by farmers is one more opportunity to open up rural tourist potential. Training, especially in technologies needed in firms located in rural areas, again increases the possibility of members of rural households finding jobs. Indicators for monitoring and evaluation Availability and collection of data Most output and result related data should be available from the operators of projects funded through the Structural Funds, who should be required to routinely collect simple data on, for example, the number, size and type of farms assisted. Impact related data need to be collected from the farmers themselves in special surveys. Like other types of intervention, the involvement of outside teams to collect this information may be useful in ensuring that answers are reliable. It is important to use sample surveys and extrapolate from them, since when the cost of comprehensive surveys is prohibitive. the overall number of projects is high, indicators and calculating the possible Impacts on rural households and attractiveness of the region can be examined through comprehensive surveys. In some cases it is possible to rely on earlier analyses of measures. By adjusting coefficients with result or output impact of assistance, sufficiently reliable and cost-effective estimations can be made. This method is often used to identify the temporary employment effects of village renewal or to calculate the employment effects of agro-tourism (in FTEs). Some context indicators relate to the national or regional context as appropriate. Data for them may be available from European, national and sometimes regional administrative records, statistics and surveys. Eurostat has developed a special database of annualised regional statistics, providing Europe-wide comparable data. For agriculture and rural development there are some problems of statistical coverage. Information on small sized on-farm activities (e. g. agro-tourism) is not provided by statistics. Agricultural data on regional farm structure: age of farmers; size; farm type, is only available with a substantial time lag, which limits the possibilities is the context implemented. in which a programme for monitoring • Key indicators The key indicators presented below lend themselves to comparisons both between different interventions of the same programme and between those of different programmes, and can in certain cases be aggregated at a European level. Agriculture and rural development Level Key indicator Resource Output Result Impact Rate of consumption of budget (% of allocated funds) % of projects (in financial disadvantaged rural areas terms) concerning the most Selection rate (% of projects in financial terms accepted) Number of individuals receiving direct assistance or services as a result of the programme (incl. % of men / women) Number of economic units (farms, etc. ) receiving direct assistance or services as a result of the programme (large, medium, small, individual) Number of new economic units (tourist accommodation and attractions, new farms, etc. ) receiving direct assistance or services as a result of the programme Coverage (% of addressees or recipients, for example young farmers, of the total number of potential addressees or recipients) recipients situated % of addressees or disadvantaged areas Leverage effect accompanying the financial support received) (spending by addressees or in the most recipients % of assisted new businesses (diversified farms, campsites, farms taken over by young farmers, etc. ) that are still active after 24 / 36 months Gross value added generated (after 12 months in terms of euros / year / employee) Net employment created or maintained (FTEs incl. % held by women) after 12 months Residential attractiveness (% of inhabitants wishing to remain in the area) For more information see. Grajewski, R. , Schrader, H. , Tissen, G. (1995), Vorschläge für die Wirkungsanalyse und Bewertung von Mabnahmen zur Anpassung der Agrarstruktur und zur Entwicklung der ländlichen Räume in den neuen Bundesländern gemäb Ziel 5a und 5b der Reform der EG- Strukturfonds (1991-1993). Braunschweig. Riedel, J. , Scharr, F. , König, M. et al. (1997), IFO, Institut für Wirtschaftsforschung Niederlassung Dresden (Hrsg. ): Die Europäischen Strukturfonds in den neuen Bundesländern. Indicators for monitoring and evaluation Zwischenbewertung des Mitteleinsatzes von 1994 bis 1996. Dresden. Tissen, G. , Toepel, K. (1998), Europäische Strukturpolitik ländlichen Raum. Zwischenbewertung des Ziel-5b-Programms Schleswig- Holstein. Frankfurt am Main. im Young, N. , Isla, M. , Schrader, H. et al. (1991), Monitoring and Evaluation of Objective 5b Regions. Report Wye for the European Commission. Indicators for five priorities Equal opportunities Support provided under the Structural Policies may eventually have effects on the entire population, though these are generally not uniform across the social groups and may even exacerbate the situation of certain categories of addressees or recipients, particularly women. It is for this reason that the equal opportunity dimension of employment effects needs to be systematically integrated into evaluations. The European priority of the promotion of equal opportunities between women and men is closely, but not exclusively, linked to employment issues. In this context it does not seem relevant to develop numerous indicators for the issue of equal opportunities, but to consider equal opportunities as a mainstreaming issue, which means that as many indicators as possible should be broken down by gender. Box 17 - Gender mainstreaming The European priority in favour of equal opportunities for women and men should be systematically integrated into all programmes, measures and projects. The term "gender mainstreaming" means that the respective situation and needs of women and men are actively and openly taken into account at all stages of the programming cycle from ex ante diagnosis to ex post evaluation. In order to apply the concept of mainstreaming, those who design a system of indicators should, when relevant, break down figures into the categories of women and men. The mainstreaming principle may also be applied to other policy priorities, such as SMEs. Interventions within the field Interventions having the promotion of equal opportunities as their main objective are relatively rare. In fact, the importance of the issue requires that the promotion of gender equity affect all sectors, rather than focusing on a limited number of specific measures. Indicators for monitoring and evaluation Interventions having an impact in the field The following example is drawn from the field of urban development, but it could have referred to any other of the fields presented in this Volume. In terms of equal opportunities, the most relevant indicators are situated at the impact level, for instance the proportion of jobs created or maintained which are taken by women. However, in order to more adequately address this issue, indicators need to go further and provide information on the number of women managers and decision makers, as well as examining their pay levels in relation to those of male employees. According to the principle of gender mainstreaming, all indicators should be broken down into women / men so as to take gender issues into account. In practice, this implies that the percentage of women should systematically be counted for all relevant indicators, starting from impact indicators and going backward to result and output indicators, and even to resource indicators. This principle is demonstrated in the following example. Example 27: Esta blishment or re-generation of community centres Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Facilities Capacity Result indicators Satisfaction Impact indicators Employment Purchased, constructed or refurbished buildings in m: (area of which is devoted to meeting women's needs, i. e. nursery, etc. ) Maximum number of users that can be served (including the maximum number of users of resources specifically intended for women) % of users that are satisfied or very satisfied with the project (of which women / men) Maximum number of users in nurseries in the area Number of users that hold a permanent job (of which women / men) % of workers in the area who use nursery services Availability and collection of data The type of information held in monitoring systems should be improved in order to enable a breakdown of indicators according to gender whenever this is relevant. Equal opportunities Reports from relevant research and labour market analyses (including the results of previous evaluations of relevant interventions) provide a stock of both quantitative and qualitative information that may be used for estimating context indicators for equal opportunities. For instance, labour force surveys provide figures in terms of women / men on at least an annual basis for both the EU as a whole and individual Member States. If the monitoring system provides limited data (or even no data) about the respective situations of women and men, it may be decided to collect these data at the evaluation stage by surveying a sample of projects in order to reconstruct missing output, result, and impact indicators. Key indicators The key indicators presented below lend themselves to comparisons both between different interventions of the same programme and between those of different programmes, and can in certain cases be aggregated at a European level. Level Key indicator Resource Number of persons mobilised to implement the interventions in job years (incl. % of women) Output Result Impact Number of addressees or recipients (incl. % of women) Number of persons who have been immediately hired during the life of the project (incl. % of women) Jobs created or maintained after 12 months (FTEs, of which held by women) Number of users that hold a permanent job (of which women / men) For more information see. European Commission (1998. ) One hundred words for equality - a glossary of terms on equality between women and men. Luxembourg: OOPEC. Environment Structural Fund interventions concerning the environment may include environment-related infrastructure projects (e. g. the construction or enhancement of waste-management facilities), support of environment investments (e. g. assistance for the establishment of new related companies providing environmental services), or projects which are intended to develop environmental skills (e. g. training managers in more environmentally friendly production techniques, or training the unemployed in skills relevant to re-cycling). Much attention is also paid to ensuring that programmes supported by the Structural Funds result in sustainable development (i. e. that development takes place in such a way that it does not damage the environment or deplete limited resources, therefore allowing the needs of the present to be met without compromising the ability of future generations to meet their needs). In the Fifth Action Programme on the Environment which was adopted by Council in 1993 the notion of sustainability is defined as reflecting ". a policy and strategy for continued economic and social development without detriment to the environment and the natural resources on which continued human activity and further development depend. " Such efforts largely concentrate on mitigating the adverse effects of an activity on the environment. Mitigation consists of measures that avoid, reduce or remedy negative environmental impacts, or lead to a net improvement, therefore achieving sustainability. Interventions within the field Resource indicators At the programme level, resources devoted to activities having an explicit indicators of environmental objective can be monitored by using "compliance with project cost" (i. e. the most recently available information on the total cost of the projects in proportion to the cost which appeared in the programming document). Output indicators Examples of output indicators include the capacity and standard of new water treatment facilities, number and standard of training courses funded, covering relevant subjects, etc. These indicators are essentially usable for monitoring project completion and duration. Indicators for monitoring and evaluation Some output indicators can be related to indicators of endowment. For instance, the capacity of new waste water treatment plants may be related to regional endowment in terms of waste water treatment capacity. Result indicators Immediate benefits for addressees or recipients are easily measurable in the case of training and support to private investment. In these cases result indicators measure the new skills acquired or the new equipment constructed. In the case of infrastructure, result indicators measure improvements in the quality of services (i. e. drinking water respecting European standards) and/or lowering of prices charged to end-users. Impact indicators Regardless of the type of activity being assisted, its ultimate effect on the environment depends on the use which is made of the immediate output after several months or years. A re-cycling facility will only have an effect on the environment proportionate to the degree to which it is used. Similarly, new eco-industries will only affect the environment to the extent that they are surviving and developing, training in environmentally sensitive business practices will only be influential to the extent that a sufficient number of trainees apply their new skills. The types of impact which structural interventions have on the environment are extremely diverse and their identification should take place in the programming phase. This identification is related either to the explicit environmental objectives of the intervention or to established regulations and norms on the environmental management of publicly financed activity. It is important to take into account the fact that environmental impacts may occur in either the short or long-term and may be either discrete or cumulative. In 1994 a Commission memorandum noted that the measurement of changes in the overall environmental context, that are clearly attributable to assisted activity, "is difficult, time consuming and resource intensive for a number of reasons. ". The pressure that economic activities exert on the environment is easier to identify. In other words, since the environment is open to such a variety of influences, it is often simply impractical to establish an empirical connection between programme outputs and results, on the one hand, and impacts measured in terms of a changed state of the Environment environment, on the other. For example, it is probable that many general levels of atmospheric or water contaminants would not be noticeably influenced for a considerable time by even relatively substantial individual projects. The problems of defining impact indicators for the environment are less taxing in practice than they might be, due to the fact that the objectives of a structural intervention in the field of the environment are often expressed in terms of changing pressures on the environment. In the case of point sources of contamination, an impact indicator is simply provided by the measure of decreased pressure (reduction of a specified pollutant from the source in question). Impact indicators expressed in terms of reducing pressures on the environment could then be cross-checked with context indicators obtained from established data-series and also expressed in terms of pressure. Box 18 - The "Pressure-State-Response" model is one of a number of models The "Pressure-State-Response" framework developed for analysing environmental policies (others include the Driving Force- State-Response model, which also allows agri-environmental linkages and sustainable agriculture to be taken into account). It is not far from the "Result- Impact-Context" model that provides the basis of this document. Pressures are the driving forces that modify the environment (pollution, use of water). The state of the environment at a given time results from the combined action of three forces: the natural forces that govern the environmental system, the pressures of socio-economic activities and the responses of environmental policies. Both pressures on and state of the environment can be described with context indicators. Responses are the results and impacts of interventions that have an environmental objective. They may also be the results or impacts of the mitigation of non-environmental projects. Indicators for monitoring and evaluation Example 28: Improvement of water supply system Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output Indicators Progress Result Indicators Coverage Quality of service Impact indicators Consumption Indirect economic effect Compliance with project duration Rate of completion Number of households potentially served by new / improved networks % of households served by water supply networks Number of days with insufficient supply In quality or quantity (average number of days per 1000 households) % of regional households facing insufficient water supply in quality or quantity Volume of water consumed through new / improved networks (m3 / year) after one year Average water consumption per head per year. Number of economic units (firms, farms, etc. ) who declare that the new water supply system has released a major constraint for their development Water resources available per annum Environment Withdrawls In m3 per person per annum Environment Example 29: Construction of wastewater treatment plants. Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output Indicators Progress Capacity Result Indicators Quality of treatment Impact indicators Consumption Indirect economic effect Environment See Example 28 Number of new waste water treatment plants constructed Maximum annual throughput (m3) % of waste water undergoing primary treatment % of waste water undergoing secondary treatment (1) Volume of waste water treated Number of households / businesses connected to waste-water treatment facilities by programme after one year % of waste water treated % of households / businesses connected to waste water treatment facilities % of waste water samples from point sources downstream from the stations, showing a specified level of decline of identified pollutants See Example 28 % of km of rivers monitored on which pollution exceeds a specified standard (1) Quality standards for wastewater are set out in EC directive 76 / 464 / EEC and sister directives. Primary treatment is mechanical, allowing the physical separation of effluents; secondary treatment is biological, involving the use of micro-organisms. Indicators for monitoring and evaluation Example 30: Improvement of solid waste management facilities Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output Indicators Progress Capacity Result Indicators Coverage Impact indicators Solid waste collected for recycling Solid waste recycled for reuse as raw materials Indirect economic effect Environment See Example 28 Maximum annual throughput (tonnes) Number of households potentially covered by waste recovery collection services Amount of solid household waste collected for recycling in the areas of assisted projects (tonnes / year) after one year % of solid waste recycled for reuse as raw materials in the areas of assisted projects after one year % of unauthorised landfill sites closed / rehabilitated in the areas of assisted projects Amount of solid waste produced In the area (tonnes / year) % of solid waste recycled for reuse as raw materials In the area See Example 28 Number of unauthorised landfill sites in the area % of underground water sources suffering from pollution emanating from buried solid waste Environment Example 31 : Sup &se Dort for new SMEs providing environmental goods rvices. '1' Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output Indicators Volume of support services Number of addressees or recipients Result Indicators Diversification Certification Impact indicators Environment-related activities Environment related innovations Briefings with experts providing environmental advice within the programme (hrs) Number of firms receiving environmental advice from experts (at least 10 days or of a minimum value of 3,500 euro) Number of recipient firms newly established in the environment related sector % of firms qualifying for certification in respect to environmental norms Volume of business of assisted firms in the environment related sector after two years of operation (value added per year) Number of new products / processes introduced by assisted firms in the field of environment after two years Number of regional firms in the environment related sector Volume of environment related goods & services as a % of total of goods & services in the area (1) There do not appear to be any established standard definitions of an environmental good or service. Other interventions having an impact in the field At the resource level a specific problem appears when an attempt is made to measure the environmental cost of a given project, for instance, a motorway or an airport. It is not possible to measure mitigation efforts with separate costings systems. The environmental cost should be estimated twice: ex ante at the project selection stage and ex post at the programme evaluation stage. through standard accounting Whilst output and result indicators are specific to other fields (e. g. km of motorways, number of passengers), their monitoring is a basis for measuring the pressure on the environment and for quantifying impact indicators. It is important to bear in mind that environmental impacts may be due to both the immediate consequences of assisted activity (e. g. the damage Indicators for monitoring and evaluation item of caused to natural habitats by the construction of a major infrastructure such as a motorway) and the mid-term consequences of these or other actions (e. g. subsequent increases in air pollution due to increased volumes of traffic using the motorway). In addition to harmful impacts on the environment of non-environmental interventions, there are also instances where positive impacts on the environment occur, as in the case of the construction of an inter-urban light railway. Example 32: Construction of an inter-urban light railway between A and Β Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) O u t p ut indicators (in the field of transport infrastructure) Quantity | km of light rail network Result indicators (in the field of transport infrastructure) Speed E. I. S. S. between A and Β Average E. I. S. S. between all relevant urban centres In area I m p a ct indicators (in the fields of infrastructure and environment) Traffic flow Transport system Number of passengers using the railway line (km χ passengers per year) after one year % of passengers who declare that they would use a private car in the absence of the railway Reduction in volume of private vehicle traffic (vehicle χ km per year) after one year Environment Reduction in the total emission of NO2 & volatile organic compound (VOC) (K / year) Emission of NO2 & volatile organic compound (VOC) (K / year) Average national emission per vehicle χ km Availability and collection of data Output related information is often not available in a synthetic form for measures with an environmental objective. Indeed, data on the individual outputs of specific projects may themselves need to be synthesised in order to obtain a single indicator the "level of project completion". Indicators of "compliance with estimated project duration" should be a standard element in monitoring systems and provided as a matter of course at least every six months. Environment In some cases, results can be quantified relatively easily through indicators of demand (e. g. energy use in agriculture, consumption of water per capita); in other cases, ad hoc surveys of end users (e. g. reduced use of energy resulting from implementation of energy audits by companies) must be undertaken. In some instances monitoring systems may include information on a programme's environmental impact in terms of pressure changes (e. g. reduced output of pollutants from point sources). However, because of the relatively complex and novel nature of environmental impact assessment, in many instances custom surveys or investigations of the results of specific projects (particularly those for which a large-scale or concentrated impact has been foreseen) may be necessary. Data on changes in pressures on the environment may be available from established time-series context data on a national or even regional level. Context indicators refer to the general quality of the environment within the region. Time-series data providing information for this type of indicator are becoming increasingly harmonised, detailed and comprehensive. For the EU as a whole and for each Member State, Eurostat provides a growing number of data series in both printed and electronic form. The recently established European Environment Agency is also beginning to produce relevant work. On the national level a wealth of information may be obtained from the statistical services of the various Member States. However, established time series tend to be less available and less detailed on the regional level and custom surveys less relevant and practical. Key indicators The key indicators presented below lend themselves to comparisons both between different interventions of the same programme and between those of different programmes, and can in certain cases be aggregated at a European level. Indicators for monitoring and evaluation Level Key indicator Resource Output Result Impact Rate of consumption of budget (% of allocated funds) % of budget devoted to environmental mitigation measures Selection rate (% of projects accepted in financial terms) Rate of completion of project (% of objective) Compliance with project duration Number of potential connections (domestic / economic units) to networks of basic services (e. g. water treatment facilities) % of domestic / economic units receiving a level service satisfying European norms through the network (e. g. drinking water) in domestic / economic units (e. g. water Number of users connected to the new infrastructures, broken down treatment facilities) after one year Net employment created or maintained (FTEs of which held by women) For more information see. Department of the Environment (1993J, Environmental Appraisal of Development Plans, A good practice guide. London: HMSO. European Environment Agency (1995), Environment in the European Union, Report for the Fifth Environmental Action Programme, Copenhagen: EEA. Eurostat (1996), Indicators of Sustainable Development. Luxembourg, Eurostat. Eurostat (1996), Statistiques de l'environnement 1996. Luxembourg, Eurostat. OECD (1997), Environmental Indicators for Agriculture. Paris: OECD. Information society Structural include: interventions related to the "information society" infrastructure projects (such as the deployment and upgrading of telecommunications and telematics networks), direct financial support (e. g. in establishing new businesses in the multimedia industry), projects aimed at the establishment of "permanent conditions" for the development of Information and Communication Technology (ICT) applications (such as a telework or a telemedicine centre), and human resources development projects (e. g. training initiatives concerning new technologies). ICT In the area of interventions and applications ICT infrastructure interventions are perceptibly different. the to modernisation of a telephone network) infrastructure interventions in other fields, such as transportation, even though the services provided by new telecommunications infrastructures may not be as easily identified, especially as far as the advanced services (e. g. interactive multimedia services) are concerned. This means that result and especially impact indicators are often of minor significance. (e. g. they are quite similar infrastructures Interventions in the area of ICT applications development (i. e. telework, distance education, and telemedicine) are usually aimed at trials rather than large-scale diffusion. This means that it is rather difficult to evaluate the programme with reference to its context. For example, telework projects are presently conducted on an experimental basis and usually involve a limited number of workers. As a consequence, context indicators (in this case, the percentage of teleworkers in relation to the total number of employees of a given geographic area and / or industry) are usually not significant and need not be monitored. However, an evaluation of the potential contribution of a specific project to the general context (level of employment in the case of telework, level of qualification in the case of professional on-line education, and so on) could be provided, at least on a qualitative basis. Not all indicators for interventions in the field of the information society are fully accepted, because of the intangible nature of ICT-based services, the rapid development of new technologies and uncertainty about their organisational impact on complex systems like education or healthcare. A logic of experimentation is thus required in the monitoring and evaluation of such projects. Indicators for monitoring and evaluation Interventions within the field Resource indicators With reference to the distinction between telecommunications infrastructure interventions and application interventions (see above), resource indicators may include "compliance with project costs" for infrastructure interventions. In the case of application interventions the use of financial resources may fluctuate due to demand factors such as higher or lower (than estimated) participation of addressees or recipients in initiatives. Output indicators Interventions providing telecommunications infrastructure can easily be monitored with physical indicators, related both to the assisted project (like kilometres of fibre deployed or number of telephone exchanges equipped with devices based on new technologies) and to the general ICT context of the considered region (e. g. percentage of upgraded network). If service provision itself is included in the project, a measurement of the provided services (e. g. number of CATV - cable television - channels, hours of broadcast, or any other measure of available services) should be required as an output indicator. Outputs of interventions in favour of applications can be monitored by measuring the range of services offered, with a series of indicators that strictly depend on the specific application: number of telework stations in a telework centre, number of on-line courses provided, number of services provided through a network (where the term "service" must be precisely defined within the extent of the specific project). Result indicators Immediate benefits for end-users of telecommunications infrastructure can be in terms of network performance (bit rate, number of failures, etc. ). Results of application interventions can be monitored by programme indicators, such as the "number of addressees or recipients involved in the level of programme" (e. g. people trained exploitation of technology facilities (e. g. of telework stations in a telework centre), and the "level of satisfaction of the addressees or recipients involved". Applying result indicators to the context is usually not consistent, because of the experimental nature of applications programmes and the limited number of addressees or recipients involved. in new technologies), the Information society Impact indicators for impact indicators Connections to new or upgraded networks after 1 year and the level of usage of a network are telecommunications infrastructure interventions. While the indicator related to the number of connections can quite easily be applied both to the programme and to the context (in order to measure, for example, the penetration of ICT within households), generally the usage of a network cannot be applied to the context. In fact this indicator depends on the type of services provided through the network (CATV, for example, may or may not be an alternative to traditional TV, depending on marketing strategies of the broadcasters), and can be used only when the new network (and / or services provided through it) can be compared to the pre-existing one. Furthermore, the relationship between the network and its usage must be well known and assessed. For example, a higher speed of connection to the Internet (provided by a broadband network) may result in a reduced time of use (due to a more efficient and less time-consuming interactivity) or in an increased total time of use (due to a considerable number of new users attracted by better connection conditions). These relationships are still under assessment in the technology community and clearer evidence is required before developing appropriate indicators. Programme level impacts of application interventions can be evaluated through "survival" indicators focusing on how many addressees or recipients are still involved in the project after a given period of time (depending on the specific project). This type of indicator gives a credible measurement of the project's feasibility. They can also be evaluated by using "objective" indicators relating to the aims of the specific project (e. g. transportation time reductions for teleworkers, start-ups of "virtual" firms and so on). Context impact indicators, on the other hand, are usually not significant in the case of application interventions, because of the limited number of addressees or recipients involved in the experiment. In this case, a qualitative evaluation regarding the potential contribution of the specific application (e. g. a telework centre) to the context (e. g. increase or minor decrease of employment in a given region) should be provided. Indicators for monitoring and evaluation Example 33: Deployment of a broadband Hybrid Fibre Coax (HFC) network Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output Indicators Progress Compliance with project duration Rate of completion Quantity Number of HFC kms laid % of HFC kms in the total network (in the considered region) Capacity Number of buildings/apartments passed % of buildings / apartments passed Result Indicators Network performance Services supplied Impact indicators(l) Connections Bit rate capacity Number of network failures Number of services Number of TV channels Hours of broadcasting supplied (through the network) Number of households that have connected their equipment (TV, PC) to the HFC network (after 6 months) PC and / or CATV(2) penetration In households in the assisted area ICT usage Number of subscriptions to the services Total hours of connection / month Adoption rate of teleservices in households in the assisted area (1) Classification of the main teleservices can be found in the most relevant ICT surveys conducted at the European level (e. g. Measuring Information Society 1997, ISPO). (2) CATV stands for Cable television. Information society Example 34: Upgrading of an existing narrowband network with ADSL(1) technology Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output Indicators Progress Quantity Capacity See Example 33 Number of telephone exchanges equipped with ADSL devices % of exchanges equipped with ADSL devices in the assisted area Number of apartments potentially served by the equipped exchanges % of apartments potentially served by the equipped exchanges in the assisted area Result Indicators Network performance See Example 33 Supply of services Impact indicators Connections ICT usage Number of services provided (interactive TV channels, Internet access provision, etc. ) Number of households that have asked for ADSL connection (after 6 months) % of households that have asked for ADSL connection in the assisted area Number of subscription to services (after 6 months) Total hours of connection / month (after 6 months) (1) ADSL stands for Asymmetrical Digital Subscriber Loop. Indicators for monitoring and evaluation Example 35: Setting up of a suburban telework centre Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output Indicators Capacity Number of (tele) work-stations installed Result Indicators Quality of services supplied Impact indicators Participation Usage Transport times Business start-ups Employment Number of opening hours / week Number of support services provided (training, secretarial support, e-mail server, web hosting, etc. ) Number of firms participating in the initiative (after 6 months) Number of stable / mobile teleworkers(l) hosted (after 6 months) Exploitation of (tele) work- stations, i. e. time usage (after 6 months) Reduction in transport times for the teleworkers involved Time lost due to traffic congestion in urban areas Number of start-ups of new "virtual" firms Number of new businesses established in the region per year Number of jobs (FTEs incl. % held by women) created by start-ups of "virtual" firms Number of jobs (FTE incl. % held by women) created by new businesses in the region (1) Stable teleworkers are employees whose workstation is located in a telework centre (at least 3 days / week, according to the accepted definition of telework); the telework station is usually rented by a company on a contractual basis. Mobile teleworkers are workers who use telework facilities on demand, for a limited period of time (from 1 hour to 1 day). Since recent experiments have assessed that a telework centre is not yet profitable with only telework-related activity, most telework centres also provide facilities for traditional activities, such as on-site training courses and business support. The mix of services provided by the specific centre depends on the nature of business activity and on the market structure of business support in the region, and cannot be precisely monitored with structured and general indicators. Information society Example 36: On-line education project in professional training(1) Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output Indicators Activity of the operators Volume of training supplied Number of trainees Result Indicators Qualification Accessibility Flexibility Number of on-line courses % of on-line training in total training (planned) Number of hours of on-line training dispensed (hours χ trainees) Total annual number of trainees in the region Number of individuals trained (of which women) Number of trainees graduating in relation to total number of participants % of participants who declare that they could not have attended a traditional on site course because of distance constraints % of participants who declare that they could not have attended a traditional on site course because of time constraints Satisfaction of addressees or recipients % of trainees satisfied or very satisfied with courses Wastage Wastage rate in on-line courses Wastage rate in traditional courses Impact indicators Placement % of participants having found a job after 12 months (of which women) Annual rate of reemployment of unemployed persons in the region (1) On-line education usually refers to different intensities of ICT use in training activity, ranging from a complete on-line course to simple on-line tutoring complementary to on-site courses. A certain proportion of on site lessons is often required, in order to allow participants to get to know each other and thus enhance later on-line group learning. The mix between on-line and on-site training depends on specific course subjects, teaching methodologies (e. g. personal vs. group interaction) and the technologies adopted (the use of videoconferencing, for instance, reduces the need for on- site lessons). These elements are usually specified in the project documentation, and can thus be monitored. Other interventions having an impact in the field Interventions in other fields, which include ICT applications (even though not ICT-centred), may have a considerable impact on the development of the information society. A typical example is the use of ICT as an alternative way to provide information or to contact potential addressees or recipients (e. g. by building Indicators for monitoring and evaluation information a web site). Many kinds of business support include intangible services like information, orientation and training, that could be provided at least partially through the Internet. The same concept could be applied to delivery of traffic in urban traffic control projects and other similar interventions. Another example is the use of telematic networks to enhance (e. g. communications among epidemiology projects could use telematics to gather data from general practitioners). in a project the subjects involved In such cases evaluation may be conducted by using output indicators to measure the level of technological innovation in service provision (e. g. the number of business support services provided through a web site). Result indicators should measure the use of technology in the project by the addressees or recipients (e. g. to access information, to contact operators or to send data, as shown in the examples above). Finally, impact indicators should measure the impact of the project on the general acceptance, interest and use of ICT (e. g. % of addressees or recipients who declare to have become more confident after using IT technology during the project). Availability and collection of data Output related information is usually available or can easily be obtained in a concise version from the managers of the funded projects. Most result indicators should be quantified by the operators, through the monitoring of addressees or recipients, although not all indicators are fully accepted, especially the ones referring to the most innovative applications (like telework). Impact related data should be collected among the addressees or recipients through special surveys or case studies. This information is generally interesting for operators, who might thus be motivated enough to be involved in data collection. Case studies can provide better and more accurate information than surveys, especially in interventions concerning ICT application development, which are often specific and require in depth analysis. This confirms that qualitative data remain predominant in relation to quantitative data in the field of ITC as in many innovative policy fields. Context indicators for interventions in this field relate to the general ICT context of the region or regions in question. In case of interventions for ICT application development, as previously mentioned, context indicators are mainly qualitative and should be included in case study results. Context Information society indicators for telecommunications infrastructure interventions may be derived from European (e. g. ISPO and Eurostat) and national ICT surveys. Key indicators The key indicators presented below lend themselves to comparisons both between different interventions of the same programme and between those of different programmes, and can in certain cases be aggregated at a European level. Level Key indicator Resource Rate of consumption of budget (% of allocated funds) Output Result Impact Rate of completion of project (% of objective) Compliance with project duration Number of potential connections (businesses / households) to networks of basic services(1) Number of hours of training (on-line) received by participants of courses (incl. % of men / women) Number of domestic / economic units receiving a level of service satisfying European norms through the network Satisfaction rate (% of addressees or recipients of on-line courses) Immediate success rate (number of addressees or recipients qualifying at the end of the course) Number of users connected to the new infrastructures, broken down in domestic / economic units (e. g. HFC network) Gross value added generated (after 12 months in terms of euros / year / employee) Net employment created or maintained (FTEs, incl. % held by women) (1)When making comparisons between countries, attention must be paid to some country-specific elements, such as the digitallsation level of national telephone networks or the pre-existence of cable TV coaxed networks in some areas. Differences in ICT legislation (universal access, regulatory environment, data privacy laws, etc. ) must also be taken into account. Indicators for monitoring and evaluation For more information see. Eurostat (1997), Information and Communication Services - Annual Statistics 1980-1995. Luxembourg: Eurostat. Jackson, P. J. and Van der Wielen (1998), Teleworking - International Perspectives. London: Routledge. Smith, C. (1995), 'Measuring the usability and effectiveness of advanced in: Held, P. & Kugemann, W. F. (eds) learning technologies' the Telematics Telematics for Education and Training Conference, Düsseldorf / Neuss, 24-26 November 1994, Amsterdam: IOS Press. for Education and Training, Proceedings of Spectrum Strategy Consultants, DTI (1996,), Development of the Information Society - An International Analysis. London: HMSO. Thorpe, M. (1993), Evaluating Open and Distance Learning, 2° edition. Harlow (UK): Longman. Competitiveness of S M Es and enterprises in general Structural interventions in the field of enterprises consist of the provision of industrial sites, direct financial support and business support services. Business support services include assistance, information, advice and training, which is provided directly by business development agencies or indirectly by consultants, technical experts, etc. The ERDF and ESF are both relevant. Interventions concerning enterprises are also financed from EAGGF and FIFG. For the 1994-1999 programming period, 15 to 20% of all resources (between 23,000 and 30,000 million euros) are being specifically devoted to supporting SME development. The addressees or recipients of interventions are mainly SMEs and micro- enterprises with a priority to indigenous firms. Larger enterprises are also eligible for support, though eligibility conditions are much more restrictive than in the case of SMEs. The rationale for specific assistance to SMEs has several dimensions. One is that they are more exposed to economic fluctuations than larger firms, because of their narrower financial and human resource base and their focus on smaller markets. They also face greater difficulties in raising external finance, penetrating overseas markets and acquiring managerial and other key skills, so may have greater need for advice, guidance and training. They are generally more labour intensive and create more jobs per unit of output, and they may be more innovative, adaptable and flexible than larger firms. Since competitiveness of SMEs is a political priority for the European Union, systems of indicators should specifically record the situation of SMEs whenever it is relevant. According to the principle of "mainstreaming" (see Box 17), not only should output indicators be broken down into large/medium/small enterprise, but also result, impact, and even resource indicators. There are critical specificities in the monitoring and evaluation of support for enterprises that shape the systems of indicators. Firstly, the level of support is often small in relation to the activities of the recipient firm and this raises difficulties in understanding its consequences, even for the manager. Some interventions provide indirect support to a group (a network) of enterprises and this leads to contact with the addressees or recipients being lost. Furthermore, the evolution of an enterprise in a competitive environment is very much driven by the context, making it difficult to track the specific consequences of public interventions. Lastly, the relevant territory for an enterprise often has nothing to do with the eligible area. Indicators for monitoring and evaluation • Interventions within the field Resource indicators Many business support services are demand-led, so take-up and the resulting indicator of financial consumption can fluctuate over time according to factors such as the business cycle and macro-economic conditions. Output indicators Outputs of interventions concerned with providing physical infrastructure can be monitored with a physical indicator like "floor space (m2) of premises provided or refurbished". However, many types of business support are essentially services (e. g. information, advice, management training and consultancy) rather than physical products (e. g. roads or factory buildings). In these cases, output indicators may measure the activity taking place, the scale of services provided (e. g. number of business advisers, number of one-stop support offices created), and the volume of services financed. In the general case of financial or service support, the main result indicator is the "number of businesses assisted". This is a rather basic indicator with limited value on its own. However, when combined with the level of financial resources spent on support, it gives a useful indication of intensity of support, which may be helpful for comparative purposes. This indicator can be broken down into categories like the average size and anticipated size of the assisted firms (e. g. in terms of the number of employees), the present or future activity of the assisted firms (e. g. in terms of growing or declining markets). Result indicators indicators provide Result productive structures in terms of business formation and diversification. immediate changes information about to "amount of expenditure The indicator expenditures) measures the "leverage effect". incurred by firms (investment or other such in relation to the support received" The "proportion of assisted businesses in the defined area or group" is also a relevant indicator for interventions in this field. This indicates the degree of penetration of the programme among the total number of businesses in the defined area, group or sector. Competitiveness of SMEs and enterprises in general A further result indicator is the "proportion of addressees or recipients reporting high satisfaction ratings". The indicator will have to be based on a follow up of addressees or recipients at the end of their contact with the programme. Impact indicators Interventions concerned with providing physical infrastructure produce specific impacts that can be measured with a physical indicator like the "floor space (m2) of premises which have been bought or are being rented by recipient firms after one / three years". Support for firms may have an impact on the attitudes and behaviour of the people working in them, and on various aspects of the performance of their businesses. It may increase the knowledge and skills of owner- managers and workers, or help improve business planning, management, marketing, products, and production and distribution systems. The ultimate objective is to assist companies to expand and create (or to prevent decline of) market shares, turn over, value added (in practice, measured in terms of wages and salaries plus profits) and eventually to create employment impacts. to Impact indicators must incorporate the time-scale over which effects are produced (normally one to three years after the intervention). Note that these time periods are likely to extend beyond the life of the overall SPD / regional programme, creating management difficulties. Support may also have knock-on effects on the performance of suppliers (increased output), customers (improved quality and / or price) and competitors (reduced sales). These need to be considered when assessing the overall net impact of the interventions. There may also be wider multiplier effects arising from the employment and income created in assisted businesses. However, the principal objective of most firms is to increase profits and few businesses have employment as their prime objective. In most instances employment is a derived demand arising from the need to employ people to get work done. Businesses may perceive there to be a trade-off between employment and profits because of the cost of wages. In some cases (e. g. assistance towards technical innovation in products and processes, or improved management and production systems) support may improve business efficiency and profitability, but not create any additional jobs, especially in the short-term. It is generally hoped that such improvements in competitiveness will generate wider economic benefits in the longer term. Indicators for monitoring and evaluation Contextual conditions are also in a depressed economic environment, the impact of the same level of support on output and employment may be considerably less than in more buoyant conditions. Sensitivity to these circumstances is important in monitoring and evaluating programmes. important: Some firms succeed through low prices, achieved by keeping costs down (including labour) and perhaps sacrificing environmental standards, while others succeed innovation, product differentiation and specialisation, and high skills and productivity. Different combinations of impact indicators should detect these differences. through Example 37: Support for entrepreneurs creating new businesses'1' Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Number of operators Number of one-stop support offices Volume of services provided Result indicators Business creation Satisfaction rate Impact indicators New business survival Direct employment Number of hours of expert advice received by addressees or recipients wishing to launch a business Number of new businesses launched following assistance (¡nel. % owned by women) Business creation rate in number of business / year per 10 thousand ' inhabitants (incl. % launched by j women) % of addressees or recipients who are satisfied or very satisfied with the support services provided Survival rate(2) of assisted business start-ups after 18 / 24 / 36 months (of which created by women) Number of net jobs(3) in the assisted new businesses after 18 months (FTEs incl. % held by women) Business survival rate after 1 8 / 2 4 / 36 months (1) New businesses are defined as commercial enterprises of various kinds (including self-employment) that have commenced trading and which have been helped to start under the relevant programme. They do not include people who were on the programme but dropped out, or who are yet to start. (2) The survival rate is the proportion of new businesses that are still trading after 18 months. This is beyond the normal subsidy period and is sufficient to gauge whether or not the business is basically viable. (3) Employment should be measured In terms of full-time equivalent (FTE) jobs generated as a result of support provided under the programme. Part-time jobs may be converted into FTEs on the basis of two to one. Jobs can be defined as permanent If they continue after the end of the intervention or if they last for a pre-defined period after the end of the intervention. Competitiveness of SMEs and enterprises in general Example 38: Informational support for SMEs to promote exports Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Number of addressees or recipients Result indicators Satisfaction rate Geographical diversification I m p a ct i n d i c a t o rs Exports(3) Number of assisted SMEs (1) (2) Number of SMEs in the eligible area % of addressees or recipients who are satisfied or very satisfied with the support services provided Number of SMEs becoming new exporters Number of SMEs exporting to new markets % of export sales in the turnover of assisted SMEs after 18 months Exports of SMEs related to GDP of the area Value added Value added generated after 18 months Direct employment Number of net jobs created / maintained (FTEs incl. % held by women) in firms in relation to export sales arterie months Average value added by employee in the area Total number of unemployed In the assisted area (1) The European Commission has agreed a standard definition of SMEs as: employing less than 250 people; having an annual turnover of less than 40 million euro or balance sheet assets of less than 27 million euro, and being Independent, or having no more than 2 5% of their capital owned by a larger firm. Small enterprises are defined as: employing less than 50 people; having a turnover of less than 7 million euro or assets of less than 5 million euro, and being 'independent'. Micro-enterprises employ less than 10 people. (2) A minimum definition of assistance is required to avoid five-minute conversations with businesses being counted as support. Within the framework of UK objective 2 programmes, " assistance " is a minimum of 5 days of support, or its financial equivalent (#3,500 euro). (3) Export sales are defined as the value of all sales and contracts by businesses to consumers, companies and organisations outside the region concerned. A further indicator may be the value of sales beyond the country concerned. Indicators for monitoring and evaluation Example 39: Subsidised infrastructure for the setting up of firms Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Quantity Hectares of industrial sites made available for firms Total area of industrial sites in the area Number of addressees or recipients Number of assisted projects (of which SMEs) Result indicators Leverage effect Investment by recipient firms Impact indicators Value added See Example 38 Direct employment Knock-on effect Number of net jobs created (FTEs of which held by women) in the recipient firms alter 18 months Regional firms as a % of suppliers to recipient firms after 18 months See Example 38 See Example 38 Other interventions having an impact in the field Interventions in other fields such as RTD and transport are likely to have an impact on the competitiveness of SMEs and larger enterprises. Impacts of interventions in the transport field are often indirect and mostly occur through macro-economic mechanisms. They are therefore very difficult to monitor, or even to evaluate, through indicators. Indicators are more likely to be useful in the case of RTD interventions. A frequent situation is one where an intervention is targeted at research centres, with the aim to indirectly strengthen the innovative capacity of regional enterprises, and to ultimately increase regional competitiveness. In this case, it is possible to use specific impact indicators in an evaluation survey. Competitiveness of SMEs and enterprises in general Example 40: Support for projects in regional research centres Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators (in the field of RTD) Number of projects assisted Number of research projects assisted Number of research centres in | the area Result indicators (in the field of RTD) Project progress Networking Impact indicators Economic benefits Value added(1) % of projects successfully concluded with a publication in a refereed journal Number of regional enterprises involved in supported joint research projects (of which SMEs) Number of regional enterprises that have been Involved in supported research projects and that declare positive spin-offs after 18 months (of which SMEs) Total value added arising from supported research projects (1) The value added generated by a research project is likely to occur over a long period. In order to sum up the declarations of several enterprises, annual value added should be weighted by using a discount rate. The indicator of value added should also be weighted by using coefficients of likelihood when possible future spin-offs are estimated. Availability and collection of data Most output and result-related data should be available from the managers of projects funded through the Structural Funds. They should be required routinely to collect simple data on, for example, the number, size and type of businesses assisted, and satisfaction with services provided. This should also apply to businesses indirectly assisted by the supply of industrial premises or by supporting networks of firms. Standard breakdowns should be systematically applied, at least for the size of businesses (0-10 employees, 11-50, 51-250, more) and for the sector (growing / declining demand). Impact related data need to be collected from the businesses themselves in special surveys. The use of outside teams to collect this information may be useful in ensuring that answers are reliable. The cost of comprehensive surveys is likely to be prohibitive, so it is important to use sample surveys and extrapolate from them. Due regard should however be paid to the fact Indicators for monitoring and evaluation that extrapolation involves a risk of error. The complexity and sensitivity of the issues surrounding causality and attribution of effects mean that focus groups and face-to-face interviews with businesses are more likely to get closer to the truth than postal questionnaire surveys. Telephone-based interviews may be a compromise. It is important that decisions about survey techniques are made locally in the light of their cultural suitability (see Volume 3 of the MEANS Collection). Context indicators refer to the regional or national context as appropriate. They may be available from European and national administrative records, statistics and surveys (e. g. those collected for taxation purposes). Eurostat has developed a special database of regional industrial statistics with annual data since 1994. Key indicators The key indicators presented below lend themselves to comparisons both between different interventions of the same programme and between those of different programmes, and can in certain cases be aggregated at a European level. Competitiveness of SMEs and enterprises in general Level Key indicator Resource Rate of consumption of budget (% of allocated funds) Output Result Impact Number of contacts between operators and addressees or recipients (of which SMEs) Number of project applications (of which by SMEs) Selection rate (% of projects in financial terms accepted and % of which are proposed by SMEs) Selection rate for projects in rapidly growing sectors (in proportion to the average selection rate and % of which are proposed by SMEs) Number of hours of expert advice received by addressees or recipients (e. g. to launch a business) Number of firms receiving direct assistance or services as a result of the programme (% of which SMEs) % of recipient firms active in rapidly growing sectors (% of which SMEs) % of recipient firms involved in high-tech projects (% of which SMEs) Satisfaction Rate (% of addressees or recipients satisfied / very satisfied by services provided) Leverage effect (private sector spending generated by the programme in relation to financial support received) % of assisted new businesses that are still active after 18, 24 and 36 months Value added generated (after 18 months in terms of euros / year /employee) Net employment created or maintained (FTEs, % of which are in SMEs / of which held by women) Regional knock-on effects (regional firms, % of which SMEs, as a % of suppliers to assisted businesses after 18 months) Indicators for monitoring and evaluation For more information see. Bachtler J. and Michie R. (1995), 'A new era in EU policy evaluation? The appraisal of the Structural Funds' in: Regional Studies, Volume 29, 745-751. Foley P. (1992), 'Local Economic Policy and Job Creation: A Review of Evaluation Studies' in: Urban Studies, Volume 29. HM Treasury (1991), Economic Appraisal in Central Government: A Technical Guide for Government Departments. London: HMSO. McEldowney, J. (1991), 'Evaluation and European regional policy' in: Regional Studies, Volume 25, 261 -268. Turok, I. (1997), 'Evaluating European Support for Business Development: in: the Structural Funds Evidence Entrepreneurship and Regional Development, Volume 9, 335-352. in Scotland' from Urban development The definition of urban development employed here is "the set of activities which affect individual and social welfare through changes in the nature or intensity of land use in urban areas, and the regulation of those changes". Within the framework of the Structural Funds, interventions have four major objectives in this field. The first objective is to generate an economic impact in the form of jobs and enterprise development, based on the premise that every individual should in an income-earning activity and to find a be able to participate remunerative job. Every to individual should have equitable access opportunities, including employment, health care and schooling, without regard to social or ethnic origin or gender. The second objective is therefore to generate a social impact. A third objective is that of generating an environmental impact in the sense that urban activities have an effect on the natural and built environment in urban areas and on the use of physical resources. The development of urban activities supported by an intervention should be sustainable (i. e. it should respect the natural and built environment in urban areas and not lead to the depletion of natural resources). When negative environmental impacts do occur, they should be the object of mitigation measures. Last but not least, interventions have the objective of generating a civic impact, allowing individuals to take an active part in the processes which fashion their lives and their urban environment. The logic of urban development often mixes these four dimensions, making sector-based indicators largely irrelevant. Some Structural Fund interventions are specifically targeted at urban development. This is the case for the URBAN Initiative and the Urban Pilot Projects and many Objective 2 programme measures. The aim of urban development interventions is often twofold: fostering the development of assisted towns and cities and reducing discrepancies between poor and privileged areas within them. These two objectives are generally seen as complementary rather than competing. When the objective is global urban development, interventions mainly have an economic and social dimension. When the programme aims at integrating poor areas into the city, most interventions tend to develop social activities and citizenship. Most Structural Fund interventions, which may seem to be largely irrelevant to issues of urban development at first, can sometimes have a significant Indicators for monitoring and evaluation impact on urban areas. For instance, structural adjustment occurring in the fisheries and agricultural sectors can significantly affect the welfare of major coastal towns and important market towns in rural areas. The monitoring and evaluation of support for urban development has a number of specific characteristics. Firstly, the relevant areas of urban development are often small, local and do not correspond with administrative or statistical boundaries. Secondly, urban development encompasses a very broad range of aims and activities, from the most technical (e. g. providing premises to new enterprises), to the most political (e. g. enhancing citizenship in depressed areas). Interventions within the field Resource indicators financial consumption should be broken down Indicators of into expenditure benefiting different towns and cities and expenditure benefiting poor areas within the assisted towns and cities. However, unless this has been planned from the beginning of the programme, this break down will not be quantified by the monitoring system and an estimate will have to be provided through a special study at the evaluation stage. Output indicators indicators Output like completion rates and compliance with project timetable are useful for physical projects such as the reclamation of derelict land or housing renovation. Output indicators will also include a physical measure of the project such as the "number of houses renovated" or "hectares of derelict land reclaimed". Many interventions targeted at urban development offer services (e. g. types of schooling, housing, and interventions, urban development is monitored by indicators focusing on the activity of the operator and the volume of services provided (e. g. number of advice sessions χ hours). In the case of training). these Some output indicators can be monitored at the programme level in the form of the proportion of a given output produced in urban areas or in poor areas within the assisted towns and cities. Nevertheless, output indicators may fail to convincingly separate out the urban part of a development programme. For instance, a training course that is organised in the business centre of a city may recruit trainees in the poorest boroughs. Urban development Result indicators In the case of urban renewal projects, the distinction between direct and indirect addressees or recipients can be problematical. For example, the redevelopment of a derelict industrial site or an area of poor quality housing improves the urban environment of those inhabitants in the immediate vicinity. It is for this reason that inhabitants within a given distance (often defined as 1 km) of the project are considered as addressees or recipients of an intervention. Therefore, the number of neighbouring inhabitants is a result indicator. Impact indicators It is relatively easy to build indicators for the specific impact of a given measure. For instance, addressees or recipients may be asked to rate their feeling of segregation or the quality of environment. Nevertheless, difficulties may occur since urban development programmes are typically made up of many small measures (or even of a simple collection of projects) which are supposed to produce a global change in the context rather than specific impacts on addressees or recipients. Consequently, much attention should be paid to the choice of impact indicators, which can be related to both the programme and the urban context. Example 41 : Cre; coui ition of new organisations providing debt iselling and local credit schemes Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted urban area) O u t p ut i n d i c a t o rs Number of addressees or recipients Number of organisations and schemes receiving support Result indicators Credit cost I m p a ct i n d i c a t o rs Use Indebtedness Average interest rate offered in % of standard commercial rates Number of local inhabitants using supported debt counselling and local credit schemes per year after one year % of users who have succeeded in eliminating their debts after one / three years Number of local inhabitants Indicators for monitoring and evaluation Example 42: Renovation of a rundown urban area Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted urban area) Number of housing units renovated or built | Number of shop units renovated or built Number of housing starts in the area Output indicators Quantity Result indicators Quality of urban environment Estimated number of residents located in the vicinity(1) of the renovation area Number of inhabitants in the area Impact indicators Number of addressees or recipients Value added Employment Residential attractiveness Health Number of inhabitants buying or renting housing units Number of businesses setting up in renovated or new units after 1 year Value added generated in local businesses after one / three years Number of inhabitants in the area Total value added in the area Number of jobs created or maintained in the local area (FTEs of which held by women) after one / three years Number of jobs created in the area(FTEs of which held by women) after one / three years % of residents located in the assisted urban areas who declare that they wish to remain in the area in the next 5 years Number of local inhabitants who have had diseases associated with poor quality housing (1) In terms of residential areas, vicinity can be defined by 'a distance of less than 1 kilometre". However, this definition must be adapted to practical situations found in residential areas. Urban development Example 43: Support for community organisations Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted urban area) Output indicators Number of recipient organisations R e s u lt i n d i c a t o rs Focus on less favoured areas I m p a ct i n d i c a t o rs Number of community organisations that have had their project accepted (with less than 5, between 5 to 25, more than 25 unpaid volunteers) % of assisted organisations that are located in poor urban sub-areas or whose members live in poor urban sub-areas Activity of community organisations Number of adults active in assisted community organisations after one year Civic participation Segregation / exclusion % of adults active in assisted community organisation and who declare that they 'feel segregated / excluded' Number of adults In the urban area active in at least one community organisation % of voters who voted at latest local elections % of all citizens who declare that they 'feel segregated or excluded' Other interventions having an impact in the field A typical example of an intervention occurring outside the field of urban development, but which creates impacts within the field is the case of training programmes for young unemployed workers. In this instance, the initial round of impacts can generally be evaluated with an indicator of the "number of addressees or recipients finding a job within χ months". However, many of the second and subsequent rounds of impacts, whether associated directly or indirectly with urban development (e. g. respectively an increase in the income of addressees or recipients giving rise to increased private investment in housing in the target area, or the demonstration effect whereby employers learn to have a less negative stereotype about the potential contribution of young workers who were previously unemployed), do not lend themselves to quantification in the form of indicators. In order to be practical, monitoring will normally be limited to results and to the first round of impacts. In many cases, only in-depth and qualitative evaluation studies will be able to track second and third round effects and convincingly assess if urban development goals have been attained. Indicators for monitoring and evaluation Example 44: Training programme for long-term unemployed(l) Programme Indicators (Related to the Intervention and its effects) Context Indicators (Related to the assisted area) Output indicators (in the field of training) Number of trainees Number of individuals trained Result indicators (in the field of training) Qualification Number of trainees successfully completing the course Satisfaction of addressees or recipients % of trainees satisfied or very satisfied with courses Number of long-term unemployed in poor urban sub-areas Impact indicators Unemployment Poverty Number of trainees who live in poor urban sub-areas and who are no longer unemployed after one year (incl. % of women) Average proportion of long-term unemployed who stop being unemployed within one year Number of trainees who live in poor urban sub-areas and who no longer receive any selective welfare benefit after one year (incl. % of women) % of households in urban areas receiving at least one selective welfare benefit (1) Long-term is defined as a period of twelve months or more. Examples of selective welfare benefits are, for instance, RMI in France, Minimex in Belgium. Availability and collection of data There are five statistical bases for the concept of the town: urban localities (defined by the town's administrative boundaries or by its status in law), sub urban areas (often defined by statistical offices), urban agglomerations or urban units (embracing continuously built-up urban centres forming either part of one administrative unit or a group of several units), urban regions (comprising a nucleus town and its sphere of influence or employment catchment area, frequently defined in terms of commuting), polynuclear urban regions, or conurbations (these may be continuously built-up but comprise a number of centres around which social and economic activities are polarised). In most countries, each of these levels is chosen for the collection of at least some statistics on urban development activities, depending on their purpose. Operators are unlikely to spontaneously adopt common definitions of urban areas when they collect data. Therefore, programme managers must make a very strong effort in terms of co-ordination if they want to develop monitoring indicators for urban development (output and Urban development results). Thus, an indicator of urban development should not be quoted without making clear what definition of urban has been chosen. As far as possible, urban areas should be defined in accordance with administrative units since the range of statistics collected tends to be greater for such units. Moreover, these limits are relevant for the authorities that participate in "structural interventions" and in local policy initiatives. When evaluation data are collected through special surveys, it is easy to design ad hoc territorial units. However, statistical units must be strictly respected if survey data are to be compared with context indicators. Box 19 - Indicators of the urban audit Themes Issue Socio-economic aspects Income levels Structure of unemployment Example indicators Total GDP for a city area / total resident population % of unemployed population out of work for more than six months Civic involvement Participation in elections % turnout in last election Levels of training and education Vocational skills Access to childcare Environment Traffic patterns Green areas and population density Recreation and culture Library facilities % of unemployed undertaking vocational training Number of children aged between 18 months and school age Number of public and private crèche places % of labour force travelling to work by car, by public transport (bus,. tram, train) and by foot and bicycle Total residential population / total land area in hectares % of local residents registered with libraries Indicators for monitoring and evaluation National and regional statistics are a source of context indicators. For instance, a club of 30 German cities has developed a statistical database on urban development, including very relevant indicators like opinions about social and ethnic integration. There are also numerous ranking exercises of European cities and local regions across a very wide range of indicators4. A broad range of contextual indicators is available from official publications. Key sources of indicators include the following: Eurostat5, DGXVI's publications, DGXI's Urban Environment Unit and Local sustainability database (DGXI: http://cities21. com/europractice), and the European Environment Agency. DGXVI is building upon this work with an "Urban Audit" which compares a large number of European cities. This type of work will provide a new impacts of resource against which to benchmark and evaluate the Structural Fund interventions as well as a bridge to begin linking project impacts to wider imperatives (see Box 19). Key indicators The key indicators presented below lend themselves to comparisons both between different interventions of the same programme and between those of different programmes, and can in certain cases be aggregated at a European level. 4 This includes the ambitious work of Paul Cheshire, Dennis Hay and Gianni Carbonaro, which examines 103 functional urban regions, using a database of 200 variables aggregated into four indexes on income, unemployment, migration and travel. The league table had Frankfurt, Munich and Venice at the top and Glasgow, Sunderland and Liverpool at the bottom. 5 Key Eurostat sources Include Portrait of the Regions and RegioMap. Urban development Level Key indicator Resource Output Result Impact Rate of consumption of budget (% of allocated funds) % of projects (in financial disadvantaged areas terms) concerning the most Selection rate (% of projects in financial terms accepted) (men / women) Rate of completion of project (% of objective) Compliance with project duration Number of hours of services (training, counselling, etc. ) received by addressees or recipients Number of individuals receiving direct assistance or services as a result of the programme (of which male / female), including the number living in poor urban sub-areas Satisfaction Rate (% of addressees or recipients satisfied / very satisfied by services provided) % of addressees or recipients not needing the help of services after 18 months % of addressees or recipients employed after 18 months (in a stable long-term job) Residential attractiveness (% of inhabitants wishing to remain in the area) For more information see. Bentham, OG. (1985), 'Which Areas Have the Worst Urban Problems ?' in: Urban Studies, Volume 22 ,119-131. Coombes, M. Raybould, S. and Wong, C. (1992), Developing Indicators to Assess the Potential for Urban Regeneration, Department of the Environment Inner Cities Research Programme. London: HMSO. European Commission (1997), Community involvement regeneration. Luxembourg: Official Publications Office of European Community. in urban the European Commission (1997,), Europe's Cities: Community Measures in Urban Areas. Luxembourg: Official Publications Office of the European Community. Healey and Baker (1997), Business Base Across the World. London: Healey. Indicators for monitoring and evaluation OECD (1997), Seffer Understanding Our Cities: The Role of Urban Indicators, Paris: OECD. Robson, B. Bradford, M. , Deas, I. Hall, E. Harirson, E. Parkinson, M. Evans, the R. Garside, R Harding, A. & Robinson, E (1994), Assessing Impact of Urban Policy. London, HMSO. Indicators for three major aims Economic d e v e l o p m e nt One of the major aims of structural interventions is to promote regional and local economic development. Interventions financed by Structural Funds also pursue global objectives like creating job opportunities and reducing disparities in the quality of life. However, because these aims are considered as both social and economic goals, they are the subject of information sheets. The ultimate goal of economic development is overall income growth of the population in assisted areas. According to a number of assumptions, some of which will be discussed later, the growth in the volume of regional economic activity (i. e. the development of the economy) is a proxy for the income growth of the population. Economic measures dominate most structural programmes. As for non- economic measures, these too are often guided by economic considerations at the design and implementation stages. This focus on the economic dimension as the main catalyst of regional development gives economic indicators a particular status. The most direct contributions to economic development take place in fields such as transport infrastructure, energy, information society, training, RTD, enterprises, tourism, agriculture and fisheries. Many themes covered in this Volume interact with the economic sphere and involve various factors that drive economic performance (e. g. attractiveness, accessibility, etc. ). Consequently, most of the related indicators are discussed separately in the relevant sections of this Volume. This section focuses on economic indicators stricto sensu, which include income, GDP, value added, investment, exports, etc. Since effects on regional and local economic development arise at the impact level from interventions in other fields, the section "Interventions in the field" is not applicable here. • Impacts on economic development Impact indicators Indicators that measure economic development primarily focus on impacts, and especially the most indirect impacts that are of a macro- economic nature. Indicators for monitoring and evaluation In order to evaluate the comparative performance of a region, a large number of context indicators are available at the sub-regional, regional, national and European levels. The main indicator for appreciating economic development is GDP per capita, which measures the total value added generated by a region's economic units in relation to the number of inhabitants. Since value added is eventually distributed in the form of wages, interest, taxes and so on, GDP is considered as a proxy for regional income. However, regional GDP per capita does not correspond exactly with regional financial resources distributed to the regional population. It is also important to bear in mind that inter-regional social transfers occur and that a proportion of shareholders, and to a lesser extent employees of regional enterprises, are not necessarily resident in the region. individual Since GDP is an aggregate of value added generated by economic units, the use of both of these indicators creates interesting links between programme impacts and the regional context. The indicator of "value added generated by supported projects" can be viewed as the micro-economic counterpart of the regional GDP. However, the micro- macro link is subject to much reservation. The different ways of measuring value added are presented in Box 20. Box 20 - Value added: a key impact indicator Value added is a key impact indicator as it not only captures the wealth generated in the local area through Structural Fund interventions, but can also be used to assess the 'quality' of employment opportunities. Value added can be defined and measured in a number of ways and at a number of levels. For example, at the level of an individual economic unit, it can be measured as the: • difference between the revenue generated by the sale of output and the value intermediate goods) inputs (raw materials, services and of non-labour consumed to generate that output; or as the: • combination of returns to factors of production employed. This comprises the wages and salaries paid to labour, plus the profits generated for owners, plus rental paid for the use of land. In practice, value added is measured in terms of wages and salaries plus profits. The aggregation of value added across all productive sectors of an economy can be termed Gross Domestic Product (provided all payments to factors are remitted within the region) which is an indicator of regional wealth generation. Economie development An initial problem is that the expected impact can arise a long time after the intervention, since structural adjustment is a long-term process of adaptation involving social, technological and economic dynamics. The effect on wealth creation can only be recorded after production processes have been effectively transformed. Thus, the economic impact of an investment subsidy will only materialise after the supported investment has come into operation, a process that may take several years. Furthermore, economic performance depends to a large extent on the prevailing macro- economic conditions. The effectiveness of interventions will be greater in periods of sustained economic growth than when the economy is going through a period of recession. For example, the leverage effect of investment subsidies is likely to be greater when a high level of demand can be forecast over a significant period rather than when demand is declining or weak. Box 21 - Regional disparities How can indicators give a clear view of the global economic performance of a region? Given that regions are the subject of a wide variety of external pressures (economic climate, international competition, national and supra-national regulation and policy decisions, etc. ), few clear-cut conclusions can be obtained from the analysis of absolute performance. The construction of indicators in relative terms is more appropriate. Considering the economic performance of a sub-region (for example, a NUTS 3 region eligible for Objective 2), disparities can be measured by referring to the region (for example, the relevant NUTS 2 region), the relevant national state and Europe. But why is it necessary to take so many referentials into account? This procedure allows, for example, observers to see if the region is performing comparatively badly in relation to both the national and European economies, but has a better performance than the region to which it belongs. More specifically, consider a sub-region for which both GDP per capita and GDP growth are inferior to these observed for the region, the state and Europe. is implemented which leads to a reduction in the difference between the sub-region and its region but not vis-à-vis the state and Europe. Such an observation can give an indication that the first step is being successfully undertaken in this sub- region (compared to the region) and some improvement can be expected in the near future (compared to the state and Europe). its development, a policy to stimulate intervention In order It is useful to recall that indirect and induced quantitative impacts of measures are generally larger than direct ones. Subsidies and services financed by the Structural Funds can be expected to generate additional Indicators for monitoring and evaluation to creating value added, generating turnover, which will contribute additional incomes in the region and benefiting regional businesses, and so provide added impetus to a new round of growth within the regional economy. If Structural Fund support leads to the firm ordering from regional sub-contractors (knock-on effect), this demand will also generate additional value added. This process, by which an injection of expenditure creates a subsequent series of waves of investment and consumption, is known as the "Keynesian multiplier". Only econometric models or input-output analysis are able to assess the links between the value added generated by the programme at the micro level and GDP at the macro level. Because this type of analysis is time consuming and expensive, related indicators cannot be quantified on a regular basis. Since the development of regional economies is influenced by a large variety of external pressures, few clear-cut conclusions can be obtained from the analysis of their absolute performance. The construction of indicators allowing the relative evolution of regional disparities to be compared is more appropriate (see Box 21). Result, output and resource indicators The micro-macro problem can be addressed in a more operational fashion through the following proposal. Instead of summing up individual impacts to more and more macro levels, a reverse approach can be adopted that is based on the mainstreaming principle. This principle has been applied in the field of equal opportunities. Its use implies that focus is primarily put on a given strategic priority. Indicators are broken down in accordance with the priority when this is relevant. Context indicators take into account a priority at the macro level and programme indicators at micro level. In the example of the promotion of economic diversification (illustrated in Box 22), activities in rapidly growing sectors are taken into account by breaking down data at each stage of the causal chain. Consequently the resources devoted to interventions in rapidly growing sectors, and the outputs, results and impacts produced in these sectors are clearly identifiable. Economie development Box 22 - Appi theí ying the principle of mainsti itrategic priority of economi earning: An example of c diversification Programme Indicators Context Indicators % of regional GDP in rapidly growing sectors Impact Result Output Resource Value added generated after one year (of which in rapidly growing sectors) Leverage effect (of which in rapidly growing sectors) Number of economic units supported (of which in rapidly growing sectors) Total budgetary expenses (of which to support projects in rapidly growing sectors) Depending on the economic development strategies selected by various programme designers, the mainstreaming principle can apply to various priorities. Below are presented three frequently cited priorities. Many programmes are built upon a strategy of economic diversification. Financial assistance from Structural Funds is aimed at reducing the weight of declining sectors as a proportion of regional economic activity. In such instances, it can be useful to break down major programme indicators into growing / declining sectors (e. g. amount of subsidies, number of beneficiaries, amount of leveraged private investment). One of the problems with this approach is that there is no standard typology available to make this distinction. Furthermore the classification of activities in terms of growing / declining sectors is a rather delicate distinction to make given the complex way in which markets are segmented and the cyclic nature of certain activities. Another problem is that within declining markets there are always rapidly growing niches. In the case of programme indicators, it seems possible to categorise supported projects by using the following questions: "What is the most relevant market for the project?"-"Has this market grown significantly during the last five years?"-"ls this market likely to grow within the next five years?" Economic development programmes also include a mix of measures local initiatives and attracting new productive targeted at promoting activities. A high degree of inward investment compared to local projects Indicators for monitoring and evaluation located outside the region. As a consequence increases economic dependency on activities managed from decision centres the regional economy is likely to become more sensitive to external shocks. The stimulation of local entrepreneurship is a process that produces a less visible impact, but these impacts are more firmly rooted in the regional territory, and are more likely to be longer lasting. For this reason it may be considered as a strategic priority. In such a case, it is interesting to break down as many indicators as possible into projects of an endogenous / exogenous nature. Some economic units are active in markets that are open to European and worldwide competition (e. g. automobile or computer producers), while others are located in smaller regional or local markets (e. g. housing or retail trade). This means that the latter are to a certain extent sheltered from external competition. Public support for the sheltered sector can prove to be effective in the short run, but only the growth of the competitive sector can generate long term economic development. If a strategic priority is given to internationally competitive activities, it is interesting to break down programme indicators into projects situated in the competitive / sheltered sectors. Economic development and diversification are primarily measured in terms of impacts However, according to the principle of mainstreaming the strategic priority should also be monitored in terms of results, outputs and resources. Economie development Example 45: Support for the construction of a conference and exhibition centre Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Progress Compliance with project duration Rate of completion Physical output Floor space (m2) Result indicators for e n d o g e n o us economic development Usage Number of conferences and exhibitions organised (% of conferences and exhibitions that are mainly linked with local economic activities) Locally owned and managed firms as a % of enterprises in the area I m p a ct indicators for e n d o g e n o us e c o n o m ic d e v e l o p m e nt Commercial contacts Usage Number of new commercial contacts (incl. % for locally owned and managed firms) due to the activity of centre after one year Number of firms having used the centre for promoting their sources after one year (% of which locally owned and managed) Example 46: Direct support for capital investment in enterprises Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators for diversification Number of addressees or recipients Number of firms receiving grants (% of which In rapidly growing markets / sectors) Result indicators for diversification Leverage effect Additional private investment (% of which rapidly growing martlets / sectors) Impact indicators for diversification Diversification Annual value added that has been generated by investments in euros / year after one year (% of which in rapidly growing markets / sectors) % regional GDP generated in rapidly growing markets / sectors Indicators for monitoring and evaluation Example 47: Venture capital scheme for small business development Programme Indicators (Related to the Intervention and its effects) Context Indicators (Related to the assisted area) Output indicators for competitiveness Number of addressees or recipients Number of SMEs having had at least one loan backed by venture capital (incl. % of which In non-sheltered sectors) Result indicators competitiveness Leverage Additional private investment that is generated by loans (¡nel. % of which in non-sheltered sectors) Impact indicators for competitiveness Value added Exports Annual value added that has been generated by venture capital backed loans (¡nel. % of which In non-sheltered sectors) Exports that have been generated by assisted SMEs after one year (of which in non-sheltered sectors) % of non-sheltered sectors in regional GDP Exports as a % of regional GDP Availability and collection of data Information for programme indicators at the output and result level, whilst largely available from the monitoring systems of operators, are not automatically broken down into the categories (i. e. growing or declining sectors, exogenous or endogenous development, competitive or sheltered activities). therefore be undertaken as an integral part of an evaluation. As for data to provide information for gross impact indicators, these will often have to be based on addressees or recipients' expectations. Net impact indicators will have to rely upon complementary surveys. If necessary, this classification of data will Many economic indicators at the context level, except income per capita, are presently available for a majority of regions and published in the Eurostat Regional statistical yearbook. Examples of available indicators include GDP per capita, productivity, value added and investment. Most of these context indicators are broken down into sectors such as: agriculture, energy, manufacturing industry, construction, market services, non-market services. Economie development As these indicators are vital for assessing regional economic performance, some additional efforts should be devoted to collect them for all the NUTS 2 regions and accelerate their updating. Given that the targeting of the European regional policy also descends as far as the NUTS 3 level, it will be of interest to select the most representative indicators, at least one in each category and whenever possible the most representative, and collect them at this level. Key indicators A number of key indicators, presented below, are suitable for aggregation and comparison of Structural Fund interventions in the field of economic development Level Key indicator Resource Output % of budget devoted to projects of locally owned and managed firms % of budget devoted to projects in rapidly growing markets % of budget devoted to projects in non-sheltered sectors Number of economic units receiving direct assistance or services as a result of the programme (of which involved in locally owned and / or managed firms, with rapidly growing markets, in non-sheltered sectors) Impact (programme) Value added generated by the programme after 18 months in terms of euro / year / employee (of which generated by locally owned and / or managed firms, by firms in rapidly growing markets, by firms in non-sheltered sectors) Impact (context) Investment / capita, GDP / capita, Value added / employee. Exports in % of regional GDP, % of regional GDP in locally owned and managed firms, rapidly growing markets, non- sheltered sectors Employment The promotion of viable employment is a major component of the Cohesion Policy and activity to this end is supported by all of the Structural Funds. For example, training activities or support for new enterprises will have direct employment effects, while investment in health infrastructure or in the protection of the environment will also create jobs, although in a more indirect manner. However, it remains the case that virtually all of the activity assisted under the Structural Policies may eventually have some employment effect, though these are not generally uniform across the population and may in fact exacerbate the situation of certain categories of addressees or recipients, for instance women or the long term unemployed. The aims of combating unemployment and promoting employment have for long been an implicit strand of European Community policy. However, since the mid-1990s these aims have become increasingly explicit and pervasive across a wide range of Community policies, including the Structural Policies. Beginning with the White Paper on Development, Competitiveness and Employment and the conclusions of the subsequent Essen Council, activity in this policy area proceeded with proposals made by the Commission in its paper "Action for Employment in Europe" (1996). Policy continues to develop with the formulation of annual Employment Guidelines by the Council and national employment action plans by each Member State. These efforts are intended to respond to the process of structural change, which the European economy has experienced over the past two decades, with a set of structural labour market policies that will enhance the creation of viable employment. Strategic priorities have been progressively refined and improving employability, developing entrepreneurship, encouraging adaptability of businesses and employees to economic changes, and strengthening equality of opportunity (Council Resolution on the 1998 Employment Guidelines - December 1997). The last priority is the subject of a separate information sheet in this Volume. they now concentrate on four main lines: Impacts on employment Because of the horizontal and pervasive nature of employment concerns, the commentary provided in this section describes interventions that have isolate to artificially an impact on employment employment interventions and discuss indicators related to them. rather than trying Indicators for monitoring and evaluation Fields from which the following examples are drawn are respectively, training and tourism, but any other field could have been referred to if other examples had been chosen. indicators are of particular Some resource interest for the field of employment. These indicators measure "human resources" used by the programme, i. e. : the mobilisation of workforce in the implementation of an intervention. For example, the "number of trainers employed by an assisted training course" is an indicator of the human resources invested in the implementation, but it is also a component (although a minor one) of the employment related to the project. Investment in physical infrastructure will in most cases itself have short-term consequences for employment through construction activities. Since this type of job tends to be temporary in nature, units of measurement should be expressed in job χ years to clearly show its transitory nature. Strictly speaking, this type of employment should be considered as a resource rather than an impact. In order to simplify matters, all employment created is often grouped together and quantified with a single indicator. The outputs of interventions having an impact on employment tend to take the form of enhancing factors that have been identified as promoting job creation. Consequently, output targeted on employment do not focus directly on employment itself but rather on the infrastructure and services which are needed to enhance the employability, adaptability or entrepreneurship of target groups. Interventions that have job creation as an output are certainly exceptional (e. g. a measure which consists of training personnel who would then become trainers, would have job creation as an output indicator). for actions indicators In many instances, the creation or maintenance of jobs may be a result of an intervention. This occurs when employment comes about as a direct and immediate consequence of assisted activity (e. g. job of a new assisted entrepreneur, job of a new assisted aquaculture farmer). Employment becomes an impact when it is created or maintained as a mid term and / or indirect consequence of the intervention. For instance, employment impacts may consist of lasting benefits for those who have received assistance (e. g. trainees whose jobs have been maintained, trainees who are employed after 12 months). Jobs may be created in the (e. g. ports, management and maintenance of new telecommunications networks, and waste disposal plants). All of these are infrastructures Employment examples of employment that is a direct mid-term effect of structural interventions. Employment impacts may also consist of indirect effects, mainly through the development of addressees' or recipients' activities (e. g. jobs generated in assisted firms after 12 months, jobs generated in the suppliers of assisted firms, jobs generated in the region through income multiplier effects). Employment impacts may arise in a more indirect way due to: the growth of economic activity which is either generated or sustained by the existence of new infrastructure, the enhanced competitiveness of assisted firms, or the enhanced competitiveness of companies benefiting from a better qualified management and / or workforce. It may be argued that analysis of indirect effects is extremely complex and that concern with such impact indicators is more the role of qualitative in-depth studies, or should the occasion arise, macro-economic analysis. Of the utmost importance is the need to make the distinction between gross and net employment effects. Employment lend themselves to easy quantification are generally gross indicators like the number of jobs existing in a new assisted company after 12 months, number of jobs related to the operating of new infrastructure, number of jobs that assisted companies declare have been safeguarded thanks to the intervention. indicators that Gross indicators do not provide a definitive view of employment effects because of the influence of confounding factors in the micro and macro- economic contexts. Since causal analysis is always difficult, it is unrealistic to seek completely definitive measures of net job creation. Nonetheless, a number of steps can be taken to ensure that indicators are selected and interpreted in a fashion which increases their relevance and accuracy. limited by ensuring First, underestimation can be indirect consequences of employment interventions are taken into consideration, including any which arise from an increase in demand related to the intervention, for instance by using standard coefficients for supplier and multiplier effects. Second, overestimation can be prevented by taking account of the possibility of deadweight (changes in employment which would have occurred even had the intervention not been made). that the Displacement and substitution effects are more difficult issues. Displacement is the creation of employment in one location merely at the expense of economic activity in another location. A case of substitution is the hiring of an assisted unemployed individual at the expense of another Indicators for monitoring and evaluation worker whose job has disappeared. Some displacement or substitution effects may be intended (e. g. creating jobs in an assisted area at the expense of a large dynamic capital city; facilitating the return to work of the long-term unemployed, even at the expense of other workers, etc. ). It is always a specific decision to take or not to take displacement and substitution into account. Finally, the net employment effects of an intervention are defined as the gross employment effects to circumstances, less displacement and / or substitution less deadweight and, according It is often important to make some attempt to identify the quality of the employment, which arises from an intervention. At its most basic, this may be reflected in income levels but it might also be related to the sustainability of employment. Example 48: Return-to-job programme for long-term unemployed Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Number of addressees or recipients Result indicators Rate of immediate transition I m p a ct i n d i c a t o rs Gross placement rate Net placement rate Quality of jobs j Number of long-term unemployed ¡ participating in the programme 1 (of which women) Number of addressees or recipients who are either hired or in a better social situation at the end of their participation in the programme (incl. % of women) Number of addressees or recipients hired after 12 (incl. % of women) Gross rate less estimated deadweight (trainees who would have been hired without assistance) less unacceptable substitution (trainees whose hiring has led another worker to long-term unemployment). % of hired addressees or recipients (incl. % of women) who's job is full-time and permanent after 12 months Number of long-term unemployed in the area % of jobs in the area that are permanent and full-time Employment Example 49: Assistance for the setting up of an equestrian event centre Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators (in the field of tourism) Compliance with project duration Date of expected completion of project Result indicators (in the field of tourism) Planned utilsab'on Impact indicators Gross mid-term direct employment effect Indirect employment effect Number of visitors Number of event planned in the first year Number of visitors expected in the first year Numbers employed in administration of centre and provision of services after 18 months (FTEs, % of held by women) Estimated number of jobs (% of held by women) generated by the stay of tourists attracted by the centre Average number of jobs (ITEs) in the tourism sector for 1,000 visitors χ nights Number of visitors to events held at centre (incl. % which comes from out of the region) after 1 year Number of visitors in the area (nights per year) Availability and collection of data The type of information held in management information systems can generally serve as a basis for evaluating employment effects. Monitoring indicators can often produce information about temporary jobs and about gross permanent jobs when employment effects are direct and short term. Reports from relevant research and labour market analyses (including the results of previous evaluations of relevant interventions) provide a stock of both quantitative and qualitative information that may be used for estimating indirect job creation. Interesting possibilities also arise from the use of job ratios (e. g. number of jobs generated by 1,000 tourist visits, number of jobs generated by 1,000 euros of value added in a given sector, etc. ). Even deadweight can be derived from previous evaluations of similar interventions in similar contexts. Nevertheless, only an in-depth evaluation can provide a satisfactory estimate of net job creation or maintenance. Such an evaluation involves an effort to track the effects by surveying addressees or recipients. It also involves a judgement about whether displacement and substitution effects are acceptable or not. Indicators for monitoring and evaluation The most general sources of context indicators on employment and labour market issues are the permanent statistical surveys and time series data that are comparatively well established at the European, national and regional levels. Labour force surveys are available on at least an annual basis for both the EU as a whole and individual Member States, as is data on various aspects of unemployment. One problem associated with information of this kind is that it is often not as frequently available on a regional level as it is on the national or Community level. Moreover, the regions on which data is provided frequently do not coincide with the regions which are eligible for assistance from interventions. A second problem with this type of information is that its value for the evaluation of interventions is limited by the time lags that are inevitable in producing data for standard statistical surveys. Key indicators The key indicators presented below lend themselves to comparisons both between different interventions of the same programme and between those of different programmes, and can in certain cases be aggregated at a European level. Level Key indicator Resource Number of persons mobilised to implement the interventions (in job years, incl. % of held by women) Result Impact Number of persons who have been immediately hired during the life of the project (incl. % of women) Number of gross direct jobs created or maintained (FTEs, incl. % held by women) - number of individual addressees or recipients who are at work after 12 months Number of net direct jobs created or maintained (FTEs incl. % held by women: created or maintained) - Gross number less estimated deadweight, less unacceptable displacement and substitution Total number of gross / net jobs (FTEs, incl. % held by women) - direct employment plus supplier and multiplier effect Quality of jobs - % of full-time and permanent jobs (incl. % held by women) Employment Since the employment impacts of an intervention may include both part- time and full-time jobs, this distinction should be recognised in any indicators used and both aggregations and comparisons may be made by converting raw data on numbers employed into full-time equivalents (FTEs). Dates of measurement (6, 12 or 18 months) should also be subject to the purpose of European comparisons and standardisation aggregations. The standard proposed date is 12 months. A more difficult prerequisite for comparisons is that displacement and substitution should be treated in a standard way. The standard proposed treatment is the following: displacement at the expense of any area eligible to Structural Funds should be deducted from the gross effects; substitution at the expense of any person targeted by Structural Funds should be deducted from the gross effects. for For more information see. C3E (1996), Evaluating the Contribution of the Structural Funds to Employment. Brussels: European Commission. European Commission (1996), Counting the Jobs: how to evaluate the employment effect of Structural Intervention. Brussels: European Commission. European Commission (1996), Regional Success Stories: profiles of 36 projects in Europe. Luxembourg: European Commission. Schmid, G. et al. (1997), Progress in Evaluation Research. Berlin: Wissenschaftszentrum Berlin für Sozialforschung. The fifth Volume of the MEANS Collection includes a chapter devoted entirely to the evaluation of employment effects. Quality of life The term quality of life has been widely used in recent times. It describes the well being of the population and as such has an environmental, social and economic dimension and is closely linked to sustainable development. In this context, sustainable development can be described as achieving good quality of life while not degrading the environment. At its broadest, all aspects of Structural Fund intervention have implications for quality of life. This may even be the case where it is defined in relatively narrow terms, where the main focus is on an improvement in living conditions. Any intervention that improves local living conditions has an impact on the residents' continued willingness to live, study and invest in the assisted area (or regional attractiveness). This willingness to live in the area is an objective that is particularly sought after in the case of regions with very low population densities. The rationale for incorporating direct interventions in the field of quality of life into a development programme is that they contribute significantly to the attractiveness of the assisted area. Interventions specific to this field consist of creating facilities that increase the attractiveness of a region by promoting better health, improved education, culture etc. The funding of such facilities, however, is not frequent within the framework of the Structural Funds. Because interventions in this field generally concern a large variety of different types of projects, an off-the-shelf approach to indicators needs to be pursued with caution. Ideally lists of indicators should emerge from the particular context in which they are being used and reflect the relevant concerns and issues involved. Interventions within the field Resource indicators Interventions in the field of quality of life, such as investment in health, or in other local facilities and services, are generally assessed by using the standard resource indicator of budget consumption. Output indicators Outputs in this field can firstly be measured by standard indicators such as "rate of completion" and "compliance with project duration". The volume of Indicators for monitoring and evaluation outputs can often been measured in terms of capacity. Therefore, in the case of support for local hospital facilities, outputs could be measured by the "number of beds available". For an assisted theatre or library, relevant output indicators could be the "number of seats" and the "number of publications". However, such indicators are very specific and of limited use for the monitoring and evaluation of the programme as a whole. Result indicators There are immediate benefits of new equipment or facilities when the potential users can gain access to given services in a reduced time or when the quality of given services is improved or safeguarded Impact indicators Specific impact indicators in this field generally focus on the degree to which the output is taken up after one / three years. Indicators might include the number of visitors to a library, the number of people taking part in a cultural event, the number of patients using a hospital. Indicators are expressed in number per year, or preferably in percent of the facility's capacity. Impact indicators go well beyond direct provision and the use of facilities and should be designed to measure how users are affected by the experience, e. g. : is the supported facility continuing to be regularly used? Has it reduced the sense of isolation of the users? Has it increased their confidence in the area and will it change their future willingness to live in the area? Impact indicators are the most interesting features of projects but the most difficult to measure. Impact on quality of life tends to be more complex than other types of impacts; as a consequence they are suited to qualitative techniques of data analysis. Quality of life Example 50: Construction of a new hospital Programme Indicators (Related to the Intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Capacity Result indicators Access to hospital Number of patients that can be treated, according to type of care Number of hospital beds per 1,000 population Average distance of patients house from hospital Distance of patient's house from hospital in area Quality Number of different care specialities in the hospital Average number of care specialities in national hospitals Impact indicators Need Usage Occupancy rates of beds after one year % of inhabitants who rank health services as one of the major problems with their living conditions. Occupancy rates In hospitals elsewhere in the area Other interventions having an impact in the field Although some interventions can be considered as specific to the field of quality of life, almost all Structural Fund interventions that belong to other fields may be seen as aiming to eventually improve the quality of life. For instance, motorways and airports, suburban transport systems, village renewal, reclamation of derelict lands, training, etc. One aspect of Structural Fund intervention, which fulfils a number of the objectives of the funds, is to enhance the attractiveness of a region, particularly to stimulate interest in the area among businesses and potential residents. In addition to standard resource indicators like budget consumption, it may be useful to develop a special indicator measuring the amount of funds of all types of interventions allocated to areas where living conditions are the most difficult. The indicator would be expressed as a percentage of the total funds. Delineating the most difficult areas would be a useful but difficult work, both from a technical and political standpoint. In addition to standard result indicators such as the "number of addressees or recipients" and the "satisfaction of addressees or recipients", there is a the point for developing a special programme is concentrated on residents in areas where living conditions are indicator measuring how much Indicators for monitoring and evaluation the most difficult. The indicator would be expressed as a percentage of the total number of addressees or recipients. Improved attractiveness of the assisted area is a major expected impact in the field of quality of life. Attractiveness is a multidimensional concept, applying to individuals as well as to firms. It applies to external actors who may be attracted by the area (potential residents, firms, customers, and tourists) as well as internal actors who may be retained in the area. Direct measures of attractiveness can be derived from questionnaire surveys asking questions like "do you intend to leave the area within the next five years" or "assuming that you have decided to move, would you accept to live from in this area?" More accurate measures can be derived comparisons through questions like "please rank this area among these other areas as the best places you would like to live in". Indirect measures are also available through the cost of property and land. Measures of immigration and emigration flows are generally much too delayed to be useful in the context of monitoring and evaluation. Example 51: Upg rading a bus system in neglected urban areas Programme Indicators (Related to the Intervention and its effects) Context Indicators (Related to the whole urban area) Output indicators Focusing on the problem area km of new bus routes Result indicators (in tt ie field of transport infrastructure) Access Impact indicators Attractiveness Usage Average time for residents of neglected urban areas to reach ten major urban services (i. e. university, swimming pool, hospital, library) Average time for all residents in the town to reach ten major urban services Average ranking given to the neglected area by residents who use the bus system % of users who are residents of areas where living conditions are a problem (neglected areas) % of town citizens living in the different areas Availability and collection of data Resource, output and result indicators will usually emerge from information systems used in the day-to-day management of projects. These are often comprised of financial information systems and systems used for managing Quality of life the use of services. For instance, facilities such as libraries use statistical packages that provide details on expenditure and income associated with books and link them back to a cost centre. They may also through swipe card technology have another set of indicators which offer non-financial information such as the number of users, types of items borrowed and fluctuations of library use across different days and times of the day. These information systems may be brought together for purposes of analysis. Impact data may not be quantitative in nature although they may be set up to add a quantitative element to qualitative responses. For example, questionnaire surveys involve the coding of responses to open-ended questions according to key words. It can then be said that X% mentioned libraries as an essential aspect of local quality of life. impact of specific Structural Fund To help understanding of interventions it is necessary to know the context in which they operate. Context indicators that are critical for quality of life are notably long-term unemployment and income. the Key indicators Indicators in the field of quality of life are not yet stable enough to enable key indicators to be proposed. The introduction of the dimension of quality of life in the concept of development would dramatically change basic concepts such as regional disparities or economic growth. One such indicator, the "Indicator of Human Development" (IHD), has been used by the World Bank and the United Nations for over ten years. A similar indicator is the "Index for Sustainable Economic Welfare" which is presented in Box 23. The Index for Sustainable Economic Welfare (ISEW), like GDP, is an aggregate measure. It makes 18 adjustments to consumption expenditure (the basis of GDP) so that: • social and environmental costs are deducted; • long-term environmental damage and the depreciation of natural capital is deducted; • average personal consumption is adjusted to take into account patterns of income distribution; • a value is imputed (and added) for unpaid household labour. Aggregated indicators such as GDP and the ISEW, which bring together disparate entities through the common measure of money, are somewhat controversial, partly because applied monetary values have a degree of Indicators for monitoring and evaluation subjectivity. However, aggregation can help to link different levels, both micro and macro, at which quality of life interventions take place. Significant work is required to further develop such indicators. Box 23 - The index for Sustainable Economic Welfare This index shows that between 1950 and 1996 the United Kingdom experienced a growing divergence between ISEW and GDP. This indicates that the quantity of economic wealth (GDP) does not equate with the quality of life (as proxied by ISEW). GDP and ISEW: 1950-1996 1990 pounds sterling 10,000 9,000 8,000 7,000 6,000 5,000 4,000 3,000 2,000 1,000 0 — ■ * "~ GDP ISEW 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 Source: Jackson, T,. Marks, N. , Ralls, J. & Stymne, S. , Sustainable Economic Welfare in the UK: 1950-1996,1997, University of Surrey Centre for Environmental Strategy/New Economics Foundation. For more information s e e- Cheshire, P. et al. , (1996), 'Problems of Urban Decline and Growth in EEC Countries: or measuring degrees of elephantness' in: Urban Studies, Volume 23(2), 131-149. MacGillivray, A. and Zadek, S. (1995), Accounting for Change: indicators for sustainable development. London: NEE New Economics Foundation / World Wide Fund for Nature UK (1994), Indicators for Sustainable Development: Strategies for use of indicators in national reports to the Commission on Sustainable Development and in the EC Stnjctural Funds process. NEF / WWF UK. New Economics Foundation / World Wide Fund for Nature UK (1997), Signals of Success: A users' guide to indicators. NEF / WWF UK. Ill Methods of producing and using indicators Involving suppliers and users of information Experience has shown that a system of indicators has much more chance of functioning when the suppliers and users of the information have been involved in its creation. In contrast, a closed group of specialists will be tempted to construct an expensive, technically ideal system which may never be operational. Thus, the design of a pragmatic system of indicators requires the involvement of both users and suppliers of information. As far as the users are concerned, explicit support from the highest level of the authority managing the programme has to be assured. It is then advisable to create a steering group composed of future users of the system, and to give this steering group the job of defining the indicators. The composition of this group will probably not be very different from that of the group which will have to conduct the evaluation. A team should then be appointed to run this steering group and to provide it with a secretarial service. If this team is to be effective its members must belong to the authority managing the programme and have the required human and financial resources to fulfil its function. It must, in particular, ensure that the system of indicators clearly reflects the programme objectives and ensures comparability. It is preferable for the same team that is responsible for creating indicators to subsequently be responsible for the implementation of the system. The public may also be involved in designing the system. Thus, an American experience started with a series of focus group interviews, with representatives of regional enterprises. The work of these groups made it possible to select indicators most likely to attract the public's attention and to be understood by it (see Box 24). The suppliers of information must, as stated, also be involved in designing indicators. The main suppliers are the operators who implement the programme in the field. Their participation is likely to ensure that the system is pragmatic because they are familiar with the practical possibilities and limits of data collection. Time permitting, it is advisable to involve the operators in a preliminary test of the system of indicators. The recommended procedure starts with the selection of a few volunteer operators who will participate in the design of the system. These volunteers must represent all the components of the programme. They help to choose the indicators, to define them and to plan Indicators for monitoring and evaluation the channels for collecting information. They express their needs in terms of information feedback (frequency and form of information fed back to them). The test comprises an initial quantification of all the indicators by voluntary operators. The normal duration of such a test is a year. Depending on the conclusions of the test, and after introducing the necessary modifications, the system is validated. The definitions and the data collection and restitution procedures are clearly established, and a manual is written. Box 24 - Involving addressees or recipients in the choice of indicators Political authorities in the Benton Harbor region (USA) set up a system of context indicators with a view to assessing the economic development of the Berrien metropolitan area. The key elements in the description of economic development were based on a series of focus group interviews involving the leading entrepreneurs in the region. The indicators chosen were, for example: • For the availability of qualified human capital: spending on education per child; the teacher-student ratio; the school dropout rate. • For the growth and diversification of the economy: per capita income; employment rate in the industrial sector; value added in the trade sector; index of sectoral diversity; rate of employment in SMEs; value and number of residences that were issued building permits. • For the quality of life: the cost of living (relative), the rate of property crimes and crimes against the person. Erickcek G. A. (1996) The Benton Harbor Area Benchmarking Data System Michigan: WE Upjohn Institute Information relating to the context is drawn from statistics. It is therefore advisable to involve an expert with recent and complete knowledge of exploitable statistical data, in designing the system. Depending on the case, this expert will belong to a statistics institute or a university or, if possible, to the institution running the programme. S e l e c t i on of t he m o st relevant indicators Each of the programme actors have their own responsibilities, their own areas of decision-making and therefore their own information needs. As a result, all indicators are not useful at all levels. On the contrary, it is generally accepted that each actor requires an operating report with a small number of indicators, selected as the most relevant in relation to the nature of the decisions that have to be made. It has been shown that in a situation of Producing and using indicators decision-making, a person cannot take into account more than about ten indicators at once (Innés de Neufville, 1994). When there are too many indicators the decision-makers are swamped with an excess of information. The problem of heterogeneity of programmes The experience of the Structural Funds has shown that it is difficult to choose indicators that are absolutely necessary for the monitoring and evaluation of a programme. Because the programmes are multi-sectoral and multi-objective, managers sometimes attempt to measure everything and design systems of indicators which are so heavy that it is impossible to make them work. For example, the initial proposal for the Objective 5b Programme in Burgundy (F) consisted of over 200 indicators. In the end, only about fifty of them were quantified. The following example clearly illustrates this problem. It concerns a programme (1994-1999) for the industrial reconversion of a French region (Franche-Comté). This programme comprises 48 actions of various kinds, e. g. several types of direct support to enterprises; specific support to certain sectors, to groups of firms, to professionals in the tourist business, to creators of enterprise; support for the improvement of the environment; etc. These actions are managed by different operators who all tend to measure the effects of their actions in terms of their own logic (e. g. number of subsidies; number of participants in collective projects; surface area of industrial sites improved). All this information provided by the operators is too heterogeneous to be synthesised at the programme level. Apart from the indicator of budget absorption, no indicator applies to more than 20% of the programme. For a programme like this, a complete information system would be unwieldy because it would have to include the following data: • one or two generic indicators for the monitoring of resources (e. g. rate of into actions; expected rate of budget absorption broken down overspending); • about fifty output indicators, which amounts to an average of one indicator per action; • about fifty result indicators, which amounts to one indicator per action; • about one hundred specific impact indicators, or an average of two indicators per action (e. g. survival rate of assisted businesses; placement rate of trainees; statements made by firms on their intentions to recruit or invest; number of innovations generated; number of visits by tourists); Indicators for monitoring and evaluation • s o me g e n e r ic i m p a ct i n d i c a t o rs (e. g. j o bs c r e a t e d; value a d d ed generated in the assisted firms, etc. ); • s o me c o n t e xt indicators (regional u n e m p l o y m e nt rate; number of SMEs in the region; average value a d d ed per employee in regional firms; etc. ). In total, the n u m b er of indicators of this c o m p l e te a nd obviously theoretical s y s t em can be estimated at 200. In practice, it is impossible to p r o d u ce and regularly use such a large amount of information. In several European regions, recent evaluations have s h o wn that a few dozen indicators are e n o u gh to meet the information needs of the authorities running the p r o g r a m me (See Box 25). This does not m e a n, however, that additional indicators may not be required to fill the operators' information needs. Box 25 - T he r e c o m m e n d a t i on of an evaluation: reducing t he n u m b er of indicators f r om 3 30 to 52 A socio-economic programme was financed by the European Structural Funds for the period 1994-99 in Northern Ireland. An intermediate evaluation was made of this programme, by synthesising six separate evaluations of sub-programmes. The programme designers had chosen 330 indicators for the monitoring. These indicators had been grouped into a single data base that proved to be difficult to use and created problems of manipulability and availability of information. These problems considerably reduced the use of the indicators. The evaluation team recommended a solution consisting of: • choosing a number of context indicators situated between the macro- economic indicators and the sub-programme indicators. These indicators, intended to reflect the global impact of the programme, are divided into three categories: economic growth, internal cohesion and external cohesion. • choosing a small number of programme indicators by limiting them to the main results and impacts; • delegating the quantification and use of the other indicators to the operators. In this way, the size of the system could be reduced to 52 indicators directly related to the main objectives of the programme. The recommendations were successfully applied. Colin Stutt Consulting, 1997, Northern Ireland Single Programme 1994-99; Mid Term Review External Evaluation. S u g g e s t i o ns f or limiting t he size of s y s t e ms of indicators This section takes the example of the French region cited above and describes the suggestions, m a de during the intermediate evaluation, for Producing and using indicators reducing the number of programme indicators without losing relevant information at this level. As Box 26 shows, these suggestions are based on the use of generic indicators or on the grouping together of the indicators by category of addressee or recipient. Box 26 - Example of an information system using a reduced number of indicators Number of actions (elementary components of the programme) 48 Programme indicators. of resources. of outputs 1 2 of results and specific impact 45 Context indicators Total number of indicators 51 (Rate of budget absorption) (Output rate, rate of compliance with project duration) (Average of 5 indicai of the 9 categories of addressees or recipients) (income, unemployment, attractiveness) for each Firstly, the evaluation suggested that all the resource and output indicators be grouped together in the form of three generic indicators: rate of budget absorption; completion rate; rate of compliance with project duration. These three indicators can easily be quantified for each action, on the condition that the operators have their own specific indicators. The three indicators can then be aggregated at the level of each measure and that of the programme as a whole. Secondly, it was noted that the 48 actions of the programme could be grouped together into nine categories of addressee or recipient: actions in favour of businesses, of creators of businesses, of adults undergoing training, of professionals in tourism, etc. For each category of addressee or recipient, it was possible to choose a small number of result and specific impact indicators. For example, for adults in training, two indicators could be chosen (success rate in reaching the target public and placement rate after 12 months), and the data could be broken down in three ways (women / men; employees / the unemployed; growing sectors / declining sectors). Given the conventions adopted in this Volume, this represents a total of six indicators. In total, about forty indicators could suffice for providing information about results and specific impacts. Indicators for monitoring and evaluation Some basic context indicators such as the per capita income in relation to the national average or the regional unemployment rate, as well as an indicator on the attractiveness of the region (an indicator which is clearly needed but which still needs to be constructed), could be added to these programme indicators. By following these recommendations, it should be possible to limit the information system of this programme to a reasonable number of indicators. A lighter system limits the collection and circulation of information to the most essential elements at the programme level. On the other hand, this means that the progress and results of each action will not be monitored in a detailed and centralised manner. It also means that the system focuses less on the decisions to be made by the operators and more on those to be made by the authorities managing the programme. Finding generic impact indicators Impact indicators are indispensable in evaluation but they are difficult to design and quantify. Apart from the number of jobs created, it is particularly rare to find generic impact indicators in programming documents. Box 27 - Examples of generic impact indicators ACTIONS Industrial investments Modernisation of craft industries Venture capital for SMEs Internationalisa tion of SMEs Technological development of enterprises GROUPS OF IMPACTS GENERIC INDICATORS Value added generated in growth sectors Number of innovations in assisted economic units The MEANS programme has developed a method for creating generic impact indicators. This method is presented in Volume 4 of the MEANS Collection which includes several methodological innovations. It functions in four steps in the following way: Producing and using indicators • The evaluation team gathers the main documents relating to the reports, programme brochures) and identifies all the sentences describing the objectives, the performance and the expected or real impacts. (programming document, reviews, progress • From the quotations taken from the documents, the evaluation team selects those concerning impacts. • It organises one or more seminars with the programme managers and operators. During these seminars, the participants complete the list of impacts and group them into families. Two techniques can be used to facilitate this work: impact mapping and Metaplan (for a description of these techniques see Volume 3 of the MEANS Collection). • Together, the participants of the seminar identify the signification of each family of impacts, name them and choose corresponding indicators. They check that these indicators are generic, that is to say, that they can be applied to numerous actions within the programme. Guaranteeing quantification Data collection by the operators Information collected by the monitoring system is drawn from the management of funds, the operation of projects, the processing of applications for subsidies, etc. Each operator provides information to the programme managers in the form of progress reports, periodic reviews or execution reports. These data should be collected and synthesised in the form of operating reports for each monitoring meeting. The project managers may be asked to provide exhaustive information on the following themes: • Resources - public budget absorption broken down by source of funding and overspending. • Outputs - rate of progress of projects underway; rate of compliance with project duration; volume of outputs in relation to resources (efficiency of the outputs); number of addressees or recipients broken down into a few large categories (e. g. heavy or light vehicles; large, medium-sized or small enterprises; women or men, etc. ); volume of services obtained by the addressees or recipients (e. g. hours of training; advice received, etc. ). • Results - declarations made by the addressees or recipients (e. g. investments to be made, jobs to be created); rate of addressee or recipient satisfaction; volume of results in relation to the corresponding resources (efficiency of results). Indicators for monitoring and evaluation Whenever possible, the monitoring system must ensure that the operators retain the possibility of contacting the addressees or recipients at a later stage for an evaluation survey (name and address of trainees; name and telephone number of a contact in the assisted firm). This precaution is also useful for avoiding double counting. For example, it will ensure that a firm receiving several forms of assistance from the programme is not counted several times. A system of indicators has far more chance of functioning if it does not disturb existing information channels. It must therefore request information closest to the source. In Box 28, the quantification of result indicators is part of the operators' standard management procedures. Box 28 - Collection of information by the operators The form presented in this box applies to programmes for helping foreign firms who invest in the region. Several operators are responsible for providing services to these firms in various parts of the region. Upon finalisation of an investment project with a firm, all operators must fill in this standard questionnaire: Name of the person who can be contacted in the firm: Telephone: Value of investments to be made in the framework of the assisted project: Number of jobs created or maintained in the framework of the assisted project: Quality of the services obtained by the firm: excellent [ 1 good Π average D bad 1 ] Usefulness of the services obtained by the firm: excellent ! good C average D bad □ Inspired by Hatry H. P. & al. 1990 In the context of programmes financed by the Structural Funds, a particular difficulty arises from the co-existence of systems of indicators peculiar to each funding body and which do not necessarily have the same definitions. For example, in the case of a training programme, a Commission representative wanted to obtain information on the number of trainees, Producing and using indicators considering a trainee to be any person who attended a course, even if only for a few days. However, the national operators were used to recording only those training courses which lasted for more than a month. In cases of differences between definitions chosen for programme indicators, and the operators' usual definitions, it may be preferable to quantify indicators in terms of traditional definitions which do not disturb the operators, and then to make corrections at (harmonisation- ) by means of a correction coefficient. the programme level The collection and synthesis of information may be facilitated by the setting up of a computerised system as the example in Box 29 shows. This tool must, however, be used with caution. New technologies facilitate the collection and circulation of a mass of information and the construction of systems of overabundant indicators. Computerisation must always be preceded by thorough reflection as a basis for the selection of the most useful indicators. information Box 29 - Computerisation of the monitoring system France, Objective 5b Regional Programme 1994-99 The monitoring system of the rural development programme in the region of Burgundy has been in operation since the beginning of 1996. The operators report to the five authorities which distribute the programme funds. These authorities are connected via a telematic network to a central data base which receives quantified information for about 1,500 projects per year. A computer program was specially designed for monitoring the programme and for processing information in real time. The information is synthesised at a regional level in an operating report comprising 40 indicators. These indicators are presented and discussed before each monitoring committee meeting. To encourage operators to supply the required information, it is normal to guarantee them feedback in the form of information with substantial value added. For example, each operator receives an annual report on their individual results, compared to the average for all the operators. Apart from these incentives, it is also usual to formally establish a contract for data collection. For example, the decision to accept a project and to grant funds to it must be accompanied by an appendix specifying the information to be produced, the date on which it must be produced and the person / organisation to whom it must be given. The programme manager must of course take all necessary steps to ensure that the data collected Indicators for monitoring and evaluation are not altered. The procedures for collecting and synthesising information must therefore be rigorous, transparent and checked periodically. Finally, the most decisive incentive consists of linking the allocation of new funds to the provision of data related to the preceding subsidy. Collecting information by way of a survey To quantify the sustainable impact on direct addressees or recipients, a common approach consists of selecting a sample of people or businesses which have participated in the programme, and of conducting a postal or telephone survey. This approach is facilitated by the fact that the people to be questioned are known to the operators, especially if the survey is planned from the start of the programme. The survey will be more difficult if the operators have not, in the course of their work, been able to keep track of the public concerned by the impacts (e. g. inhabitants of a rehabilitated neighbourhood; the competitors of an assisted business; tourists who have visited a protected site). In this case, the quantification is far more difficult and requires a tracking exercise, which may be difficult and time-consuming. Thus, feasiblity constraint eventually prevents the quantifying of progamme indicators. Beyond a certain limit it is no longer possible to observe impacts but only to estimate them by extrapolation. Box 30 - Use of surveys for quantification Management system Survey Statistics based on the list of addressees or recipients on the entire population regular quantification quantification during an evaluation regular quantification X X X X Resources and outputs Results Impacts (programme indicators) Impacts (context indicators) X X X Producing and using indicators Surveys of addressees or recipients must be used with caution when net effects are being estimated. For example, the evaluation team of an industrial reconversion programme in a region in England had to assess the employment impact of a measure aimed at encouraging an entrepreneurial spirit and training within firms. The evaluation team sent a postal questionnaire to several hundred assisted firms, in which their anonymity was guaranteed. The standard questions asked were: "Would you have recruited additional employees even without the subsidy?" or "Do you know whether the new employee was unemployed before being recruited?". Based on the survey results, the evaluation team estimated a coefficient of net job creation in relation to the assistance received, for the firms in the sample. By generalising the conclusions of the survey to all addressee or recipient firms, a net job creation indicator could be quantified by extrapolation. It is, however, important in such cases to be attentive to the quality of the survey because some answers may be biased due to the respondents' favourable opinions about an intervention from which they benefit. The organisation of a telephone questionnaire survey of a thousand individuals may require several months and cost between 10 and 40 KEuro. The different surveys undertaken in a region must also be carefully co ordinated so as not to swamp the addressees or recipients. For all these reasons surveys cannot be repeated frequently. During the interval between two surveys, it is nevertheless possible to estimate indicators by extrapolation, based on the data provided by the monitoring system. The Ministry of Labour in France, for instance, relies on the results of numerous surveys carried out over the past decade, for assessing the impact of employment incentives. Each of these empirical studies has enabled it to estimate impact coefficients in various situations. Owing to the accumulation of past knowledge, experts at the ministry have constructed a grid that is used to estimate impacts on the basis of monitoring data and of the main characteristics of a measure. Since 1992, meetings between experts are periodically organised to improve and adjust this grid of coefficients. Using statistical sources Context indicators are generally quantified on the basis of data obtained from statistics institutes. For example, many regionalised data can be found at Eurostat, in the REGIO data base. REGIO data are presented in the form Indicators for monitoring and evaluation of about 70 standard tables per domain: demography; economic accounts; unemployment; workforce; energy; agriculture; transport; and R&D. Eurostat uses data derived from a few major surveys to produce regional statistics. In the employment domain, numerous available data are derived from the "workforce" and "household panel" surveys. Regionalised data have recently been enriched through the exploitation of national annual surveys of industry. The information processed is available in the ELIZE data base which covers: the number of industrial enterprises; employment; salaries and wages; turnover; and investments. The data are broken down into about 30 economic sectors and two classes of size (+ / - 20 employees). In the middle term, the REGIO data base will be completed by an exploitation of the survey results for agriculture. There is also the QUID data base, managed by DG XVI, which groups together context indicators adapted to the Structural Funds. This data base is partly based on Eurostat information. The availability of statistics can become a problem when the eligible area of a programme is an administrative rather than a statistical zone. The best way of dealing with this problem is the "best correspondence" approach which has been applied for the Scottish system of indicators, as shown in Box 3 1. Box 31 -Correspondence between a statistical unit and an eligible area - United Kingdom, Objective 2 Programme 1994-99 The development programme in the East of Scotland has an eligible area that concerns several statistical units ("local government region level") but there is a lack of correspondence in the way that they are partitioned. The data provided by the Central Statistical Office is therefore not suitable for directly constructing context indicator of GDP was nevertheless estimated by aggregating data for the territories of the "central regions" and for Fife and Tayside, of which over half the inhabitants are in an area eligible for the Structural Funds. The territory of Lothian, less than half of which is eligible for Structural Funds, was disqualified. indicators. An A good estimation of context indicators can also be obtained by means of the interpolation method, as in the following (theoretical) example. One of the aims of an industrial reconversion programme is to enable the region to catch up with others in terms of innovation. A situation indicator would need to be created to help quantify this objective. A possible indicator is the Producing and using indicators innovations annual number of in the region. The annual number of innovations for 1,000 jobs, on a European scale, is provided by a Community survey carried out regularly on 40,000 firms. Based on the results of this survey, an innovation indicator can be created for the region in the following way: (1) choose an indicator of structural composition (e. g. number of jobs, broken down into sector of activity) for which the values are known at both a European and a regional level; (2) note the value of the annual innovation rate at the European level, with its breakdown by sector of activity, and (3) estimate the innovation indicator in the region by multiplying regional jobs, sector by sector, by European innovation rates. Eurostat has recently created an information system (GISCO) which cartographically presents all the statistical series existing in REGIO, as well as other regionalised data. The GISCO system also includes the outline of all the areas eligible for Structural Funds. It can therefore easily be used to break down or build up statistical information on different geographical scales. It is potentially a very powerful tool for estimating context indicators by interpolation. Assessing the quality of a system of indicators The use of indicators will be far greater if their quality is constantly improved. If evaluation has an uncontested role, this is to assess the quality of systems of indicators and to recommend ways of enhancing it. Although there is no standard method for this quality control, an approach is proposed here that is based on the following 11 criteria, which can be divided into two groups: quality criteria applicable to each indicator and quality criteria applicable to the entire system. Quality criteria applicable to each indicator The first quality criterion for an indicator is its actual existence, that is to say, it must be quantified at regular intervals. Sometimes one or more indicators featured in the programming documents have never been quantified and for all practical purposes do not therefore exist. The actual availability of data is therefore the very first quality criterion. Monitoring indicators must be quantified at each monitoring meeting, that is to say, every six to twelve months. Evaluation indicators are quantified less frequently, every three to six years. Indicators for monitoring and evaluation Once an indicator has been quantified, it may take several months or even years before the information can really be used for monitoring and evaluation. This is particularly true for certain context indicators drawn from annual statistical publications. The freshness of information then becomes an important quality criterion. Sometimes statistics are published two years or more after the collection of the data. A programme indicator must vary significantly when the programme is implemented and produces effects. This is what is called its sensitivity. In the example of an intervention supporting exports, the turnover of assisted businesses is an indicator that is not sensitive enough. A better indicator would be the turnover relating only to new customers contacted with the support of the programme. The programme actors must trust the information produced. Reliability is therefore a quality criterion. Reliability can be defined as the fact that the same measurement, taken by two different people, will produce the same value for the indicator. In reality this definition is only applicable where counts are concerned. Each time the indicator is quantified on the basis of questions put by one person to another, the reliability can no longer be defined mechanically. Reliability and credibility are two notions that then tend to merge. The usefulness of an indicator depends largely on whether it allows for internal comparisons between different measures of the programme or inter-regional external comparisons. The comparability of the indicator is therefore a criterion of its quality. This criterion is important, but difficult to meet. The last quality criterion of an indicator is similar to comparability but with an additional requirement: normativity. Indicators must include a reference so that the outcome can be judged to be satisfactory or not. Any value observed by means of the indicator must therefore be compared to a norm, for example: objective to be met; norm to be surpassed; European average to be attained; etc. A good indicator must be understood without ambiguity by everyone who has to use it. In the minds of both decision-makers and the public, the meaning of the indicator must be the same as for the programme managers. It must accurately reflect the concept to be measured. That is what is called the validity of construction. Producing and using indicators The example in Box 32 is an attempt to assess Portuguese operational programme indicators for the "transport" sector and for the period 1994- 1999. Box 32 - Example of an assessment of the quality of indicators Criterion availability freshness sensitivity reliability comparability normativity meaning Portugal, Objective 1 transport operational programme, 1994-99 Assessment Comments + ++ + ++ - = ++ most of the indicators are quantified with the usual frequency the main output and result indicators are quantified in real time the main programme indicators are highly responsive to its implementation the indicators are well accepted by the programme actors the system has few internal comparisons and no external comparisons only half of the indicators have a reference judgement the meaning of the indicators is fully understood by the programme actors Quality criteria applicable to the entire system The indicators selected must cover a sufficiently large proportion of the programme measures. This coverage must be equal to or greater than three-quarters of the planned expenditure. The system must consist of a good balance between indicators in the different categories. In particular, result and impact indicators must be the most numerous. A system of indicators must be simple. The selectivity criterion requires that the programme managers' capacity information be respected. The information must therefore be limited to a maximum of a few dozen figures. to absorb The relevance of the system implies that the indicators be developed primarily for those measures or themes which have significant implications Indicators for monitoring and evaluation in terms of decision-making (measures with a very high budget; innovative measures; themes considered to be strategic). Box 33 -Assessment of the quality of a system of indicators (continuation) coverage balance selectivity relevance Portugal, Objective 1 transport operational programme, 1994-99 ++ the system covers 100% of the measures ++ + there are no impact indicators; the result indicators cover only 4 6% of the programme measures the system comprises 26 indicators the system has more indicators for those measures and projects with the largest budgets Key indicators Comparisons between programmes Comparison between the different programme indicators is always an important step, but the difficulties involved must not be underestimated. These are due to various factors, such as the diversity of interventions within a programme, the diversity of regional contexts, or the incompatibility of definitions. For example, depending on the regions and programmes, tourist fréquentation is counted in terms of the number of visits or the number of nights; trainees are counted in terms of the number of participants or in hours χ trainees; and environmental protection is measured in terms of the number of projects, the number of sites or the number of hectares protected. Comparability may be sought and obtained through exchanges between managers in different regions or countries. The Interreg programmes create numerous opportunities for this type of comparison. In a different context, the Pays-de-la-Loire (France) and Emilie-Romagne (Italy) regions held bilateral meetings in 1997-1998 to discuss their evaluation work in the domain of rural tourism. In two two-day seminars, the regions analysed their respective actions and jointly defined a logic diagram of outputs, results and impacts. Common indicators were proposed, with a view to making inter-regional comparisons. Producing and using indicators Yet comparability is often easier to obtain and more profitable at all levels if it results from a co-ordinated collective effort on a higher geographical level. This approach has the advantage of multiplying the possibilities for comparison, and also allowing for the aggregation of indicators at the regional or national level. That is what the Scottish Office did w h en it created a series of standard indicators applicable to seven Scottish programmes. The British Audit Commission is another example of this approach (see Appendix 2). Box 34 - A set of standard indicators for several p r o g r a m m es Within the framework of its responsibility for monitoring and evaluating seven programmes co-financed by the European Union, the Scottish Office developed a series of standard indicators. Some of these indicators concern the context: rate of employment and unemployment; productivity; manpower; average income; etc. Some output and result indicators have also been standardised, as shown by the following examples, applicable to the infrastructure measures for enterprises: • New buildings built • Buildings renovated • Development of new sites • Improvement of existing sites • New / improved road access • Surface area for which the access roads were built or improved • Rate of occupation of the new buildings square metres square metres hectares hectares kilometres hectares percentage after one year percentage after three years • Number of training places created number To obtain a high level of comparability, the Scottish programme managers organised several meetings attended by representatives of all the programmes. Standardisation was therefore the culmination of a long process of collective discussion. Public c o m m u n i c a t i on The utility of a system of indicators is often reduced to decision-making. In reality, a system of indicators is also useful for accountability purposes, for example to the European or national parliaments, to regional or local elected representatives, to socio-economic partners, to journalists and, through them, to citizens and taxpayers. Indicators for monitoring and evaluation If they are to serve as a basis for public communication, a small number of indicators which can immediately be understood by lay people must be selected, quantified and published. These are referred to as key indicators. The publication of key indicators is normally organised in the form of simple commented in an annual review. More detailed information can also be made available through an "observatory" open to the public, or through a system of consultation on the Internet. tables, for example In defining good key indicators, the priority must be on generic indicators (applicable to many different actions within the same programme) and standard indicators (allowing for comparisons between programmes in different indicators must be understood by all the partners without long explanations and without any misinterpretation of their meaning. regions or countries). Moreover, these Proposals for key indicators Returning to the fifteen information sheets presenting the different fields of application, it is possible to propose a series of indicators that might be used as key indicators. This list is neither closed, nor directive, nor stable. It is limited to programme indicators. The indicators have been given scores, from *** to *, in decreasing order of interest. Producing and using indicators Box 35 - Proposals for key indicators: Resources Interest Indicator See example Human resources Number of operators (public and private organisations responsible for providing assistance to addressees or recipients) Number of advisors (FTEs) mobilised to provide advice to addressees or recipients Financial resources Rate of budget absorption (% of allocated funds) % projects (in financial terms) especially benefiting women ·· % projects (in financial terms) in rapidly growing markets / sectors % of budget devoted to environmental mitigation measures % projects (in financial terms) concerning the most disadvantaged areas Indicators for monitoring and evaluation Box 36 - Proposals for key indicators: Outputs Interest Indicator See example *** ** ** ** ** * * M *** *** *** ** Progress of works Rate of completion (% of objective) Compliance with project duration Capacity of finished works Number of potential connections (business / households) to networks of basic services (broken down by services) Activity of the operators in terms of attracting and selecting participants Selection rate (% of projects accepted as a proportion of eligible projects) Coverage rate (Penetration): % of the target population who have been (should be) participants in the programme % of addressees or recipients belonging to priority groups (e. g. long-term unemployed, early school leavers) % of addressees or recipients situated in the most disadvantaged areas % of addressees or recipients involved in rapidly growing markets / sector % of women in addressees or recipients % of SMEs in addressees or recipients Services funded by the programme Number of individual addressees or recipients having received services, advice, training Number of economic units (enterprise, farm, ship owner, fish farm, tourism professional) having received services, advice, training 1 , 2 , 3, 4 , 5 , 6, 7, 8, 9, 18,28, 29,30, 33, 34, 45 1,2,3, 4 , 5 , 6, 7, 8, 9, 18,28, 29,30, 33, 34, 45 28, 33, 34 10,30 10,15, 19, 36, 44,48 2 1 , 3 1 , 38 Number of hours of training / advice provided to addressees or recipients 10, 15, 1 9 , 2 1, 31, 36, 37 Producing and using indicators Box 37 - Proposals for key indicators: Results Interest Indicator See example Satisfaction of addressees or recipients Satisfaction rate (% of addressees or recipients that are satisfied or highly satisfied) 15, 19,25,27 36, 37, 44 Benefits gained by addressees or recipients Average speed between principal economic centres 1,2,32 Investments facilitated for addressees or recipients Leverage effect (private sector spending occurring as a counterpart of the financial support received) Box 38 - Proposals for key indicators: Impacts Interest Indicator Sustainable success Rate of placement (e. g. : % of individual addressees or recipients who are at work after 12 months, incl. % still employed after 24 months) 16,21,22,24, 39, 46, 47 See example 10,36,48 Rate of survival (e. g. : % of assisted economic units that are still active after 24 / 36 months) 23,37 Impact perceived by addressees or recipients Value added generated (e. g. : after 24 months in terms of euros / year / employee) Employment created or safeguarded (e. g. : after 12 months Full Time Equivalent) Impact globally perceived in the area Residential attractiveness (e. g. : % of inhabitants wishing to remain in the area) Indirect impact Regional knock-on effects (e. g. : % of regional firms within the suppliers of assisted firms after 12 months) 7 , 8 , 9, 13,14, 16, 1 8 , 2 1 , 2 4 , 2 6 , 3 1, 38, 39, 40, 42, 46,47 7 , 8 , 9 , 1 3, 14, 16, 18,20,23,24,35, 37, 38, 39, 42, 49 25,51 24,39 * ** ** ** * *** **# M * Indicators for monitoring and evaluation Measuring performance for better management More and more systems of indicators are created for the purposes of "performance management". These systems are a form of New Public Management which favours results and impacts obtained, as opposed to older forms of management based on the allocation of resources and the control of outputs. Managing performance rather than resources In a spirit of performance management, operators find themselves endowed with greater autonomy in the use of their resources. In return, they commit themselves to clearer objectives as regards the results and impacts to be obtained. They have to measure their performance in order to evaluate themselves and submit periodic reports. This new balance between decentralisation and performance measurement is at the base of many developments in public administration. Thus, in the U. S. , the "Government Performance and Results Act (GPRA)" of 1993 made the use of new indicators compulsory throughout the entire government administration. Box 39 describes the system of indicators of the British Employment Service, designed in the same spirit. It emphasises the measurement of results and impacts, rather than that of resources and outputs. Box 39 - Decentralisation and development of indicators The mission of the British Employment Service is to develop a competitive, effective and flexible job market. In particular, it must promote the integration of the most disadvantaged unemployed, through a system of specific funding. The national network of the Employment Service has over 1,000 agencies. Since 1994, the degree of autonomy of the agencies has increased. In return, each agency must calculate a series of results and impact indicators and compare its performance with that of neighbouring agencies. The indicators are aggregated by region and at the national level, in an Employment Service annual report. For example, in Northern Ireland, the running costs of the agencies in relation to the number of unemployed recruited, has evolved as follows: 1992-93 Baseline 1993-94 - 6% 1994-95 - 7% 1995-96 + 6% In many European regions, the administrative culture remains impervious to decentralisation and to performance management. It is therefore hardly Producing and using indicators surprising that the development of result and impact indicators is generally considered to be difficult. Programme managers are more familiar with resource and output indicators. Cultural changes are, however, slowly but surely taking place in certain countries, under pressure from administrative reforms initiated by national governments. The monitoring and evaluation of programmes co-financed by the European Union is a factor favouring performance management in terms of results and impacts. In a perspective of performance management, indicators play an increasingly important role for operators and must therefore be used with caution. Interpreting and comparing indicators Situations exist in which indicators require little interpretation, but these are exceptions. In general, indicators have to be interpreted by means of the relevant comparison or breakdown. In the following example, the comparison of three indicators showed that the training financed with European assistance did not reach the long-term unemployed as it should have. Box 40 - Comparing several indicators for revealing a phenomenon Within the framework of its intermediate evaluation of the European Social Funds in the Italian Objective 1 regions, the ISFOL gathered and interpreted a series of monitoring indicators. In certain cases, the comparisons made between these indicators led to strong and clear conclusions. For example, the part of the programme benefiting the long-term unemployed can be measured by means of three indicators: planned expenditure on the long-term unemployed: 18% of the total planned expenditure funds committed to the long-term unemployed: 11 % of the total funds committed funds actually spent for the long-term unemployed: 8% of the total funds spent The comparison of these figures clearly shows that the long-term unemployed are disqualified from the programme at each step of its implementation. For a reader who glances at the table of indicators and sees these three figures in isolation, the information would be meaningless. To reveal the problem of reaching the priority public, it was necessary to break down Indicators for monitoring and evaluation these three indicators in terms of the length of unemployment of the direct addressees or recipients, and for the ISFOL to think of comparing the three pieces of information. Whenever possible, it is interesting to compare programme indicators (e. g. number of unemployed trained) with the appropriate context indicators (e. g. number of unemployed in the region). This type of information can be presented in a concise way by means of derived indicators that express the programme outputs or effects as a percentage of the context indicator, like those described in Box 4 1. Box 41 - C SF (Objective and context ind 1) 1 9 8 9 - 93 - C o m p a r i s on of p r o g r a m me cators - Spain Number of digital telephone lines Recorded data 239 800 Funded universities 20 Data as a percentage 6 % of the digital lines installed in Spain during the period 91 % of the Universities in the Objective 1 regions Creation and improvement of irrigated land (in ha) 34 236 1,5 % of the surfaces irrigated in the Objective 1 regions Number of modernised farms Number of young farmers subsidised 107 000 4,8 % of the farms in Spain (1992) 11 000 0,5 % of the farms in Spain (1992) Number of persons trained 2 693 000 17,7 % of the working population in Spain Most evaluation professionals recognise indicators have to be interpreted by also relying on qualitative information. To interpret indicators, it is therefore necessary to consider the context as a whole, the factors which help to facilitate or hinder the performance of the programme, the raison d'être of the programme, and the process of implementation of all sorts of qualitative elements (Burt Perrin Associates, 1997). that An original technique consists of asking an expert panel, capable of combining the quantitative and qualitative elements of the situation, to interpret the performance measures. This technique has been used for about ten years to evaluate the performance of university research centres is monitored in the UK. The work of all the UK research centres Producing and using indicators quantitatively and periodically evaluated by expert panels. In their indicators (e. g. number of evaluation, the experts take into account publications) but also purely qualitative elements (e. g. quality of each researcher's in a four best publications). The evaluation classification of the research teams (on a scale of 1 to 5). The allocation of public funds is directly related to this classification (Bone & Carter, 1997). results The interpretation of indicators often makes use of comparisons between the performance of two interventions. This type of comparison mainly concerns efficiency indicators, for example: cost of a trainee's training; cost of a job created; cost of a home connected to the sewerage system; etc. However, comparisons are not always acceptable for programme operators and managers. In particular, performance comparisons will seem unfair to managers who work in the most difficult contexts. To make comparisons acceptable, the comparison procedures must be considerably refined with a view to putting those interventions functioning in a difficult environment, on an equal footing. An interesting technique was tested in the domain of education. The technique consists of reconstituting, for each intervention to be compared, a group of several similar interventions implemented in other regions in an environment with the same degree of difficulty. The comparison is made with the average of the interventions of this "benchmark group". Box 42 - Example of comparison of performance accepted by the operators The comparison of performance between training organisations often stumbles over the fact that the different publics trained do not have the same difficulties. A few years ago an evaluation team had to compare 133 training organisations (school districts) in different states in the U. S. The team decided to compare each organisation to a "benchmark group" of 14 other organisations with similarities as regards the characteristics of the public enrolled (poverty; home language; parents' level of education; belonging to a minority group; etc. ). To form the standard groups, the evaluation combined a factor analysis and expert panels. At the end of the evaluation, the performance comparisons were accepted without question. Henry & ai, 1992 This approach assumes that a performance indicator, measured in a comparable way, is available for a large number of interventions. The method is difficult to apply if the performance is judged on several criteria, Indicators for monitoring and evaluation indicators but is not then be adopted, in which and even more so if the performance is not completely measurable. A the comparison global approach must encompasses limited to them. The appropriate technique, in this situation, is that of benchmarking which consists of comparing an intervention to a model outside the programme, recognised for its good performance. Use of this technique implies that a reference for comparison can be found, for example in a data bank of the "best practices". Benchmarking is a tool which produces acceptable conclusions because it can take into account multiple, and not only quantitative, elements of comparison. One of the advantages of benchmarking is that it orientates the comparisons towards the best practices and that it therefore avoids convergence towards average performance. The benchmarking technique is described in Volume 3 of the MEANS Collection. Avoiding adverse effects of indicators Use of indicators is often hindered by the fear of provoking adverse effects. An example of adverse effects is the skimming-off effect which appears when the performance of training organisations is measured by the placement rate of trainees. To obtain a better placement rate for their trainees, it is in the organisations' interests to recruit people in the best possible situation who also meet the eligibility criteria. The operators therefore tend to "skim off" potential direct addressees or recipients by favouring those whose employability is higher. This effect is undesirable because it helps to focus assistance on those who are relatively less in need. to result indicators Another example of an adverse effect appeared after a few years of implementation of the British Audit Commission's system of indicators (see Appendix 2). This system organises the quantification of about two hundred output and the services offered by relating municipalities. The indicators are comparable from one town to the next and are published annually in the local press. This system creates a very strong impression of the town when one of the services performs badly. As a result, many municipalities increased the budgets of their services with the lowest performance, in the hope of raising the standard. In these cases, financial resources were sometimes drawn from the most effective services. Use of indicators thus caused a reduction in differences by convergence towards the average. This was an adverse effect since the measurement towards exercise would have been expected excellence. to push performance Producing and using indicators An adverse effect appears when an indicator encourages behaviour leading to sub-standard performance. This occurs when the indicator rewards undesired results or when the system causes the operators to work for the indicator rather than for the result. Adverse effects inevitably appear after two or three years of functioning of a system of indicators, no matter how well it is designed. These undesirable effects are generally not foreseeable. The probable appearance of adverse effects must not be an argument for refusing to measure performance. It is relatively easy to guard against them, either by amending the indicator causing the problem, or by creating a procedure for interpretation of the indicator by expert panels. It is then important to watch out for the appearance of adverse effects and to correct the system when these effects appear. Creating performance incentives Box 43 - Selection of projects in terms of their effectiveness Within the framework of the Objective 2 programme for the period 1989-93 in the Pays-de-la-Loire region (F), the ESF actions were managed by the Regional Council. For the management of the ESF, the services of the Regional Council applied a project selection method used for their own training activities. This method is based on performance management and functions in the following way. The training organisations filing their first application for funds are subjected to less severe selection procedures, on the basis of the quality of their projects. This selection mechanism is exceptional since "primary applications" are rare. Organisations already receiving assistance and which apply for a new subsidy (secondary applications) enter into the mechanism of normal project selection. In this normal procedure, funds are granted first to organisations with the best placement rate. The monitoring system obliges all organisations that offer subsidised training to provide the Regional Council with an evaluation of the placement rate of its trainees after six months. The effect of this mechanism is to favour those organisations which are most tuned in to the needs of business and which have the best impacts. There are several ways of using indicators to promote an improvement in operators' performance. These include: • Operators with poor performance receive specific technical assistance to help them progress. If the situation does not improve, the budget is Indicators for monitoring and evaluation restricted. This method functions on the principle that it is not the mistake that must be punished, but rather the inability to correct mistakes. • Operators with the best performance are granted greater autonomy and are controlled less. • Operators with the best performance receive support for presenting their outputs and results to the general public. • Operators who did not perform well enough are disqualified from the selection procedures for future projects, as in the example in Box 43. • Operators with the best performance are offered additional funds. Conclusion Box 44 summarises the main message conveyed by this Volume. It is presented in the form of a set of assumptions that are made, often implicitly, when a system of indicators is created with the intention of improving a programme's performance. Box 44 - The usefulness of a system of indicators The system of indicators is constructed \^ \^ It reflects the objectives and makes them more legible Information is disseminated to the public \. / / It produces the desired information * The information is correctly interpreted V 1 Γ The information is used to provide an incentive for better performance N. \ τ A watch is kept for perverse effects ' Managers make better decisions / / ' The performance of the programme improves ' / ! 1 1 Citizens are in a position to form an opinion The advantage of this diagram is that it shows that the construction of a system of indicators is not really a first step. At least six other steps occur before the system is actually useful. The first step must therefore be treated pragmatically, always keeping in view the prospect of maintaining the effort until the end of all the other steps. Appendixes Appendix 1 - General bibliography The bibliography concerning indicators is immense. The following titles have been selected because they are general and recent. Auditor General of Canada (1997), Reporting Performance the Expenditure Management System, Report of the Auditor General of Canada to the House of Commons, Chapter 5. in Auditor General of Canada (1997), Moving towards Managing for Results, Report of the Auditor General of Canada to the House of Commons, Chapter 11. Bone, J. , Carter, N. (1997), On the Performance of Performance Indicators, l'évaluation des Communication au colloque politiques publiques de l'école normale supérieure de Cachan, 5 / 6 February 1997. International sur Boyle, R. (1996), Designing and using performance indicators, Dublin Institute of Public Administration. Burt Perrin Associates (1997), 'Evaluation in Partnership: The May, 1997 Canadian Evaluation Society Conference' in: Evaluation 3(3), 377-379. Carter, N. , Klein, R. and Day, P. (1992) How Organisations Measure in Government. the Use of Performance Indicators Success: London: Routledge. Commission européenne (1994) Le rapport européen sur les indicateurs scientifiques et technologiques. Série: Sciences et technologies nucléaires. Luxembourg: Office des publications officielles des Communautés européennes. Commission européenne (1995) Orientations communes pour le suivi et les évaluations intermédiaires. Luxembourg: Office des publications officielles des Communautés européennes. Conseil du Trésor du Canada (1997), Rendre compte des résultats, Rapport annuel au Parlement, Direction des affaires publiques, Secrétariat du Conseil du Trésor du Canada. Derosières A. (1997) 'Refléter ou instituer : l'invention des indicateurs statistiques' in Dupoirier E. et Parado J. L. Les indicateurs socio- économiques aujourd'hui. Paris: L'Harmattan, 15-34. Indicators for monitoring and evaluation Halachmi, Α. , Bouckaert G. (Eds) (1995), Public Productivity through Quality and Strategic Management, Amsterdam: IOS press. Hatry, H. P. , Fall, M. , Singer, TO. , Liner, E. B. (1990), Monitoring the outcomes of economic development programs. A manual, Washington, D. C. : The Urban Institute Press. Henry G. , McTaggart M. , Macmillan J. (1992) 'Establishing benchmarks for outcome indicators, a statistical approach to developing performance standards'. Evaluation review, 16(2), 131-150 Innés De Neufville J. E. (1994), Knowledge and Policy, the Search for Meaningful Indicators. New Brunswick: Transaction Publishers. Kumar, K. (1995) Measuring the Performance of Agricultural and Rural Development Programs in Evaluation and Development. The proceedings of the 1994 World Bank Conference. Washington, D. C. : The International Bank for Reconstruction and Development. OCDE (1994) 'Mesure des performances et gestion axée sur les résultats', Gestion publique: La gestion des performances dans l'administration, Collection Etudes Hors Série, 3. Paris: OCDE. OCDE (1994) Créer des indicateurs ruraux pour étayer la politique territoriale, Paris: OCDE. OCDE, (1994), Making Education Count: Developing and Using International Indicators. Washington, D. C. : Centre for Educational Research and Innovation. Pollitt, C. (1986), 'Performance Measurement in the Public Services: Some Political Implications', in: Parliamentary Affairs, 39 (3) pp 315-329. Oxford University Press. Pollitt, C, Summa, H. (1995), Performance Audit: Travellers' Tales, paper presented at The Canadian Evaluation Society and the American Evaluation Society Conference, October 1995. Newcomer K. E. (1997) 'Using Performance Measurement to Improve Programs', in: New Directions for Evaluation, (75), 5. Salzer, M. , Nixon, CT. , Schut, L. J. A. , Karver, M. S, Bickman, L (1997), Validating quality indicators. Quality as relationship between structure, process, and outcome, Evaluation Review, 21(3). London: Sage Publications, 292-309. Appendixes Sorber, Α. (1993), 'Performance measurement in the Central Government Departments of the Netherlands', Public Productivity and Management Review, autumn, vol. 17(1). United Nations Development Programme (1997), Results-oriented Monitoring and Evaluation, New York: UNDP, OESP Handbook Series. Vakkuri, J. , Meklin, P. (1997), Risks in value for money-related performance measurement, Leuven: Conférence annuelle du Groupe Européen d'Administration Publique. Wholey, J. S. , Newcomer K. E. (1997) 'Clarifying Goals, Reporting Results', New Directions for Evaluation, (75), 91. Wishlade, F. , Yuill, D. (1997), Measuring disparities for area designation purposes: issues for the European Union, Regional and Industrial Research Paper Series, N°24. Glasgow: European Policies Centre, University of Strathclyde. World Bank - Operations Evaluation Department (1997) Evaluating Development Operations: Methods for Judging Outcomes and Impacts, Washington: World Bank, OED, Lessons & Practices, Number 10. Appendix 2 - Lessons from international experience In the 1970s this success gave rise to International experience concerning indicators is vast and has existed for a long time. At the origin of all developments in this domain, we find the success of a few major macro-economic indicators (GDP, price index, unemployment rate). the development of social indicators, an activity which mobilised considerable energy and was the source equally great disappointment. Numerous lessons were drawn from this experience and were soundly established. They could almost be summarised in the titles of the three noteworthy publications by Judith Innes de Neufville: Social indicators and public policy: (1975); Disappointments and Legacies of Social Indicators (1989); Knowledge and Policy: the Search for Meaningful Indicators (1994). of design and application processes interactive Whereas more recent developments called "performance measures" are appearing, some public invest, sometimes massively, in indicators. For example, the UN and the World Bank launched a 2 M euros project to develop a system of indicators for the urban environment. institutions continue to The countries involved can give only a very partial view of the experience which has been capitalised world-wide. The examples have been chosen for their similarities with the main subject of this Volume. Science and Technology Indicators at the Commission Massive investment Initially, from 1988 to 1992, the European Commission (DG XII) formed a network of experts on the basis of one specialist per country. This network held two meetings per year for four years, under the aegis of the MONITOR programme. Two large conferences were organised in Paris and Leiden (about 100 participants). The Commission's investment in this network represented about 4 persons χ month per year. Part of the network leadership was sub-contracted to an outside consultant. At the end of the MONITOR programme the Commission stopped funding the network, although it supported a new meeting held recently in Antwerp. After this first investment, DG XII formed an "Indicators" unit, representing an investment of 30 to 70 persons χ month, depending on the year. This unit Indicators for monitoring and evaluation sub-contracts specific tasks to numerous consultants. The unit issues a 400-page report every two years (the last editions were in 1994 and 1996). This report presents research context indicators in 50 countries, as well as resource, output, results and indicators for European policy impact (framework programme). It is partly based on the work of Eurostat, which devotes a large proportion of its resources to science and technology indicators (e. g. REDIS News, Research and Development: annual statistics). Highly favourable international environment There is a strong convergence of the Commission's interests and those of certain Member States. France has created a Science and Technology Observatory and publishes a report similar to that of the Commission. The Netherlands also invests substantially, with the MERIT programme, and Germany has developed systematic surveys on research and innovation. The United States has, in many respects, been a forerunner in this field, with significant investments in S&T indicators by both public and private organisations (NSF and the Science Citation Index, respectively). Thus, the Commission's efforts concerning science and technology indicators have been supported and encouraged by previous developments of at least equal importance, as in the USA and certain Member States, and the OECD. Problems in regionalising indicators In 1993 and 1994 an interservices commission drew up a complete inventory of the possibilities of regionalising R&D indicators. This work was presented in a report and in a chapter on regional R&D indicators in DG XII publications. The interservices commission then carefully selected all the indicators that could be quantified on a regional level and which had good information potential for regional policy. It turned out that certain indicators could be regionalised: R&D Spending; Employment in R&D; and patents. The quantification was rarely broken down below the NUTS 2 level. These limited results reflect the difficulty of regionalising indicators. International OECD indicators Twenty years later in the domain of education the The OECD intermediary of its Centre for Educational Research and Innovation. The invested massively in education indicators through Appendixes effort can be traced back to the 1970s when it was part of the social indicators movement. Initial ambitions were too great and the disappointments of the late seventies nearly cost the programme its life. The current analysis of this setback is as follows: research was the predominant focus, the indicators were expected to show too much, and they were not linked closely enough to the main political issues. The work was revived at the initiative of the American government which, from 1983, became heavily involved in improving the effectiveness of education. Towards the end of the 1980s, the OECD its programme for developing indicators, through a series of international conferences supported by volunteer countries (e. g. USA, France). The programme produced numerous results in 4 years (at least 7 significant publications, including "Education at a Glance, OECD indicators"). launched Contrary to its initial ambitions, current work is considered by the OECD to be highly pragmatic. The system of indicators has been constructed without any coherent theoretical model. The theory remains to be done - if ever it can be done in such a highly sensitive field. The other areas in which the OECD has invested heavily in terms of indicators are: Science and Technology; Health; Development aid; the Environment; Tourism; Fisheries and Agriculture. The working method is roughly the same as in the domain of education, i. e. : • the initiative is supported by one or more countries highly involved in the development of a system of indicators for their own use; • an international task force is formed for several years, to negotiate a common language and set pragmatic objectives; • a detailed questionnaire is sent to the Member States which provide the best quantification possible for the indicators selected by the task force; • the first version of a directory is published with information partially, or totally lacking for several countries; • all the countries progressively complete their statistics in order to join the project. The development of systems of indicators is very progressive. For example, the first directories for the environment contained indicators which were new for many countries. Thus, the quality of their quantification varied widely. However, these directories act as models and the Member States Indicators for monitoring and evaluation are gradually equipping themselves so as to be able to answer OECD questionnaires and thus obtain references for international comparison. Régionalisation and territorialisation of statistics is potentially The OECD has a territorial development service which interested in indicators developed for the Structural Funds. However, the only progress made by the OECD in this field has concerned rural development. The OECD Rural Development Programme was launched in 1991. It developed a set of indicators concerning population (density; evolution; structure; households), the economy (working population; employment; sectoral structure; productivity; and investment), social welfare (income; housing; education; health; and security) and the environment (topography and climate; land use; habitats and species; ground and water; air quality). A rural indicators data base gives internationally comparable information for categories of territory corresponding essentially to degrees of population density. The reason for the OECD not developing other territorial indicators is probably that the Organisation is made up of countries and is not supposed to concern itself with regional affairs. The régionalisation of systems of indicators could be justified in the framework of co-operation between member states, in the field of regional policies, but such co-operation is very limited. This is because the United States does not have significant regional policies comparable to those of Europe and Canada. World Bank monitoring and evaluation indicators Operations financed by the World Bank account for an annual expenditure virtually equivalent to that of the Structural Funds. Self-evaluation reports are systematically written at the end of each operation, by the operations departments. These reports are submitted to the borrower countries for their opinions. About 280 self-evaluation reports are produced annually. The Bank has an Operations Evaluation Department (OED) primarily for the purposes of "counter-evaluation" (about a quarter of the reports submitted by operational departments are verified in the field), "returning to the field" after five years (about 10% of the operations), more in-depth analysis of certain themes (e. g. women's participation), evaluations by country, and periodic checks of the monitoring and evaluation procedures. Appendixes For a long time the Bank's information system focused on volumes of financial commitments and on physical output indicators. The Wapenhans report in 1992, recommending that "results in the field" be given more attention, was a turning point. The Bank then undertook to establish menus of results and impact indicators for each domain. In 1993 and 1994 each sector service published its menu of indicators, to which a list of generic indicators was added. This work was consulted and used in the present study. The quality of the work was found to vary widely, with many indicators oriented more towards a description of context than towards results and indicators for specific domains. Moreover, these menus are generally abundant but hardly related to the Bank's objectives. The OED recently audited the Bank's monitoring / evaluation practices. It showed that the use of indicators is way below the stated ambitions. Yet, in a few years, the proportion of operations using results and impact indicators has risen from 12% to 36%. Without abandoning its project to enhance these menus of sectoral indicators, the Bank now places the emphasis on the use of a limited number of indicators peculiar to each operation. These indicators are to be constructed in relation to the specific logic of each operation, by means of the logical framework method. Efforts point towards a high degree of decentralisation in the development of indicators. Rather than unifying systems of indicators, the idea is rather to unify the operations managers' procedures for constructing indicators. In this bottom-up approach, comparability is a secondary concern. The OED's instructions are as follows: • use a very small number of indicators for each operation (between ten and twenty); these indicators must cover all the logical steps: resources, outputs, results and impacts. • favour risk indicators, likely to sound the alarm in cases of failure of the operation. • Performance indicators at the Audit Commission Development combining obligation and partnership In July 1991 the British government launched the "Citizens' Charter", an initiative aimed at improving the quality of public services. One of the key points in this initiative was the development and use of output and result indicators. Indicators for monitoring and evaluation The Audit Commission is responsible for developing and managing a system of indicators applied to about one thousand local authorities in England and Wales. The role of the Commission and local authorities varies, depending on the case: • Initially, the local authorities are always encouraged to set their own output and results objectives and to choose appropriate indicators for monitoring them. • In certain cases, the Commission sets the indicators and the way of quantifying them, but does not define objectives. The Commission may also act in a more authoritarian way by deciding that certain indicators must be quantified and that the local authorities must set their own objectives for these indicators. Lastly, the Commission may decide on both the indicators and the objectives. Each year, the Audit Commission publishes a manual with the detailed list of about 200 indicators that must be measured by the local authorities during the following year. No authority is obliged to quantify more than 130 indicators. The their performance measurements in the local newspapers. The Commission regularly gives itself the means to ensure that the required systems are in place and produce accurate information. local authorities are legally obliged to publish relating The Commission produces a report in three volumes (Local Authority Performance) which presents a synthesis of all data to performance, with national averages. Performance indicators are also available on computer disk. A limited report concentrates on about forty comparable indicators aggregated at national level. This report includes between one and five indicators for each of the following fifteen fields: Education; Social affairs; Libraries; Expenditure management; Social housing; Recycling; Planning; Tax collection; Inspection of food outlets; Handing of complaints, calls to the police and emergency services; Crime and investigations; Resources of the police and fire brigade. For instance, the percentage of recycled household waste is one of the indicators of the "Recycling" domain. Schedule for implementation of new indicators To update and improve the system, the Commission maintains in-depth interaction with local authorities, consumer groups, government services, Appendixes professional bodies and other stakeholders. It also employs consultants to carry out national surveys to determine what must be measured, what is understood by good performance, what should be published, etc. The introduction of a new indicator takes about four years, according to the following schedule: Yearl September Consultation and survey December Guidelines published by the Commission Year 2 March Beginning of performance measurement by the Local Authorities Year 3 March First performance measurement December Year 4 March Publication of performance measurement in local newspapers Publication of performance measurement in the national annual report Eligibility indicators for German regional policy It seems that Germany is the most advanced European country in the use of statistics for regional policy purposes. For several years, local and regional statistics have been the object of long-term investments and have been used effectively in policy planning. The following lines attempt to summarise an old and varied practice. Use of indicators for eligibility is partly responsible for the The German regional planning ministry "Common Task" (Gemeinschaftsaufgabe), which makes up German regional policy. This policy is managed intergovernmentally, by a planning committee consisting of ministers from both levels (Federal and the Länder). Traditionally, the areas eligible for regional support are determined rigorously on the basis of a system of indicators. This system is revised periodically, under the responsibility of the planning committee and a sub committee of experts. The last revision mobilised several institutes: Bundesforschungsanstalt für Landeskunde und Raumordnung (BFLR) - Bonn, für Arbeitsmarkt und Berufsforschung - Nürnberg. - Dortmund, and für Raumplanung Institut Institut the The current system consists of four key indicators: unemployment rate; wage level; infrastructure index (see below); and manpower requirement Indicators for monitoring and evaluation forecasts. The infrastructure index has been one of the key indicators for regional policy since the 1980s. In its original form, the index consisted of 22 sub-indicators. The new index does not fully warrant its initial name. It consists of 10 sub-indicators: four in the transport domain, five in the human resources domain, and one population density indicator. For transport, the four indicators are as follows: accessibility by car of the closest regional centre; accessibility by train of the closest regional centre; accessibility by any means of transport of the main European cities (see following paragraph); travelling time to the closest train/bus station. For human resources, the five indicators are as follows: number of young people in occupational training exceeding the capacities of the local industry; number of places in occupational training in regional public institutions; percentage of the regional working population benefiting from occupational training; percentage of the regional working population benefiting from technical training. Accessibility indicators The BFLR calculates an accessibility indicator for the 34 largest European cities, by the best means of transport. Calculation of the indicators makes use of a model associated with a European data base. It is applied to the main towns in all the NUTS 3 areas in all the Member States. This indicator appeared in a study for DG XVI in 1992. The last quantification was made in 1995. the gain in accessibility Accessibility indicators are used for determining eligibility. They are also very useful in ex ante evaluations. For example, a recent study by the BFLR estimated these two basic infrastructure programmes in the new Länder. The first programme concerned 11 new railway links between the western and the eastern Länder. The second programme was composed of motorway projects. The evaluation showed an increase in accessibility that was far greater in the case of the first programme than in that of the second one. There is no example of use of accessibility indicators for intermediate or ex post evaluations. resulting from • An original approach to the régionalisation of statistics Over and above its responsibilities as regards Regional Policy indicators, the BFLR also runs a regional statistical data base, similar to the Eurostat REGIO data base. These regional statistics came into existence in 1975. At the time, Appendixes it was discovered that the German statistics institutes were organised in such as way as to make any comparison on a level lower than the regions impossible. In fact, the federal office was only responsible for comparability at the Länder level. Only the offices of the Länder were in a position to harmonise definitions at the sub-regional level. Today, owing partly to the efforts of the BFLR, 95% of all local data are comparable. The BFLR orders special temporal series from the statistics institutes (e. g. migratory flows into and out of the territory), amounting to about 100 K euros / year. The BFLR also supports data collection at the sub-regional level. Regional data are disseminated in various ways, such as publication of a 300-page statistical report every two years and provision of data to the different ministries and to private companies and academic institutions which can procure updated disks at a reasonable price (approx. 100 customers / year). The data base relies partly on a regional observation system which gathers data by means of regular surveys on samples of about 4,000 households. Questions concern population, mobility, housing, social environment, foreigners, etc. Because of the method used to gather the data, the system supplies statistics not on particular areas but on only a few categories of area. The main categories (type of housing) combine the degree of centrality of the region with the local population density. Still close to the field of regional policy, the BFLR is currently undertaking interesting work on intra-urban statistics. The aim of this work is to provide data on population, housing, land use, and mobility in relation to the different categories of urban area (city centre, suburbs, etc. ). The BFLR has created a network of 60 volunteer towns and has developed an initial list of statistical series to collect. After a few years of working together, 35 towns supply all the required data. This project has produced comparable data and national averages. It has attracted more and more volunteer towns. A similar process has been launched in the same field, on a European scale, by Eurostat. Appendix 3 - Programme indicators for the 1994-99 period: a state of the art The elements presented in this appendix were drawn from an inventory of the indicators cited most frequently in Objective 1 and Objective 5b programming documents for the period 1994-99. This inventory is limited to thirteen typical interventions covering the sectors with the largest budgets. The list drawn up on this basis was submitted to about fifty professionals who are thoroughly familiar with the Structural Funds. They were asked to select those indicators considered to be realistic and relevant. This selection affords a view of the state of the art on the question of indicators during the 1990s. Basic infrastructure Indicator Unit and definition Application Length of road/rail Length of journey Hours of transport Population served Km of road/rail built/improved, broken down by type: road, motorway, standard railway, high-speed railway, etc. Transport Road, motorway, railway Hours of travelling between two points. Shortening of time in % All types of transport Total number of hours saved per year by users Number of households served All types of transport Water and environment Supply networks for water/ gas/electricity Water & environment Water supply network Water consumption m3 (thousands) of water consumed per year per household. Waste processing facilities Volume of waste Number of facilities installed. Water & environment Waste/sewerage treatment Tonnes of waste treated per year broken down into type of treatment (incineration, recycling) Water & environment Waste treatment Indicators for monitoring and evaluation Human resources Indicator Unit and definition Application Hours of training proposed Trainees having completed the course Placement rate Duration (hours) of training financed broken down by type of training programme Training Number of trainees broken down by status of the trainee in the job market (unemployed, long-term unemployed, employees, employers) Proportion of trainees who found a job broken down by type of beneficiary (man/woman, age, etc. ) Training / R&D Training Training Rate of coverage Proportion of eligible public (e. g. long-term unemployed recruited by the programme) Appendixes Productive environment Indicator Unit and definition Application Partnerships between research and enterprise Surface area of developed business areas Occupation rate of business areas Survival rate of businesses Investments generated Value added / wages Export turnover Assisted economic units Operations to promote quality Surface of protected sites Visitors to protected sites Jobs created Number of businesses participating in joint research projects, in partnership with an R&D centre Hectares Broken down by type of area (new development or rehabilitation) Measures for promoting research and innovation Measures for developing business areas Surface built in percentage of building land Measures for developing business areas Number of businesses which remain active, as a share of all assisted businesses broken down in terms of the period of observation (2 years, 5 years) In euro, amount of investments generated by the assisted economic units In euro, value added generated by the assisted economic units In euro, turnover generated by exports in the assisted economic units broken down by size (SME, other) and by sector of activity) Measures for assisting the creation of new businesses All measures All measures Measures to promote exports Number broken down by size (SMEs, others) All measures that include assistance Number Measures to promote quality Surface in hectares Number of visitors per year Measures to protect the rural environment Measures to protect the rural environment Number of jobs created owing to aid granted to the assisted economic units All measures that include assistance Appendix 4 Grid for assessing the quality of a system of indicators This appendix proposes a procedure for analysing the system of indicators of a programme and for assessing its quality. The procedure consists of seven steps. Defining the system to be assessed (1) Specify the document to be analysed. This may be: • a single programming document (SDP) and its appendixes • a data base used by the programme managers • one or more reports submitted to the Monitoring Committee • a report published by an observatory Dividing the system into homogeneous components (2) A component is homogeneous if: • it is possible to apply the same output indicator to all the projects; • all the projects have the same target public. The homogeneous components are generally measures or actions. For each homogeneous component, note is taken of the budget allocated to it and its budgetary weight as a percentage. The following steps from 3 to 6 are taken for each homogeneous component. Identifying and counting indicators (3) In the analysed documents all the information that can be qualified as indicators is looked for. All quantitative information is likely to be an indicator. After this step, the ratio of "density of indicators" is calculated for each component (number of indicators / total planned expenditure, in hundreds of millions of euro). This ratio is an element for assessing the relevance of the system. The density of indicators must be homogeneous. It may also be more important for the most strategic measures. Indicators for monitoring and evaluation How to count the indicators of a system ? • The following conventions are adopted: • an indicator broken down into parts is counted as a single indicator (e. g. table with the number of applications broken down per year and per target public) • if an indicator is broken down a second time, it is counted twice (e. g. 2nd table of the number of applications broken down per year and per territory) • a compound indicator is counted as a single indicator if its components are not quantified separately and as several indicators if they are. Classifying the indicators by category (4) When an indicator is identified, it should be placed into one of the following four categories: resources, outputs, results, and impacts, as defined in this Volume. At the end of this step, the percentage of components which have at least one result or impact indicator must be counted, for the entire programme. Attributing a quality score to each indicator (5) Each indicator is given a score, from 0 to 3, for the applicable quality reliability, comparability, criteria: availability, normativity, and meaning. freshness, sensitivity; Multicriteria scoring for each component (6) For each component, the average scores are calculated for each criterion (availability, freshness, sensitivity, reliability, comparability, normativity, meaning) and for each category of indicator (resources, outputs, results, impacts). Then the average score per category is calculated. The result may be represented in the form of the following table: Appendixes A v e r a ge scor e of t he in d i c a t o rs of a c o m p o n e nt Category Number of indie. Average score per criterion and per category Score per category Resources Outputs Results Impacts 1 10 3 0 crit. 1 c r i t2 3,0 3,0 2,1 / 3,0 2,5 1. 1 / crit 7 3,0 2 0,9 / Average 3,0 2,2 1,2 0,0 1,6 With this procedure a maximum score can be given to a component which has only one indicator in each of the four categories, provided that these indicators are of a high quality. The average score decreases if more indicators are added and if they are of not of the same quality. Thus, this scoring procedure favours the quality of the system and not the quantity of the indicators. Moreover, the average score declines slightly if there is no result or impact indicator. Attributing a global score to the entire system (7) The global score of the entire system is the average weighted score of the indicators of all the components. These are scores obtained in step 6, weighted by the budgetary weight of the corresponding components (see step 2). The assessment procedure described above has integrated, in its very construction, the global assessment criteria: coverage, balance, selectiveness and relevance (see Box 33). Appendix 5 - List of examples Example 1: New section of motorway connecting A and Β Example 2: Improvement of a railway line between A and B, bringing it up to the high speed range Example 3: Expansion of a regional airport Example 4: Improvement of a container terminal in a sea port Example 5: Upgrading of a railway line for freight operations Example 6: Building an intermodal (rail / road) centre Example 7: New electricity plant using fossil fuels Example 8: Constructing or upgrading an electric power distribution network Example 9: Upgrading a gas distribution network Example 10: Financing of training for new trades for the long-term unemployed Example 11: Skills improvement programme for young people with few qualifications Example 12: Assistance to businesses for adapting their employees' skills Example 13: Support for the creation and improvement of accommodation Example 14: Support for the creation of tourist facilities (museums, amusement parks, etc. ) Example 15: Funding for language training Example 16: Grants for aquaculture farms Example 17: Subsidies for laying up fishing vessels Example 18: Modernisation of a fishing port Example 19: Support for retraining of fishermen Example 20: Support for a science and technology park for SMEs 53 54 55 56 57 57 63 64 65 72 73 74 80 81 82 87 88 89 90 97 Indicators for monitoring and evaluation Example 21: Financial and informational support for technology acquisition by SMEs Example 22: Support for post-graduate research Example 23: Financial support to assist the setting up of young farmers Example 24: Supported investment in agricultural product processing plants Example 25: Village renewal - public and private projects Example 26: Support for establishing tourist accommodation on farms Example 27. Establishment or regeneration of community centres Example 28 Improvement of water supply system Example 29. Construction of wastewater treatment plants Example 30. Improvement of solid waste management facilities 98 99 105 106 106 107 112 118 119 120 Example 31 Support for new SMEs providing environmental goods & services 121 Example 32: Construction of an inter-urban light railway between A and Β Example 33: Deployment of a broadband Hybrid Fibre Coax (HFC) network Example 34: Upgrading of an existing narrowband network with ADSL technology Example 35: Setting up of a suburban telework centre Example 36: On-line education project in professional training Example 37: Support for entrepreneurs creating new businesses 122 128 129 130 131 138 Example 38: Informational support for SMEs to promote exports 139 Example 39: Subsidised infrastructure for setting up of firms Example 40: Support for projects in regional research centres Example 41: Creation of new organisations providing debt counselling and local credit schemes 140 141 147 Appendixes Example 42: Renovation of a rundown urban area Example 43: Support for community organisations Example 44: Training programme for long-term unemployed Example 45: Support for the construction of a conference and exhibition centre Example 46: Direct support for capital investment in enterprises Example 47: Venture capital scheme for small business development 148 149 150 161 161 162 Example 48: Return-to-job programme for long-term unemployed 168 Example 49: Assistance for the setting up of an equestrian event centre Example 50: Construction of a new hospital Example 51: Upgrading a bus system in neglected urban areas 169 175 176 Index Index accountability 197 actor 29 addressee or recipient 30 adverse effect 115, 206 aggregation 2 1, 197 Agriculture and rural development 103 availability 193 Β balance 195 baseline 26 benchmark group 205 benchmarking 152, 206 displacement effect 34,167,169,171 double counting 188 Economic development 155 effect 34 effectiveness 35, 207 efficiency 35, 205 elementary indicator 19 ELIZE data base 192 Employment 165 endowment 26, 27, 5 1, 116 Energy infrastructure 61 Environment 115 Equal opportunities 111 best correspondence approach 192 evaluation indicator 37 best practice 206 breakdown 20 expert panel 204 externality 33 comparability 194, 196 Competitiveness of SMEs 135, 161 compound indicator 20 confounding factor 26, 34, 167 context indicator 24, 26, 184, 204 coverage 195 field 43 Fisheries 85 freshness 194 deadweight 34, 167, 168, 169 derived indicator 20, 204 descriptor 18 direct addressee 25, 29, 30, 32, 36, 40, 206 disaggregation 20 disparity 26, 157 generic indicator 2 1, 186, 198 global impact 28, 29, 43, 184 global objective 24, 28, 155 Gross Domestic Product 156 gross effect 34, 168 Η harmonisation 189 I immaterial services 40 impact 33 indicator of structural composition 193 indirect addressee 29, 30, 40 information feedback 182, 189 Information society 125 information system (GISCO) 193 interpolation method 192 interpretation 203 key indicator 2 1, 198 knock-on effect 137 logical framework 28 M mainstreaming 111,135,158,159 management by results 37 meaning 194 monitoring indicator 35 N need 26,. 34 net effect 34, 168 normativity 194 Indicators for monitoring and evaluation performance 35, 202, 207 physical outputs 39 priority 43 programme funder 29 programme indicator 24, 184, 204 qualitative information 204 Quality of life 173 QUID data base 192 REGIO data base 191 relevance 34, 195 reliability 194 Research and Technological Development 93 resource 28, 29, 31 responsibility 37 result 28, 29, 32 sector 43 selectivity 195 sensitivity 194 skimming-off effect 206 specific impact 28, 29, 33 specific indicator 21 spin-off 33 operating report 18, 182, 187, 189 standard indicator 197, 198 operator 29, 188, 207 output 28, 29, 32 substitution effect 34, 167, 169, 171 sustainable development 115 Index τ temporary employment 52, 166 theme See field Tourism 77, 161, 169 Training 69, 168 Transport infrastructure 49, 176 υ unexpected impact 33 Urban development 145 " value added 156 Acknowledgements The preparation of this Volume took place over several years from 1996 to 1998. The preparatory work included the following steps : • Several workshops and meetings allowed experts and European desks officers to provide their points of view on indicators for evaluation. • A survey was conducted of several institutions which have implemented systems of indicators. • A questionnaire survey was undertaken on the subject of indicators on the occasion of the second MEANS conference in Berlin. The production of this Volume was coordinated by Andrea Mairate (D. G. XVI. G. 2 : Coordination of Evaluation) and involved several Directorate-Generals of the European Commission, notably D. G. II, V, VI, XIV and XVI and in particular Philippe Goybet, Anne Bucher, Anastassios Bougas, Miguel Benito Alonso, Sylvie Finne, Olivier Rouland and Patrick Salez. This Volume has been written by the Centre for European Evaluation Expertise, notably with the participation of, Cristina Diaz Gomez and Kevin Williams. It was translated into English by Liz Libbrecht. The authors wish to thank all those who contributed, either directly or indirectly, to the production of this Volume, and in particular the following experts: Transport infrastructure: José Viegas, Cesur, Lisboa Energy infrastructure: Maurizio di Palma, Ecoter, Rome Training: Patrick Werquin, Cereq, Marseille Tourism: Daniel Poulenard, Parcourir, Toulouse Agriculture and rural development: Gunter Tissen, Fal, Braunschweig Information society: Paolo Manacorda, Reseau, Rome Competitiveness of SMEs and enterprises in general: Ivan Turock, University of Glasgow Urban development: Tony Bovaird, Aston University, Birmingham Economic development: Henri Capron, Université Libre de Bnjxelles Quality of life: Sanjiv Lingayah et Alex Macgillivray, New Economics Foundation, London. European Commission Evaluating socio-economic programmes Selection and use of indicators for monitoring and evaluation Luxembourg: Office for Official Publications of the European Communities, 1999 1999 — 252 pp. — 14. 8 χ 21 cm ISBN 92-828-6626-2 Venta · Salg · Verkauf · Πωλήσεις · Sales · Vente · Vendita · Verkoop * Venda · Myynti · Försäljning BELGIQUEÆELGIE OSTERREICH J«an D· Laiinoy Avenue du Roi 202. Koningslaan 202 B1190 BnjxellesHrussel Tél. (322) S38 43 08 Fax (322) 538 08 41 Email: («an de lannoyOmfoboard. be URL: http. 'www jeandelannoy be La librairie européenne/ De Europese Boekhandel Rue de la Loi 244/Wetstraat 244 B1040 Brux alles/Brussel Tel. (32-2)295 26 39 Fax (322) 735 08 60 Email. mailOlibeurop. be URL http/Zwww libeurop. be Moniteur belge/Belgi »c h Staatsblad Rue de Louvain 4042/Leuvenseweg 4042 B1000 Bruxelles'Brussel Tél. (322)552 22 11 Fax (322) 511 01 84 DANMARK J. H. Schultz Information A/S Herstedvang 1012 DK2620 Albertsiund TU (45) 43 63 23 00 Fax (45) 43 63 19 69 Email 5chulU0schulU. dk URL: http://www. schuiu dk DEUTSCHLAND Bundesanzeiger Verlag GmbH V e ft neb sa b le ι lung Amsterdamer Straße 192 D50735 Koin Tel. (49221)97 66 80 Fax (49221) 97 66 B2 78 EMail: vertrieb O bundesanzeiger de URL: http://www bundosanzeiger. de ΕΛΛΛΛΑ GREECE G. C. Eletiheroudakls SA International Bookstore Panepistimiou 17 GR10564 Athina Tel (301) 331 41 8071/2/3/4/5 Fax (301) 323 98 21 Email eiebooksOnetor. gr ESPANA Boletín Oficial del Estado Trafalgar. 27 E2B071 Madrid Tel (34)915 38 21 11 (Libros). 913 84 1 7 l 5 ( S u s c n p) Fax (34) 915 38 21 21 (Libros). 913 B4 17 14(Suscnp) Email. dieniesO com. boe. es URL: http//www boe es Mundi Prensa Libros, SA Castella, 37 E28001 Madrid Tel (34)914 36 37 00 Fax (34) 915 75 39 98 Email litweriaOmundiprensa es URL: http//www mundi prensa com FRANCE Journal officiel Service des publications des CE 26. rue De san F75727 Paris Cedex 15 Tel (33) 140 58 77 31 Fax (33) 140 58 77 00 URL: http //www (oumalofficiel gouv ft IRELAND Government Supplies Agency Publications Section 45 Harcourt Road Dublin 2 Tel (3531)661 31 11 Fax (3531)475 27 60 ITALIA Licosa SpA Via Duca di Calabria. 1/1 Casella postate 552 150125 Firerne Tel (39)055 64 83 1 Fax (391055 64 12 57 Εmari iKXisaOtTbcc n URL: http www ftbcc itflicosa LUXEMBOURG Mani sehe Verlags und Unlversltåtsbuchhandlung GmbH Kohlmarkt 16 A1014 Wien Tel (431)53 1611 00 Fax (431) 53 16 11 67 EMail bestellen Omani co at URL. htlp //www mani, a L'index htm PORTUGAL Distribuidora de Livros Bertrand Ld. ' Grupo Benrand, SA Rua das Tenas dos Vaies, 4A Apartado 60037 P2700 Amadora Tel. (351-1)495 90 50 Fax (3511)496 02 55 Imprensa NacionalCasa da Moeda, EP Rua Marquês Sa da Bandeira. 16A P1050LtsDoa Codex Tel (3511)35303 99 Fax (3511) 353 02 94 Email: dehncmemail. telepac. pt URL: httpJ/www inem pt SUOMI FINLAND Akateeminen Kirjakauppa/ Akademiska Bokhandeln Keskuskatu 1/Centralgatan 1 PL/PB 12B FIN00101 HelsinkLrHelsrngfors Ρ. /tfn (3589) 121 44 18 F/lax (3569)121 44 35 Sah k o po s 11. akali la us β aka 1 ee minen o URL: http//www akateeminen com SVERIGE B T J AB Traktorvägen 11 S221 82 Lund T!n(4S46) 1800 00 Fax (4646) 30 79 47 Epost bt|eupubObt|. s< URL: http //www. btj. se UNITED KINGDOM The Stationery Office Ltd International Sales Agency 51 Nine Elms Lano London SWa 5DR Tel (44171)873 90 90 Fax (44171)873 84 63 Email: ipa encuinesStheso. co uk URL: htlp//www ihestationeryofficeci ISLAND Bokabud Larusar Blónda I Skulavordustig. 2 IS101 Reykjavik Tel (354)551 56 50 Fax (354) 552 55 60 NORGE Swets Norge AS Oston joveicn 18 Boks 6512 Etierslad H-0606OSIO Tel (47-22)97 45 00 Fax (47-22) 97 45 45 SCHWEIZ SUISSE'SVIZZERA Euro Info Center Schweiz c/o OSEC Stampfenbachstrafle θ 5 PF 492 CH8035 Zürich Tel (411)365 5315 Fax (411)365 54 11 Email eesOosecch URL: hnp//wvfwosec ch/etes BALGARUA Europress Euromedia Ltd 59. blvd Vitosha BG1000 Sofia Tel (3592) 980 37 66 Fax (3592) 980 42 30 Email: MilenaOmbox en bg CESKAREPUBLIKA USIS NlSprodojna Havelkova 22 CZl3OO0Praha3 Tet (4202)24 23 14 66 Fax (4202) 24 23 11 14 Email nkpospOdec ms cz URL http7/usiscr cz s du livre SARL CYPRUS HRVATSKA Medlatrade Ltd Pavia Haua ι HR10000 Zagreb Tel (3851)481 94 π Fax (3851)481 94 11 MAGYAROHSZAG Euro Into Service Europa Haz Margitsziget PO Box 475 H1396 Budapest 62 Tel (361)350 80 25 Fax (3611350 90 32 Email: euroinfoOmail matav hu URL: http Λν/ww. eu rointo. hu/index. htm ROY International 41, Mishmar Hayarden Streel PO Box 13056 61130 Tel Aviv Tel ¡9723)649 94 69 Fax (9723) 648 60 39 Email. royilOnetvision nel il URL: http://www. royint co il Subagent for the Palestinian Authonty: Index Information Services PO Box 19502 Jerusalem Tol (9722)627 16 34 Fax (9722) 627 12 19 MALTA Miller Distributors Ltd Malta Inte mal lonal Airport PO Box 25 Luqa LOA 05 Tel. (356)66 44 88 Fax (356) 67 67 99 Email: gwirthOusa net POLSKA Ara Polona Krakowskw Przedmiescie 7 Skr poeztowa 1001 PL00950 Warszawa Tel. (4822)826 12 01 Fax (4822) 826 62 40 Email ars_potebevy. hsn com. pl ROMANIA Euromedia Sir Gral Bermelo! Nr 4 RO70749 Bucuresti Tel (401)31544 03 Fax (401 ) 314 22 S6 RQSSIYA CCEC 60lelrya Oktyabrya Av. 9 117312 Moscow Tel. (7095) 135 52 27 Fax (7095) 135 52 27 SLOVAKIA Centrum VTI SR Nam Slobody. 19 SK81223 Bratislava Tel (4217)54 41 83 64 Fax (4217) 54 41 83 64 Email. europOtbbl s\\Y stuba ι URL http /www sitk. stuba sk SLOVENUA Gospodarskl Vestnlk Duna|ska cesta 5 S LO1000 Ljubljana Tel (386)613 09 16 40 Fax (386) 613 09 16 45 Email eurooíígve. tr k :. · URL: http ,Vwww gvestnik. si TURKIYE Du nya Infot el AS 100. Yil Mahallossi 34440 TR80050 Bagcilarlslanbul Tel. (90212)629 46 89 Fax (90212)629 46 27 Email infoici O duny agaze te c AUSTRALIA Hunter Publications PO Box 404 3067 Abbotsford, Victoria Tel (613)94 17 53 61 Fax (613) 94 19 71 54 Email | pria vie s β oz email com au CANADA Les éditions La Liberté Inc. Tel (1418)658 37 63 Fax (1800)567 54 49 Email hberloOmediom pc ca Renouf Publishing Co. Ltd 5369 Chemm Canotek Road Unit 1 KU 9J3 Ottawa. Ontano Tel (1613)745 26 65 Fax (1613) 745 76 60 Email: order. deptOrerKiufbooks. com URL: h rtp//www renouf book s com EGYPT The Middle East Observer 41 Sherif Streel Tel (202)392 6919 Fax (202) 393 97 32 Email: matoudaOmeobservar. com eg URL: http//www meob server com eg PSIJapan Asahi Sanbancfio Plaza »206 71 Sanbancho. Chryodaku Tokyo 102 Tel. (813)32 34 69 21 Fax (B13)32 34 69 15 Email: booksOpsijapan cojp URL: ht!p://www psi|apan com MALAYSIA EBIC Malaysia Level 7, Wisma Hong Leong 18 Jalan Perak 50450 Kuala Lumpur Tel (603)262 62 98 Fax (603$ 262 61 98 Email ebickiemol. net. my Mundi Prensa Mexico, SA do CV Río Panuco No 141 Colonia Cuauhlomoc MX06500 Mexico, DF Toi. (52-5)533 56 58 Fax (525) 514 67 99 Email: 101545 2361 Ocompuservo com PHILIPPINES EBIC Philippines 191h Floor, PS Bank Towor Son. Gil J. Puyat Ave. cor Tindalo St. Makali City Metro Manilla Tei (632) 759 66 80 Fax (632) 759 66 90 Email ocepcom β globe com ph URL http//www eccp com EBIC Sri Lank. ] Trans Asia Hotel 115 Sir chi It amp a I am A Gardiner Mawolfia Colombo 2 Tel. (941)074 71 50 78 Fax (941) 44 87 79 Email: ebicslOitmin com EBIC Thailand 29 Van issa Building, 8 th Floor Soi Chidlom Ploenchil 10330 Bangkok Tel (66-21655 06 27 Fax (662) 655 06 28 Email 0biCbkkOksc15 thcom URL: hnp/www. ebicbkk org UNITED STATES OF AMERICA Berna π Associates 4611 F Assembly Dnve Lanham MD20706 Tel (1800) 274 44 47 (loll free telephor Fax (1800) 865 34 50 (loti free tax) Email queryObeman com URL. http//www bernan com Bitte wenden Sie sich an ein Büro Ihrer Wahl/ Please contact the satas office ol your choice/ Veuillez vous adresser au bureau de vente de voire choix 2, rue Morc»or L2985 Luxembourg Tel (352)29 2942455 Fax (352) 29 2942758 Email inlomfoOopocececbe URL: http//eufop eu. inf 5, rue Rartfeisen L2411 Luxembourg Tel (352)4010 20 Fax [352)49 06 61 Emad marlOmdilu URL: hrtp://www mol. lu NEDERLAND SDU Servicecentrum Ulloevers Christoffei Pianti]nsiraat 2 Postbus 20014 2500 EA Den Haag Tel (3170)376 9880 Fax (3170) 378 97 83 Email eduOsdunl URL: http//Www sdu. nl Cyprus Chamber of Commerce and Industry PO Box 1455 CY1509 Nicosia Tel (357-2)66 95 00 Fax (3572) 66 10 44 Email de met rap O ceci org cy EESTI. INDIA Eesti Ka uba ndusToostu skoda (Estonian Chamber of Commerce and Industry) ToomKooli 17 EE0001 Tallinn Tel (372)646 02 44 Fax (372) 646 02 45 Email einfo 0 koda ee URL httpi/www. kodaee EBIC India 3rd Floor, Y. B Chavan Centre Gen J Bnosale Marg 400 021 Mumbai Tel (9122)282 60 64 Fax (9122) 285 45 64 Email ebicOgtasbmOI vsnl. net in URL. http / www ebicindia com The MEANS collection: "Evaluating socio-economic programmes" This six-volume set gives wider public access to the lessons of the MEANS programme which the European Commission introduced with a view to impro ving and promoting evaluation methods. Its publication meets a growing need for reliable benchmarks to deal with the problems of evaluation encoun tered when implementing public measures in general and Structural Fund ope rations in particular. In this context, the set constitutes a unique methodological guide that can solve the technical and organisational problems arising in the course of evaluation work and for which the existing literature provides little help. The collection is aimed not only at professional evaluators, for whom it will provide a valuable handbook, but also at non-specialists, in particular manag ing authorities and structural programme Monitoring Committees. Evaluation design and management Each volume is designed to be independent of the others, yet they are also complementary. Together, they deal with all aspects of the evaluation of socio economic programmes: I. II. Selection and use of indicators for monitoring and evaluation III. Principal evaluation techniques and tools IV. Technical solutions for evaluation within a partnership framework V. Transversal evaluation of impacts on the environment, employment and other intervention priorities VI. Glossary of 300 concepts and technical terms OFFICE FOR OFFICIAL PUBLICATIONS OF THE EUROPEAN COMMUNITIES _-2985 Luxembourg