Guidance and methods for indicator selection and specification

This document attempts to explain what is meant with the term indicator in the context of Intarese, defines what indicators are needed for and how indicators can be used in integrated risk assessments. Some of the graphs presented in this text can be found from an Analytica file.

KTL/MNP (E. Kunseler, M. Pohjola, J. Tuomisto, L. van Bree)

Introduction

At this project stage (18 months - May 2007), the integrated risk assessment methodology is about to become ready for application in policy assessment cases. Within subproject 3, case studies have been selected and development of protocols for case study implementation are in process. The issue frameworks have been formulated and full chain frameworks of the policy assessment case studies are being developed correspondingly. Indicator selection can be used as a bridge from the issue framing phase to actually carrying out the assessment. This guidance document provides information on selecting and specifying indicators and variables and on how to proceed from the issue framing to the full chain causal network description of the case.

There are several different interpretations of the term indicator and several different approaches to using indicators. This document is written in order to clarify the meaning and use of indicators as applicable in the context of integrated risk assessment. This guidance emphasizes causality in the full-chain approach and the applicability of the indicators in relation to the needs of each particular risk assessment case. In the subsequent sections, we clarify the term indicator in the context of integrated risk assessment and suggest an Intarese-approach to indicator selection and specification .

Integrated risk assessment

Before going any further in discussing indicators and their role in integrated risk assessment, it is necessary to consider some general features of integrated risk assessment. The following text in italics has been adapted from Scoping for policy assessments - guidance document by David Briggs :

Integrated risk assessment, as applied in the Intarese project, can be defined as the assessment of risks to human health from environmental stressors based on a 'whole system approach. It thus endeavours to take account of all the main factors, links, effects and impacts relating to a defined issue or problem, and is deliberately more inclusive (less reductionist) than most traditional risk assessment procedures. Key characteristics of integrated assessment are:

1. It is designed to assess complex policy-related issues and problems, in a more comprehensive and inclusive manner than that usually adopted by traditional risk assessment methods.
2. It takes a full-chain approach – i.e. it explicitly attempts to define and assess all the important links between source and impact, in order to allow the determinants and consequences of risk to be tracked in either direction through the system (from source to impact, or from impact back to source).
3. It takes account of the additive, interactive and synergistic effects within this chain and uses assessment methods that allow these to be represented in a consistent and coherent way (i.e. without double-counting or exclusion of significant effects).
4. It presents results of the assessment as a linked set of policy-relevant indicators.
5. It makes the best possible use of the available data and knowledge, whilst recognising the gaps and uncertainties that exist; it presents information on these uncertainties at all points in the chain.

Building on what was stated above, some further statements about the essence of integrated assessment can be given:

• Integrated environmental health risk assessment is a process that produces as its product a description of a certain piece of reality
• The descriptions are produced according to the (use) purposes of the product
• The risk assessment product is a description of all the relevant phenomena in relation to the chosen endpoints and their relations as a causal network
• The risk assessment product combines value judgements with the descriptions of physical phenomena
• The basic bulding block of the description is variable i.e. everything in is described as variables
• All variables in a causal network description must be causally linked to the endpoints of assessment

Indicators and proxies

The term indicator is a common concept that can be interpreted and used in several ways. Indicators are pieces of information serving the purpose of communicating the most essential aspects of a particular risk assessment to meet the needs of the uses of the assessment. Communicating here refers to conveying information about the particular phenomena to the target audience, but also to monitoring the statuses of the phenomena e.g. in evaluating effectiveness of actions taken to change rlated phenomena. In integrated assessment as applied in Intarese, indicator means variables of specific interest relative to the purpose of the assessment. For clarity, we recommend that the meaning of the word indicator is further narrowed down to mean only those variables of specific interest that are to be reported. An integrated risk assessment may thus contain also important variables that are of specific interest within the risk assessment process, but which may not be of great importace in relation to reporting to the target audience, and should therefore not be referred as indicators.

The term indicator is sometimes also used in the meaning of a proxy. Proxies are used as replacements for the actual objects of interest in a description if adequate information about the actual object of interest is not available. Proxies are indirect representations of the object of interest that ususally have some identified correlation with the actual object of interest. At least within the context of integrated risk assessment as applied in Intarese proxy and indicator have clearly different meanings and they should not be confused.

The figure 1 below clarifies the two terms:

• Proxy as a indirect replacement to the actual object
• Indicator as variable of specific interest to be reported to the target audience

Figure 1. Indicators and proxies

Different approaches to indicators

As mentioned above, there are several different interpretations of and ways to use indicators. The different approaches to indicators have been developed for situations with differing needs and thus may be representations of very different underlying principles. Some approaches to indicators are applicable in the context of integrated assessment, some are clearly inteded for different uses.

In the next section three different approaches tyo indicators are presented as examples. Following that a general classification of indicators that manages to cover at least all these examples is given. The general classification is then used to identify the types of indicators that are applicable in different cases of making integrated risk assessments. The three examples are explained in more detail in Appendix 1.

Examples of approaches

WHO

One of the most commonly known approaches to indicators is the by the World Health Organization (WHO). The WHO is based on the DPSEEA model and emphasizes that there are important phenomena in each of these steps of the model that can be represented as indicators.

The WHO approach treats indicators as relatively independent objects, i.e. e.g. an exposure indicator is defined as such from the point of view of exposure assessment and e.g. a health impact indicator is defined as such from the point of view of estimating health impacts. Linkages between different indicators is recognized based on their relative locations within the DPSEEA model, but causality between separate indicators is not explicitly emphasized.

In the WHO approach the indicator definitions are mainly intended as how to estimate this indicator? -type guidances or workplans. The WHO approach can be considered as an attempt to standardize or harmonize the assessments of different types of phenomena within the DPSEEA model.

EEA

The approach to indicators by the European Environmental Agency (EEA) is an attempt to clarify the use of indicators in different situations for different purposes. EEA has developed a typology of indicators that can be applied for selecting right types of indicator sets to meet the needs of the particular case at hand.

The EEA typology includes the following types of indicators:

• Descriptive indicators (Type A – What is happening to the environment and to humans?)
• Performance indicators (Type B – Does it matter?)
• Efficiency indicators (Type C – Are we improving?)
• Total welfare indicators (Type D – Are we on whole better off?)

The EEA indicator typology is built on the need to use environmental indicators for three major purposes in policy-making:

1. To supply information on environmental problems, in order to enable policy-makers to value their seriousness;
2. To support policy development and priority setting, by identifying key factors that cause pressure on the environment;
3. To monitor the effects of policy responses.

The EEA approach considers indicators primarily as means of communicating important information about the states of environmental and related phenomena.

RIVM

RIVM has also developed an indicator typology, which is also strongly present in the integrated risk assessment method development work in Intarese. The four indicator sets by RIVM are grouped somewhat similarly as in EEA typology to represent certain kinds of phenomena to give answers to different kinds of concerns. However, the RIVM indicator sets seem to be more defined in relation to the audience that the particular indicator set is intended to convey its messages to.

The RIVM indicator sets are:

• Policy-deficit indicators
• Health impact indicators
• Economic consequence indicators
• Risk perception indicators

In the RIVM approach the different sets of indicators address different phenomena from different perspectives in order to convey the messages about the states of affairs in right forms to the right audiences. By combining the different sets of indicators, a whole view of the assessed phenomena can be created for weighing and appraisal and to support decision making in targeting actions in relation to the phenomena. As in EEA approach, RIVM treats indicators as means for effective communication.

General classification of indicators

There are many ways to classify indicators. E.g. WHO, European Environmental Agency, RIVM, and others have developed their own ways to look at indicators.These three approaches were presented briefly above and there are several more approaches in addition to those examples. Here, we present a general classification that attempts to explain all different approaches in one framework.

Topic-based classification

The topic describes the scientific discipline to which the indicator content mostly belongs to. E.g. the RIVM classification mostly follows this thinking, but however the Policy deficit indicators in RIVM indicator sets do not belong here but on the reference-based classification below. The general topic-based classification in integrated risk assessment is:

• Health indicators.
• Economic indicators.
• Perception indicators. (Including equity and other ethical issues.)
• Ecological indicators. (Not fully covered in Intarese, as they are out of the scope of the project.)

Causality-based classification

Many currently existing and applied approaches to indicators treat independent pieces of information without a context of a causal chain (or full chain approach). WHO indicators are famous examples of this. This issue is further addressed within the next classification.

Reference-based classification

The reference is something that the indicator is compared to, and this comparison creates the actual essence of the indicator. The classification according to the reference is independent of the topic-based classification. EEA typology is an example of reference-based classification.

Type of indicator	Point of reference	Examples of use	Causality addressed?
Descriptive indicators	Not explicitly compared to anything.	EEA type A indicators Burden of disease WHO indicators are also often this type.	Possibly, most often not
Performance indicators	Some predefined policy target	EEA type B indicators Policy deficit indicators in RIVM indicator sets	Possibly
Efficiency indicators	Compared with the activity or service that causes the impact.	EEA type C indicators Cost-effectiveness analysis	Yes
Total outcome indicators	?	EEA type D indicators Green gross domestic product Index of Sustainable Economic Welfare (ISEW).	Possibly
Scenario indicatorsD↷	Some predefined policy action, usually a policy scenario compared with business as usual.	Cost-benefit analysis.	Yes

Intarese approach to indicators

In integrated risk assessment as applied in Intarese, the full chain approach is an integral part of the method, and therefore all Intarese indicators should reflect causal connections to relevant variables. According to the full-chain approach, all variables within an integrated risk assessment, and thus also indicators, must be in a causal relation to the endpoints of the assessment. Independently defined indicators, such as WHO indicators and descriptive indicators in general, are thus not applicable as Intarese indicators. All other types of indicators can be used as applicable, as long as causality is addressed.

In general, the communicative needs define the suitable types of indicators for each assessment. Therefore, the right set(s) of indicators can vary significantly. The selection of indicators can follow the lines of either topic-based classification of reference-based classification as seen reasonable. Anyhow, the set(s) of indicators should represent, at least in some way:

1. The use purpose of the assessment
2. The target audience of communication
3. The importance of the indicators in relation to the assessment and its use

Indicators can be helpful in constructing carrying out the integrated risk assessment case. They can be used for targeting efforts to the most relevant aspects of the assessments especially from the point of view of addressing the purpose and targeted users or audience of the assessments. Indicators can also be used as the backbone of the assessment when creating the causal network description of the assessed phenomena.

Using indicators in integrated risk assessment

As a starting point, let us take a look at another slightly adapted excerpt taken from the Scoping for policy assessments - guidance document by David Briggs:

The Intarese approach to risk assessment emphasizes on creation of causal linkages between the determinants and consequences in the integrated assessment process. The full-chain approach includes interconnected variables and indicators which are the leading components. The full chain variables cover the source-impact chain, which is based on different frameworks developed from the pressure-state-response (PSR) concept originally proposed by the US-EPA (e.g. DPSIR, DPSEEA) and the source-receptor models widely used to represent the fate of pollutants in the environment.

Again, building on the above, it can be said that the output of an integrated risk assessment should be a causal network description of the relevant phenomena related to the endpoints of the assessment, in accordance with the purpose of the assessment. The final description should thus:

• Address all the relevant issues as variables
• Describe the causal relations between the variables
• Explain how the variable result estimates are come up with
• Report the variables of greatest interest as indicators

In issue framing especially the inclusion and exclusion of phenomena according to their relevance in relation to the endpoints of the assessment and the purpose of the assessment is addressed. Also some understanding of what are the variables of most interest usually exists already in the phase of issue framing. Anyhow, there is still quite a long, and not necessarily at all a straightforward, way from issue framing to a relevantly complete causal network description of the assessed phenomena. The biggest challenges on this way are:

• How are variables defined and described?
• How can the causal relations between variables be defined and described?
• What is the right level of detail in describing variables?

These questions are addressed in following sections. First a general description of the process is given, then some more detailed guidance and description of indicator selection and specification is presented, and eventually a general variable and indicator structure is suggested. The text is supported with an example of an indicator description which can be found in Appendix 2.

From issue framing to a causal network description

After the issue framing, there should be a good general level understanding of what should be included in the assessment and what should be excluded. Also, most often there is a relatively good understanding of what are the most important variables within the assessment scope in relation to the endpoints that are pursued, and what are the most interesting variables within the assessment scope from the point of view of the users and other audience of the assessment. Let us call the most important variables key variables and the most interesting variables indicators. These two variable sets must include all the endpoints of the assessment. Often it may be so that these two sets completely overlap, but this may not always be the case, and therefore it is reasonable to consider these sets simultaneously but separately. These variable sets provide a good starting point for proceeding in the assessment after issue framing is done.

When identified, the key variables and indicators can be located on the source-impact chain of full-chain approach. This forms the backbone of the causal network description. The description now includes (overlappingly):

• All the endpoints of the assessment
• The key variables that are important for carrying out the assessment
• The indicators that are crucial for communicating results

Because all of the variables must be causally interconnected, the linkages between the causal relations between the key variables and indicators need to defined. Additional variables are added to the description as needed to create the causal relations across the network that interconnect all the key variables and indicators from the first input variables to the endpoint variables. This happens as an iterative process of (re-)selecting and specifying the key variables, indicators and other variables. Selection and specification of indicators and other variables are explained in more detail in the following sections.

The process typically proceeds from more general level, e.g. from general full-chain description, to a more detailed level as applicable for the particular assessment. Throughout the full chain, causality can be improved as seen necessary by e.g. combining detailed variables into more general variables, dividing general level variables into more detailed ones, adding necessary variables to the chain, removing variables that turn out irrelevant, changing the causal links etc. As an example, the general air pollutant variable can be divided into specific pollutant variables, e.g. for NOx, PM2.5 /PM10, BS etc and also further specified in terms of limitations in relation to e.g. time and space as needed in the particular assessment. Each of these specific variables naturally then have different causal relations to the consequent health effect variables. In practice, the level of detail might need to be iteratively adjusted also to meet e.g. data availability and existing understanding of causal relations between variables. Within Intarese there are plans to create a scoping tool as a part of the toolbox to help the creation of the causal network description, but unfortunately this tool is non-existent as yet.

When iteratively developing the causal network description, there are two critical tests that can be used in determining the causal relations between variables and the right scope for individual variables:

• clairvoyant test for variables
• causality test for variable relations

All variables in a proper full-chain description should pass both of these tests.

Variables should describe some real-world entities, preferably described in a way that they pass a so called clairvoyant test. The clairvoyant test determines the clarity of a variable. When a question is stated in such a precise way that a putative clairvoyant can give an exact and unambiguous answer, the question is said to pass the test. So in case of a variable, the scope of the variable should be defined so that a clairvoyant, if one existed, could give an exact and unambiguous answer to what is its result.

All variables should also be related to each causally. Causality test determines the nature of the relation between two variables. If you alter the value of a particular variable (all else being equal), the variables downstream (i.e., the variables that are expected to be causally affected by the tested variable) should change. If no change is observed, the causal link does not exist. If the change appears unreasonable, the causal link may be different than originally stated.

File:Causal links defined with variables.PNG

Figure 2: Indicators and their causal relations are specified simultaneously - Circles represent variables; Squares represent indicators.

Structure of variables/indicators

• why unified structure?
  • efficiency
  • physical reality objects & value judgements can both be described as variables
  • control of hierarchical information structures (→collective sructured learning)
• attributes of variables
  • attributes explained
  • reference to example in appendix (to be created)

1. Name
2. Scope
3. Description
   • Scale
   • Averaging period
   • References
4. Unit
5. Definition
   • Causality
   • Data
   • Formula
     • Variations and alternatives
6. Result
7. Discussion

Using indicators in communicating

que?

Appendices

Appendix 1. Different approaches to indicators

WHO indicators

WHO is developing an Environment and Health Information System (EHIS). EHIS is regarded as a valuable tool for monitoring and evaluating the implementation and modification of policies by providing systematically collected and analysed evidence. The objective is to develop a harmonized and evidence-based information system that will serve policy-makers at European, national and local levels and be accessible by the general public as well. Crucial to developing a pan-European EHIS is a set of policy-relevant indicators to measure the situation and changes over time. For this purpose, indicators must monitor the linkages between environmental changes and human health effects and be based on scientific evidence. The DPSEEA (driving forces - pressures - state - exposure - effects - action) model was adopted to specify the policy-relevant indicators along the source - impact chain. (WHO EH indicators for Europe - A pilot indicator-based report, 2004)

File:DPSEEA approach.jpg

In terms of policy relevance, exposure-side indicators and health-side indicators are of highest interest. These types cover the forward looking indicators of exposure (i.e. those that presage, and need to be linked to, a potential health effect) and the backward looking indicators of outcome or effect (i.e. those that imply, and need to be attributed to, an exposure or source). Exposure-side indicators are clearly relevant for policy, since they often provide the first indications of the potential for health risk, and the first evidence of the effects of intervention (since many policies are focused on the upper links in the source-impact chain). To be meaningful in the context of health risks, however, they must relate to factors with definable (or at least strongly plausible) links to health outcome. (Briggs D, 16.5.06) Dose-Response indicators are necessary for clarifying the exposure to health linkage. Moreover, the exposure-side indicators should linked back to its emissions and sources. Exposures can only be reduced when its sources or emission activities are known, therefore source or emission indicators should be introduced as a third type of policy-relevant indicators. Health-side (or impact) indicators represent the consequences of exposures in terms of health effect (e.g. mortality, morbidity, DALYs) or its further societal impacts (e.g. economic costs, quality of life). Again, to be meaningful in the context of the full-chain approach, they need to have an explicit link back to causal environmental exposures and risk factors. (Briggs D, 16.5.06)

A fifth type of indicator is the action or policy indicator. WHO developed this outcome indicator to assess the policy situation with regard to policy existence, implementation and enforcement. Qualitative information is classified in quantitative numbers in order to make country comparisons possible. The importance of these outcome indicators lies in their ability to express priorities for policy action. (WHO, ENHIS project)

EEA indicator typology

[The text in this section has been taken from the EEA Technical report No 25, Environmental indicators: Typology and overview, EEA, Copenhagen,1999)].

A wide variety of environmental indicators is presently in use. These indicators reflect trends in the state of the environment and monitor the progress made in realising environmental policy targets. As such, environmental indicators have become indispensable to policy-makers. However, it is becoming more and more difficult for policy-makers to grab the relevance and meaning of the existing environmental indicators, given the number and diversity of indicators presently in use. And new sets of environmental indicators are still to be expected. Therefore, some means of structuring and analysing indicators and related environment/society inter-connections is needed.

For the purpose of this INTARESE indicator paper the European Environment Agency (EEA) indicator typology and the DPSIR framework (Driving forces, Pressure, State, Impact,Response) is used.

In relation to policy-making, environmental indicators are used for three major purposes:

1. to supply information on environmental problems, in order to enable policy-makers to value their seriousness;

2. to support policy development and priority setting, by identifying key factors that cause pressure on the environment;

3. to monitor the effects of policy responses.

In addition, environmental indicators may be used as a powerful tool to raise public awareness on environmental issues. Providing information on driving forces, impacts and policy responses, is a common strategy to strengthen public support for policy measures.

EEA Typology of Indicators

Indicators can be classified into 4 simple groups which address the following questions: • ‘What is happening to the environment and to humans?’ (Type A or Descriptive Indicators) • ‘Does it matter?’ (Type B or Performance indicators) • ‘Are we improving?’ (Type C or Efficiency indicators) • ‘Are we on the whole better off?’ (Type D or Total Welfare indicators)

Descriptive indicators (Type A – What is happening to the environment and to humans?)

Most sets of indicators presently used by nations and international bodies are based on the DPSIR-framework or a subset of it. These sets describe the actual situation with regard to the main environmental issues, such as climate change, acidification, toxic contamination and wastes in relation to the geographical levels at which these issues manifest themselves. With respect to environmental health, these indicators may also be specified with respect to (personal) (source-specific) exposure indicators and health effects indicators (## of people affected, YLL, DALY, or QUALY).

Performance indicators (Type B – Does it matter?)

The indicators mentioned above all reflect the situation as it is, without reference to how the situation should be. In contrast, performance indicators compare (f)actual conditions with a specific set of reference conditions. They measure the ‘distance(s)’ between the current environmental situation and the desired situation (target): ‘distance to target’ assessment. Performance indicators are relevant if specific groups or institutions may be held accountable for changes in environmental pressures or states.

Most countries and international bodies currently develop performance indicators for monitoring their progress towards environmental targets. These performance indicators may refer to different kind of reference conditions/values, such as:

• national policy targets; • international policy targets, accepted by governments; • tentative approximations of sustainability levels.

The first and second type of reference conditions, the national policy targets and the internationally agreed targets, rarely reflect sustainability considerations as they are often compromises reached through (international) negotiation and subject to periodic review and modification. Up to now, only very limited experience exists with so-called sustainability indicators that relate to target levels of environmental quality set from the perspective of sustainable development (Sustainable Reference Values, or SRVs).

Performance indicators monitor the effect of policy measures. They indicate whether or not targets will be met, and communicate the need for additional measures.

Efficiency indicators (Type C – Are we improving?)

It is important to note that some indicators express the relation between separate elements of the causal chain. Most relevant for policy-making are the indicators that relate environmental pressures to human activities. These indicators provide insight in the efficiency of products and processes. Efficiency in terms of the resources used, the emissions and waste generated per unit of desired output.

The environmental efficiency of a nation may be described in terms of the level of emissions and waste generated per unit of GDP. The energy efficiency of cars may be described as the volume of fuel used per person per mile travelled. Apart from efficiency indicators dealing with one variable only, also aggregated efficiency indicators have been constructed. The best-known aggregated efficiency indicator is the MIPS-indicator (not covered in this report). It is used to express the Material Intensity Per Service unit and is very useful to compare the efficiency of the various ways of performing a similar function.

Efficiency indicators present information that is important both from the environmental and the economic point of view. ‘Do more with less’ is not only a slogan of environmentalists. It is also a challenge to governments, industries and researchers to develop technologies that radically reduce the level of environmental and economic resources needed for performing societal functions. Since the world population is expected to grow substantially during the next decades, raising environmental efficiency may be the only option for preventing depletion of natural resources and controlling the level of pollution.

The relevance of these and other efficiency indicators is that they reflect whether or not society is improving the quality of its products and processes in terms of resources, emissions and waste per unit output.

Total welfare indicators (Type D – Are we on whole better off?)

Some measure of total sustainability is needed in order to answer this question, for example, a kind of ‘Green GDP’, such as the Index of Sustainable Economic Welfare (ISEW). As these indicators are, however, currently outside of the EEA’s work programme, there are not further covered here.

MNP example on policy deficit indicators

To illustrate policy deficit indicators used in various environmental themes, an example has been taken from the (annual) Environmental Balance report (2006) of the Netherlands Environmental Assessment Agency visualizing a simple table format, using different colours to show what the time trends and target achievements are.

Figure 4 MNP example of policy deficit indicators

RIVM: 4 indicator sets

File:Assessment and appraisal process.PNG

WHO and EEA selection criteria

When selecting indicators in the source - impact chain in WHO and EEA projects, the policy context or commonly recognised issues are the main drive for indicator selection. WHO has for example developed children's environmental health indicators which measure the implementation of CEHAPE priority goals. (WHO, ENHIS project) Subsidiarity is important as well; information need to be collected at the most relevant level or for specific policy/management purposes. Detailed indicators for local level or specific purposes might feed into broader (core) indicators that can be used at higher policy level or for general public information. Moreover, the indicators need to be associated with a suite of methods to derive them and with methods and approaches to link the indicators across the causal chain. Also incorporation of available information from monitoring and surveillance systems on environmental stressors and health provide selection criteria for indicator development. (WHO, 2002 and Lebret E & Knol A, 2007)

Besides these principal criteria for indicator selection, which can be summarized as (i) relevance to users and acceptability, (ii) consistency, (iii) measurability there are several other issues to be taken into consideration. Indicators must be based on known and validated processes or principles; scientific credibility. Sensitivity and robustness are a precondition for indicators, since a change must be responded to while slight variations should be coped with. Moreover, the indicator must be understandable and user-friendly. (Briggs D, 2006)

WHO has selected as set of environmental health indicators based on these criteria and expert judgements, see http://www.euro.who.int/EHindicators. A protocol for pilot testing the indicators was formulated to come to a further selection of core and extended sets of indicators. Proposed indicators were screened by a group of experts in terms of their credibility, basic information on the definition, calculation method, interpretation and potential data sources. During the screening process a template for a methodology sheet was designed entailing the attributes that are summarized in the Appendix. During development of the methodology sheets, further consultation with national and international experts and international agencies as well as national ministries and agnecies and holders of environmental and health data was conducted. During development of the methodology sheets it became apparent that for some indicators insufficient data was available to continue development. Indicators were defined in the core set once their relevance for policy and availability of the data was confirmed. Indicators which were deemed policy-relevant but for which data is currently not available were included in the extended set of indicators for future development and use. In succession to the second selection round, the methodology sheets were further refined. Three major tasks were 1 - Development of a specific technical definition; 2 - Elaboration of a computation method for each indicator; 3 - Check-up of data availability in international sources. The process of development and adjustment of the methodology sheets served as a pre-screening process to determine the need for testing the indicators. Only core indicators were further considered, while adopted indicators and indicators with readily available data from international databases did not require feasibility testing. Other core indicators underwent a screening process to test the feasibility and applicability of the indicators. (WHO ENHIS final technical report, 2005)

Alternative selection criteria

[insert figure]

The first steps in indicator selection are similar to the WHO and EEA indicator selection process. The purpose and scope of the indicators are defined. Within the group of experts and potential users, the kinds of measures that provide information that fits within the defined scope should be discussed. The list of potential indicators should be scrutinized by means of defined selection criteria. Agreement on criteria definitions that should fit the specific needs of the project is essential in order to determine the relative importance of the criteria to meet the intended purpose.

[insert figure]

High weights can be assigned to important criteria and lower weights to those that are less critical. For example, the criterion 'understandable' should be high for a public report, medium for a management report and may be low for reports being used by analysts.

Appendix 2. Example of Intarese indicator selection and specification

File:Congestioncharge.png

• Variable specification in issue framework

Name: Annoyance

Scope: Number of people in Helsinki municipal area being annoyed to some degree by being exposed to traffic noise of 35 dB(A) and higher

Causality: List of parents: Exposure to noise levels equal or higher than 35 dB(A) List of daughters: Cardiovascular ilness; Psycho-social wellbeing

Appendix 3. Comparison of different variable/indicator structures

Table. A comparison of attributes used in Intarese (suggestions), ENHIS indicators, pyrkilo method, and David's earlier version.

Suggested Intarese attributes	WHO indicator attributes	Pyrkilo variable attributes	David's variable attributes
Name	Name	Name	Name
Scope	Issue	Scope	Detailed definition
Description	Definition and description	Description (part of)	-
Description (part of)	Interpretation	Description (part of)	-
Description / Scale	Scale	Scope or Description	Geographical scale
Description / Averaging period	-	Scope or description	Averaging period
Description / Variations and alternatives	-	Description	Variations and alternatives
Description (part of)	Linkage to other indicators	Description (part of)	- R↻
Unit	Units	Unit	Units of measurement
Definition / Causality	Not relevant	Definition / Causality	Links to other variables
Definition / Data R↻	Data sources or Related data	Definition / Data	Data sources, availability and quality
Definition / Formula	Computation	Definition / Formula	Computation algorithm/model
Result (a very first draft of it) R↻	Not a specific attribute	Result (a very first draft of it)	Worked example
Discussion	-	-	-
Done by using categories	-	Done by using categories	Type
Done by links to glossary	-	Done by links to glossary	Terms and concepts
Done by argumentation on the Discussion area	Specification of data needed	Done by argumentation on the Discussion page	Data needs
The postition in a causal diagram justifies the existense	Justification	The postition in a causal diagram justifies the existense	-
Not relevant	Policy context	Not relevant	Not relevant
Not relevant	Reporting obligations	Not relevant	Not relevant