Guidance and methods for indicator selection and specification

This is a guidance document for selecting and specifying indicators as a part of applying the Intarese method.

KTL/MNP (E. Kunseler, L. van Bree, M. van der Hoek, M. Pohjola, J. Tuomisto)

Introduction

Integrated risk assessment, as applied in the Intarese project, can be defined as the assessment of risks to human health from environmental stressors based on a ‘whole system’ approach. It thus endeavours to take account of all the main factors, links, effects and impacts relating to a defined issue or problem, and is deliberately more inclusive (less reductionist) than most traditional risk assessment procedures. (Briggs D, 16.05.06))

Key characteristics of integrated assessment are:

1. It is designed to assess complex policy-related issues and problems, in a more comprehensive and inclusive manner than that usually adopted by traditional risk assessment methods.
2. It takes a ‘full-chain’ approach – i.e. it explicitly attempts to define and assess all the important links between source and impact, in order to allow the determinants and consequences of risk to be tracked in either direction through the system (from source to impact, or from impact back to source).
3. It takes account of the additive, interactive and synergistic effects within this chain and uses assessment methods that allow these to be represented in a consistent and coherent way (i.e. without double-counting or exclusion of significant effects).
4. It presents results of the assessment as a linked set of policy-relevant ‘outcome indicators’.
5. It makes the best possible use of the available data and knowledge, whilst recognising the gaps and uncertainties that exist; it presents information on these uncertainties at all points in the chain. (Briggs D, 16.05.06)

The Intarese approach to risk assessment emphasizes on creation of causal linkages between the determinants and consequences in the integrated assessment process. The full chain approach includes interconnected variables which are the leading components. The full chain variables cover the source-impact chain, which is based on different frameworks developed from the pressure-state-response (PSR) concept originally proposed by the US-EPA (e.g. DPSIR, DPSEEA) and the source-receptor models widely used to represent the fate of pollutants in the environment. (Briggs D,16.05.06)

Need for guidance

At this project stage (18 months - May 2007), the assessment methodology is ready for application in policy assessment cases. Case studies have been selected and protocols for case study implementation are in process. The issue frameworks have been formulated and consequent full chain frameworks of the policy assessment case studies are being developed. This guidance document gives detailed information about development of the variables and indicators in the full chain framework. The guidance emphasizes on causality in the full-chain approach and the applicability of the indicators in relation to policy needs. An application of indicator selection and specification is available for the Transport case study on congestion charging.

What are indicators?

Indicators can be defined as variables of specific interest. The specific interests are identified by the assessment purpose and the users of the assessment. Indicators should ideally represent all the main nodes and links that make up the source-impact chain, and should be internally coherent – i.e. they should have clear and definable relationships within the context of this chain. The idea behind the indicator selection, specification and use is to highlight the most important and/or significant parts of the source-impact chain which has been/is to be assessed. Indicator selection provides the bridge between the issue framework and the assessment process. Indicators are also helpful in communicating about the assessment and in using the assessment product as well as in enhancing coherence between different assessments.

Policy-relevant indicators

Indicators can take very different forms. In terms of policy relevance, exposure-side indicators and health-side indicators are of highest interest. These types cover the forward looking indicators of exposure (i.e. those that presage, and need to be linked to, a potential health effect) and the backward looking indicators of outcome or effect (i.e. those that imply, and need to be attributed to, an exposure or source). Exposure-side indicators are clearly relevant for policy, since they often provide the first indications of the potential for health risk, and the first evidence of the effects of intervention (since many policies are focused on the upper links in the source-impact chain). To be meaningful in the context of health risks, however, they must relate to factors with definable (or at least strongly plausible) links to health outcome. (Briggs D, 16.5.06) Dose-Response indicators are necessary for clarifying the exposure to health linkage. Moreover, the importance of the backward looking part of exposure-side indicators must not be overlooked. Exposures can only be reduced when its sources or emission activities are known and being restricted, therefore source or emission indicators should be introduced as a third type of policy-relevant indicator. Health-side (or outcome) indicators represent the consequences of exposures in terms of health effect (e.g. mortality, morbidity, DALYs) or its further societal impacts (e.g. economic costs, quality of life). Again, to be meaningful in the context of the full-chain approach, they need to have an explicit link back to causal environmental exposures and risk factors. (Briggs D, 16.5.06)

A fifth type of indicator concern the action or policy incicator. Based on the DPSEEA framework, WHO developed these policy-relevant outcome indicators to assess the state-of-the-art in policy existence, implementation and enforcement. Qualitative information is classified in quantitative numbers in order to make country comparisons possible. The importance of these outcome indicators lies in their ability to express priorities for policy action. (WHO, ENHIS project)

In each type, the indicators may be expressed in different ways, depending on:

• Whether they are static (state, condition) or dynamic (process, flux) indicators;
• Whether they are expressed in quantitative (‘objective’) or qualitative (perception) measures;
• Whether or not they relate to a formal (and internal) reference level or target (performance indicators).(Briggs D, 16.5.06)

Indicators in the full-chain approach

Two approaches for selecting and specifying indicators in the full-chain approach can be identified and distinguished. The distinction is based on the point in development of the assessment framework where and how strict causal linkages between variables are defined. Once the causality between stages in the assessment framework have been defined, the main nodes and links stand out. Key variables can be selected as indicators and further specified.

Below, both approaches are described.

File:Causal links defined with variables.PNG

Figure 1: Variables and their causal relations are specified simultaneously

The first approach described in figure 1 starts from the issue framing by translating it into a set of variables (circles) and their causal relations (connecting arrows), representing the outline of the system to be assessed. The result of this is a (draft) causal chain description of the phenomena to be assessed, on a relatively high level of abstraction, representing the determinants across the chain as variables. At this point the variables do not necessarily need to be specified into any more detail than to have a name and to have some draft specification of its scope. Also the causal relations at this point need not be any more detailed than to just imply that a causal relation between certain variables exists.

The causal chain description can naturally be further improved as seen necessary by combining (too detailed level) variables into more general variables, dividing (too general level) variables into more detailed ones, adding needed variables to the chain, removing variables that turn out irrelevant, changing the causal links etc. For example, the air pollutant variable can be divided into specific pollutant variables, e.g. for NOx, PM2.5 /PM10, BS etc. Each of these pollutants has a different relation to the consequent health effect / impact variables.

After the causal chain has been described, the indicators, the variables of specific interest, can be chosen among the set of variables. (If necessary, new indicator variables can be added to the description. This may be the case for example if a desired indicator is e.g. a ratio of two variable's result values, but this indicator is not relevant in estimating the outcome variables downstream in the causal chain.)

At the same time as indicator selection takes place, also the variables used in the assessment process should be further defined and described. This helps in ensuring that all the terms used in the assessment are consistent and explicit. Descriptions should cover the scope of each variable, methods/models used to compute or derive the variable, and the data (and associated data sources) on which these are based. Variables in this context may take different forms and serve different roles (often simultaneously). They may represent inputs to models (derived variables), interim steps in the calculation process (derived variables) and single or combined output values (indicators).

The first approach described above could be referred to as developing indicators as a continuous process. The variable specifications including their result value estimates and causal relation descriptions are iteratively improved throughout the course of the assessment process as the knowledge and understanding increases. The causal relation descriptions of variables influence the estimation of the output value with the aid of data, measurements and models and vice versa. Accordingly, also indicator selection is an iterative process. In order to make this process explicit, it is likely to be helpful to produce shadow versions of the issue and full chain frameworks, showing the indicators to be used.

File:Causal links defined after variables.PNG

Figure 2: Variables are defined as individual objects and subsequently linked

The second approach desribed in Figure 2 starts from the issue framing by choosing the indicators first. The main emphasis is on identifying and defining of a set of variables of specific interest, whilst the causal linkages between them are established and defined at a later stage. Naturally, already in the issue framing, the relationship between the variables are considered, since they are in some way positioned in relation to each other. Applying this approach creates indicators that are individual components in the overall assessment. It also allows computation methods and models as well as data to be separately defined for each variable. Although maybe being a more comfortable approach, this also leaves more possibilities for inconsistencies between the indicators and may risk the transparency and reliability of the outcome variable/indicator specifications and estimates.

This second approach could be referred to as developing indicators as individual objects. This approach is applied e.g. by WHO in their indicator development. The indicator specifications are intended more as an instruction how to come up with the result value estimate. Indicator result value estimates are then compiled across the full-chain description to produce the result value estimates for the outcome indicators.

Selecting indicators

As mentioned above, the variables of specific interest can be chosen in relation to the purpose of the assessment and the needs of the intended users. In principle, any variable could be chosen as an indicator and the set of indicators could be composed of any types but should cover the steps in the full-chain description. In practice, the generally relevant types of indicators can be somewhat predefined and even some detailed indicators can be defined in relation to commonly existing purposes and user needs. This kind of generality is also helpful in bringing coherence between the assessments and their uses.

Environmental health indicators may be designed for different users and for different purposes, including:

• Policy development or priority-setting
• Health impact assessment and monitoring
• Policy implementation or economic consequence assessment
• Public information and awareness rising (WHO, 2002)

When selecting the indicators in the full chain framework, the application should be a leading consideration and result in mutually consistent indicators (over time, space and target group). WHO has for example developed children's environmental health indicators which fit in the context of the CEHAPE priority goals. (WHO, ENHIS project) Subsidiarity is important as well; information need to be collected at the most relevant level or for specific policy/management purposes. Detailed indicators for local level or specific purposes might feed into broader (core) indicators that can be used at higher policy level or for general public information. Moreover, the indicators need to be associated with a suite of methods to derive them and with methods and approaches to link the indicators across the causal chain. Also incorporation of available information from monitoring and surveillance systems on environmental stressors and health provide selection criteria for indicator development. (WHO, 2002 and Lebret E & Knol A, 2007)

Besides these principal criteria for indicator selection, which can be summarized as (i) relevance to users and acceptability, (ii) consistency, (iii) measurability there are several other issues to be taken into consideration. Indicators must be based on known and validated processes or principles; scientific credibility. Sensitivity and robustness are a precondition for indicators, since a change must be responded to while slight variations should be coped with. Moreover, the indicator must be understandable and user-friendly. (Briggs D, 2006)

At European scale, WHO is developing environmental health indicators. Their methodology and results is used as reference for indicator specification in Intarese, see next section. See http://www.euro.who.int/EHindicators. Other organisations are developing either environmental indicators, such as EEA (see: http://themes.eea.europa.eu/indicators/) or health indicators e.g. the European Community Health Indicators (ECHI).

Specifying indicators

We suggest a method for indicator development, including characteristics from both approaches towards indicator development as explained in the previous section.

We suggest that all variables, and thus also all indicators, are specified using a fixed set of attributes. The reasoning behind is to secure coherence between variable/indicator specifications and to enhance efficiency of assessment work and re-usability of the outputs of assessment work.

Below is a suggested list of variable/indicator attributes. The list has been developed based on the several principles including, but not limited to, the following:

• Variables are the basic building blocks of risk assessments
• Everything in risk assessments is to be described as variables
• Risk assessments are causal-chain descriptions of (a chosen part) of reality
• All variables in a causal-chain description must be causally linked
  • Also the causal links are described within the variable specifications (definition:causality)
    • In a diagram representation arrows only state the existence of a causal relation, it does not specify the causality
• The risk assessment process proceeds iteratively through specifications and re-specifications of variables (and their causal relations)

Suggested Intarese variable/indicator attributes

1. Name
2. Scope
3. Description
   • Scale
   • Averaging period
   • References
4. Unit
5. Definition
   • Causality
   • Data
   • Formula
     • Variations and alternatives
6. Result
7. Discussion

Example: Selection and specification of indicators in Transport congestion charge case study

File:Congestioncharge.png

Appendix: comparison of different approaches to specifying variables

Table. A comparison of attributes used in Intarese (suggestions), ENHIS indicators, pyrkilo method, and David's earlier version.

Suggested Intarese attributes	WHO indicator attributes	Pyrkilo variable attributes	David's variable attributes
Name	Name	Name	Name
Scope	Issue	Scope	Detailed definition
Description	Definition and description	Description (part of)	-
Description (part of)	Interpretation	Description (part of)	-
Description / Scale	Scale	Scope or Description	Geographical scale
Description / Averaging period	-	Scope or description	Averaging period
Description / Variations and alternatives	-	Description	Variations and alternatives
Description (part of)	Linkage to other indicators	Description (part of)	- R↻
Unit	Units	Unit	Units of measurement
Definition / Causality	Not relevant	Definition / Causality	Links to other variables
Definition / Data R↻	Data sources or Related data	Definition / Data	Data sources, availability and quality
Definition / Formula	Computation	Definition / Formula	Computation algorithm/model
Result (a very first draft of it) R↻	Not a specific attribute	Result (a very first draft of it)	Worked example
Discussion	-	-	-
Done by using categories	-	Done by using categories	Type
Done by links to glossary	-	Done by links to glossary	Terms and concepts
Done by argumentation on the Discussion area	Specification of data needed	Done by argumentation on the Discussion page	Data needs
The postition in a causal diagram justifies the existense	Justification	The postition in a causal diagram justifies the existense	-
Not relevant	Policy context	Not relevant	Not relevant
Not relevant	Reporting obligations	Not relevant	Not relevant