Guidance and methods for indicator selection and specification: Difference between revisions

From Opasnet
Jump to navigation Jump to search
mNo edit summary
(edited and variable tool table moved from the Intarese tools page)
Line 1: Line 1:
[[Category:Needs editing]]
Guidance and methods for indicator selection and specification
 
Guidance and methods
for indicator selection and specification
 
First draft
5 April 2007
 
 
INTARESE
Integrated Risk Assessment of Health Risks from Environmental Stressors in Europe
 
Workpackage 1.4 Risk Characterization
(WP1.4)
 
 
 
 
 


First draft 5 April 2007


Mikko Pohjola, Eva Kunseler, Jouni Tuomisto
Mikko Pohjola, Eva Kunseler, Jouni Tuomisto
KTL, Finland
KTL, Finland


Leendert van Bree
Leendert van Bree
MNP, The Netherlands
MNP, The Netherlands


Workpackage 1.4 Risk Characterization
Outline of the report
 
 
1. The INTARESE context: full chain development
• Causality
 
2. The terminology: Indicator
 
3. Indicator selection from different perspectives
• As a process
• As individual components
 
4. A proposal for indicator attributes


5. INTARESE Guidance for indicator selection and specification
[[Category:Needs editing]] [[Category:Intarese general method]]
   
   
1. The INTARESE context: Full chain development
==The INTARESE context: Full chain development==


Integrated risk assessment, as applied in the INTARESE project, can be defined as the assessment of risks to human health from environmental stressors based on a ‘whole system’ approach.  It thus endeavours to take account of all the main factors, links, effects and impacts relating to a defined issue or problem, and is deliberately more inclusive (less reductionist) than most traditional risk assessment procedures. (D. Briggs, 16.5.06)
Integrated risk assessment, as applied in the INTARESE project, can be defined as the assessment of risks to human health from environmental stressors based on a ‘whole system’ approach.  It thus endeavours to take account of all the main factors, links, effects and impacts relating to a defined issue or problem, and is deliberately more inclusive (less reductionist) than most traditional risk assessment procedures. (D. Briggs, 16.5.06)


Key characteristics of integrated assessment are:
Key characteristics of integrated assessment are:
1. It is designed to assess complex policy-related issues and problems, in a more comprehensive and inclusive manner than that usually adopted by traditional risk assessment methods.
#It is designed to assess complex policy-related issues and problems, in a more comprehensive and inclusive manner than that usually adopted by traditional risk assessment methods.
2. It takes a ‘full-chain’ approach – i.e. it explicitly attempts to define and assess all the important links between source and impact, in order to allow the determinants and consequences of risk to be tracked in either direction through the system (from source to impact, or from impact back to source).   
#It takes a ‘full-chain’ approach – i.e. it explicitly attempts to define and assess all the important links between source and impact, in order to allow the determinants and consequences of risk to be tracked in either direction through the system (from source to impact, or from impact back to source).   
3. It takes account of the additive, interactive and synergistic effects within this chain and uses assessment methods that allow these to be represented in a consistent and coherent way (i.e. without double-counting or exclusion of significant effects).
#It takes account of the additive, interactive and synergistic effects within this chain and uses assessment methods that allow these to be represented in a consistent and coherent way (i.e. without double-counting or exclusion of significant effects).
4. It presents results of the assessment as a linked set of policy-relevant ‘outcome indicators’.
#It presents results of the assessment as a linked set of policy-relevant ‘outcome indicators’.
5. It makes the best possible use of the available data and knowledge, whilst recognising the gaps and uncertainties that exist; it presents information on these uncertainties at all points in the chain. (D. Briggs, 16.5.06)
#It makes the best possible use of the available data and knowledge, whilst recognising the gaps and uncertainties that exist; it presents information on these uncertainties at all points in the chain. (D. Briggs, 16.5.06)
The INTARESE approach emphasizes on creation of causal linkages between the determinants and consequences in the integrated assessment process. For this purpose, a full chain diagram should be constructed in which variables are the leading components.  
The INTARESE approach emphasizes on creation of causal linkages between the determinants and consequences in the integrated assessment process. For this purpose, a full chain diagram should be constructed in which variables are the leading components.  
This full chain development represents the source-impact chain. The INTARESE assessment framework for full chain development has been based on different frameworks developed from the pressure-state-response (PSR) concept originally proposed by the US-EPA (e.g. DPSIR, DPSEEA) and the source-receptor models widely used to represent the fate of pollutants in the environment. (D. Briggs, 16.5.06)
This full chain development represents the source-impact chain. The INTARESE assessment framework for full chain development has been based on different frameworks developed from the pressure-state-response (PSR) concept originally proposed by the US-EPA (e.g. DPSIR, DPSEEA) and the source-receptor models widely used to represent the fate of pollutants in the environment. (D. Briggs, 16.5.06)
Line 59: Line 28:
Based on the INTARESE assessment framework, two approaches for full chain development can be distinguished, based on the point in development where causal linkages between variables are formulated.
Based on the INTARESE assessment framework, two approaches for full chain development can be distinguished, based on the point in development where causal linkages between variables are formulated.
   
   
[[image:Causal links defined with variables.PNG]]
Figure 1: Causal linkages are defined in line with variable development
Figure 1: Causal linkages are defined in line with variable development


[[image:Causal links defined after variables.PNG]]
   
   
Figure 2: Variables are defined and subsequently linked
Figure 2: Variables are defined and subsequently linked
   
   
The first approach in figure 1 starts from the assessment framework and translates this into a set of variables (circles) and functions (connecting arrows), representing the main elements of the system that will be assessed.  
The first approach in figure 1 starts from the assessment framework and translates this into a set of variables (circles) and functions (connecting arrows), representing the main elements of the system that will be assessed.  
To supplement this, and to ensure that all the terms used in the assessment are consistent and explicit, the variables used in the assessment process should also be defined and described. Descriptions should cover the methods/models used to compute or derive the variable, and the data (and associated data sources) on which these are based. ‘Variables’ in this context may take different forms and serve different roles (often simultaneously); they represent inputs to models (derived variables), interim steps in the calculation process (derived variables) and outputs for reporting (indicators).  
To supplement this, and to ensure that all the terms used in the assessment are consistent and explicit, the variables used in the assessment process should also be defined and described. Descriptions should cover the methods/models used to compute or derive the variable, and the data (and associated data sources) on which these are based. ‘Variables’ in this context may take different forms and serve different roles (often simultaneously); they represent inputs to models (derived variables), interim steps in the calculation process (derived variables) and outputs for reporting (indicators).  
A tool for describing variables is available in the INTARESE Wiki site, though currently the descriptions that this provides are deliberately relatively simple. As such, it provides only limited descriptions of the computation procedures. (Tuomisto and Pohjola, April 2007)
A tool for describing variables is available in the INTARESE Wiki site, though currently the descriptions that this provides are deliberately relatively simple. As such, it provides only limited descriptions of the computation procedures. (Tuomisto and Pohjola, April 2007)
The second approach in Figure 2 shows the issue framework. The main emphasis is on identification and definition of a set of variables, whilst the causal linkages between them are established and defined at a later stage. Issue framing definitely considers the relationship between the variables, since they are in some way positioned towards each other in the assessment framework. Variables are individual components in the assessment framework. Since computation methods and models as well as data are separately defined for each variable, they may be inconsistent, hampering the creation of causal linkages.  
The second approach in Figure 2 shows the issue framework. The main emphasis is on identification and definition of a set of variables, whilst the causal linkages between them are established and defined at a later stage. Issue framing definitely considers the relationship between the variables, since they are in some way positioned towards each other in the assessment framework. Variables are individual components in the assessment framework. Since computation methods and models as well as data are separately defined for each variable, they may be inconsistent, hampering the creation of causal linkages.  




2. The terminology: Indicator  
==The terminology: Indicator ==


Indicators are variables of specific interest.
Indicators are variables of specific interest.
Line 77: Line 52:




Types of indicators
==Types of indicators==
 
Indicators can take very different forms. Detailed categorisation of different types of indicator is fraught with difficulty, and is unlikely to be helpful. In terms of environmental health, a distinction has sometimes been made, however, between exposure-side indicators and health-side indicators. This distinction is useful in relation to INTARESE, because it discriminates between the forward looking indicators of exposure (i.e. those that presage, and need to be linked to, a potential health effect) and the backward looking indicators of outcome or effect (i.e. those that imply, and need to be attributed to, an exposure or source).  
Indicators can take very different forms. Detailed categorisation of different types of indicator is fraught with difficulty, and is unlikely to be helpful. In terms of environmental health, a distinction has sometimes been made, however, between exposure-side indicators and health-side indicators. This distinction is useful in relation to INTARESE, because it discriminates between the forward looking indicators of exposure (i.e. those that presage, and need to be linked to, a potential health effect) and the backward looking indicators of outcome or effect (i.e. those that imply, and need to be attributed to, an exposure or source).  


Line 85: Line 61:


In each case, the indicators may be expressed in different ways, depending on:  
In each case, the indicators may be expressed in different ways, depending on:  
*Whether they are static (state, condition) or dynamic (process, flux) indicators;
*Whether they are expressed in quantitative (‘objective’) or qualitative (perception) measures;
*Whether or not they relate to a formal (and internal) reference level or target (performance indicators). (D. Briggs, 16.5.06)
==Indicator selection from different perspectives==


Whether they are static (state, condition) or dynamic (process, flux) indicators;
[[Image:Indicator development as a process.PNG]]
Whether they are expressed in quantitative (‘objective’) or qualitative (perception) measures;
Whether or not they relate to a formal (and internal) reference level or target (performance indicators). (D. Briggs, 16.5.06)


Figure 3: Indicator development as a process e.g. Pyrkilo


3. Indicator selection from different perspectives
[[Image:Indicator development as individual components.PNG]]
Figure 4: Indicator development as individual components e.g. WHO


==Suggested Intarese attributes==


Figure 3: Indicator development as a process e.g. Pyrkilo


#Name
#Scope
#Description
#*Scale
#*Averaging period
#*References
#Unit
#Definition
#*Causality
#*Data
#*Formula
#**Variations and alternatives
#Result
#Discussion


Figure 4: Indicator development as individual components e.g. WHO


4. A proposal for indicator attributes
==Variable definition tools ==


Table. A comparison of attributes used in Intarese (suggestions), ENHIS indicators, pyrkilo method, and David's earlier version.


Name  
{|{{prettytable}}
Scope  
! Suggested Intarese attributes
Description  
! WHO indicator attributes
Scale  
! Pyrkilo variable attributes
Averaging period  
! [[Policy assessment protocols (Intarese method)|David's variable]] attributes
Non-causal links  
|-----
References
| Name
Unit  
| Name
Definition  
| Name
Causality  
| Name
Data  
|-----
Formula  
| Scope
Variations and alternatives
| Issue
Result
| Scope
Discussion
| Detailed definition
|-----
| Description
| Definition and description
| Description (part of)
| -
|-----
| Description (part of)
| Interpretation
| Description (part of)
| -
|-----
| Description / Scale
| Scale
| Scope or Description
| Geographical scale
|-----
| Description / Averaging period
| -
| Scope or description
| Averaging period
|-----
| Description / Variations and alternatives
| -
| Description
| Variations and alternatives
|-----
| Description (part of)
| Linkage to other indicators
| Description (part of)
| - {{reslink|Do non-causal links between variables exist?}}
{{discussion
|Dispute= Do non-causal links between variables exist?
|Outcome= Yes, but David's "Links to other variables" are causal and they are removed from here.
|Argumentation =
{{attack|#1: |Why should this be non-causal?|--[[User:Alexandra Kuhn|Alexandra Kuhn]] 14:56, 23 March 2007 (EET)}}
:{{attack_invalid|#4: |Because causal links are described in Definition / Causality.|--[[User:Jouni|Jouni]] 00:18, 28 March 2007 (EEST)}}
::{{attack|#7: |I thought that EVERY link to other variables is causal.|--[[User:Alexandra Kuhn|Alexandra Kuhn]] 17:04, 28 March 2007 (EEST)}}
:::{{defend|#8: |David's links to other variables seem to be, but WHO "Linkage to other indicators" are associations. David's attribute should be removed from here.|--[[User:Jouni|Jouni]] 11:51, 5 April 2007 (EEST)}}
}}
|-----
| Unit
| Units
| Unit
| Units of measurement
|-----
| Definition / Causality
| Not relevant
| Definition / Causality
| Links to other variables
|-----
| Definition / Data {{reslink|Data sources belong to Data}}
{{discussion
|Dispute= Data sources belong to data
|Outcome= Accepted.
|Argumentation =
{{defend|#2: |Why do you allocate data sources etc. to references?|--[[User:Alexandra Kuhn|Alexandra Kuhn]] 14:56, 23 March 2007 (EET)}}
:{{defend|#5: |Maybe it should be in Definition / Data. These things may change, as there is not very careful thinking behind.|--[[User:Jouni|Jouni]] 00:18, 28 March 2007 (EEST)}}
}}
| Data sources or Related data
| Definition / Data
| Data sources, availability and quality
|-----
| Definition / Formula
| Computation
| Definition / Formula
| Computation algorithm/model
|-----
| Result (a very first draft of it) {{reslink|Worked example is the same thing as Result}}
{{discussion
|Dispute= Worked example is the same thing as Result.
|Outcome= They are the same concept, but a worked example is a very first draft of what the result could be.
|Argumentation =
{{attack_invalid|#3: |If a result represents the value of a variable, e.g. the actual calculated emissions this is not the same as a worked example.|--[[User:Alexandra Kuhn|Alexandra Kuhn]] 14:56, 23 March 2007 (EET)}}
:{{attack|#6: |So what is a worked example, then, if it is not the result? I have understood something wrong.|--[[User:Jouni|Jouni]] 00:18, 28 March 2007 (EEST)}}
::{{comment|#9: |I don't know. I did not suggest it. But I thought it might be an example for calculation. Do we need that?|--[[User:Alexandra Kuhn|Alexandra Kuhn]] 17:02, 28 March 2007 (EEST)}}
:{{attack|#10: |They are the same concept, but a worked example is a very first draft of what the result could be. You are right, we don't need it as a separate attribute in the Intarese variables|--[[User:Jouni|Jouni]] 11:51, 5 April 2007 (EEST)}}
}}
| Not a specific attribute
| Result (a very first draft of it)
| Worked example
|-----
|Discussion
| -
| -
| -
|-----
| Done by using categories
| -
| Done by using categories
| Type
|-----
| Done by links to glossary
| -
| Done by links to glossary
| Terms and concepts
|-----
| Done by argumentation on the Discussion area
| Specification of data needed
| Done by argumentation on the Discussion page
| Data needs
|-----
| The postition in a causal diagram justifies the existense
| Justification
| The postition in a causal diagram justifies the existense
| -
|-----
| Not relevant
| Policy context
| Not relevant
| Not relevant
|-----
| Not relevant
| Reporting obligations
| Not relevant
| Not relevant
|}

Revision as of 19:08, 5 April 2007

Guidance and methods for indicator selection and specification

First draft 5 April 2007

Mikko Pohjola, Eva Kunseler, Jouni Tuomisto KTL, Finland

Leendert van Bree MNP, The Netherlands

Workpackage 1.4 Risk Characterization

The INTARESE context: Full chain development

Integrated risk assessment, as applied in the INTARESE project, can be defined as the assessment of risks to human health from environmental stressors based on a ‘whole system’ approach. It thus endeavours to take account of all the main factors, links, effects and impacts relating to a defined issue or problem, and is deliberately more inclusive (less reductionist) than most traditional risk assessment procedures. (D. Briggs, 16.5.06)

Key characteristics of integrated assessment are:

  1. It is designed to assess complex policy-related issues and problems, in a more comprehensive and inclusive manner than that usually adopted by traditional risk assessment methods.
  2. It takes a ‘full-chain’ approach – i.e. it explicitly attempts to define and assess all the important links between source and impact, in order to allow the determinants and consequences of risk to be tracked in either direction through the system (from source to impact, or from impact back to source).
  3. It takes account of the additive, interactive and synergistic effects within this chain and uses assessment methods that allow these to be represented in a consistent and coherent way (i.e. without double-counting or exclusion of significant effects).
  4. It presents results of the assessment as a linked set of policy-relevant ‘outcome indicators’.
  5. It makes the best possible use of the available data and knowledge, whilst recognising the gaps and uncertainties that exist; it presents information on these uncertainties at all points in the chain. (D. Briggs, 16.5.06)

The INTARESE approach emphasizes on creation of causal linkages between the determinants and consequences in the integrated assessment process. For this purpose, a full chain diagram should be constructed in which variables are the leading components. This full chain development represents the source-impact chain. The INTARESE assessment framework for full chain development has been based on different frameworks developed from the pressure-state-response (PSR) concept originally proposed by the US-EPA (e.g. DPSIR, DPSEEA) and the source-receptor models widely used to represent the fate of pollutants in the environment. (D. Briggs, 16.5.06)

Based on the INTARESE assessment framework, two approaches for full chain development can be distinguished, based on the point in development where causal linkages between variables are formulated.

File:Causal links defined with variables.PNG

Figure 1: Causal linkages are defined in line with variable development

File:Causal links defined after variables.PNG

Figure 2: Variables are defined and subsequently linked

The first approach in figure 1 starts from the assessment framework and translates this into a set of variables (circles) and functions (connecting arrows), representing the main elements of the system that will be assessed.

To supplement this, and to ensure that all the terms used in the assessment are consistent and explicit, the variables used in the assessment process should also be defined and described. Descriptions should cover the methods/models used to compute or derive the variable, and the data (and associated data sources) on which these are based. ‘Variables’ in this context may take different forms and serve different roles (often simultaneously); they represent inputs to models (derived variables), interim steps in the calculation process (derived variables) and outputs for reporting (indicators).

A tool for describing variables is available in the INTARESE Wiki site, though currently the descriptions that this provides are deliberately relatively simple. As such, it provides only limited descriptions of the computation procedures. (Tuomisto and Pohjola, April 2007)

The second approach in Figure 2 shows the issue framework. The main emphasis is on identification and definition of a set of variables, whilst the causal linkages between them are established and defined at a later stage. Issue framing definitely considers the relationship between the variables, since they are in some way positioned towards each other in the assessment framework. Variables are individual components in the assessment framework. Since computation methods and models as well as data are separately defined for each variable, they may be inconsistent, hampering the creation of causal linkages.


The terminology: Indicator

Indicators are variables of specific interest.

Indicator selection provides the bridge between the issue framework and the assessment process. It involves specifying the outcome measures to be used as a basis for risk characterisation. Indicators should ideally represent all the main nodes and links that make up the source-impact chain, and should be internally coherent – i.e. they should have clear and definable relationships within the context of this chain. (D. Briggs, 16.5.06)


Types of indicators

Indicators can take very different forms. Detailed categorisation of different types of indicator is fraught with difficulty, and is unlikely to be helpful. In terms of environmental health, a distinction has sometimes been made, however, between exposure-side indicators and health-side indicators. This distinction is useful in relation to INTARESE, because it discriminates between the forward looking indicators of exposure (i.e. those that presage, and need to be linked to, a potential health effect) and the backward looking indicators of outcome or effect (i.e. those that imply, and need to be attributed to, an exposure or source).

Exposure-side indicators are clearly relevant for policy, since they often provide the first indications of the potential for health risk, and the first evidence of the effects of intervention (since many policies are focused on the upper links in the source-impact chain). To be meaningful in the context of health risks, however, they must relate to factors with definable (or at least strongly plausible) links to health outcome.

Health-side (or outcome) indicators represent the consequences of exposures in terms of health effect (e.g. mortality, morbidity, DALYs) or its further societal impacts (e.g. economic costs, quality of life). Again, to be meaningful in the context of the full-chain approach, they need to have an explicit link back to causal environmental exposures and risk factors.

In each case, the indicators may be expressed in different ways, depending on:

  • Whether they are static (state, condition) or dynamic (process, flux) indicators;
  • Whether they are expressed in quantitative (‘objective’) or qualitative (perception) measures;
  • Whether or not they relate to a formal (and internal) reference level or target (performance indicators). (D. Briggs, 16.5.06)

Indicator selection from different perspectives

File:Indicator development as a process.PNG

Figure 3: Indicator development as a process e.g. Pyrkilo

File:Indicator development as individual components.PNG

Figure 4: Indicator development as individual components e.g. WHO

Suggested Intarese attributes

  1. Name
  2. Scope
  3. Description
    • Scale
    • Averaging period
    • References
  4. Unit
  5. Definition
    • Causality
    • Data
    • Formula
      • Variations and alternatives
  6. Result
  7. Discussion


Variable definition tools

Table. A comparison of attributes used in Intarese (suggestions), ENHIS indicators, pyrkilo method, and David's earlier version.

Suggested Intarese attributes WHO indicator attributes Pyrkilo variable attributes David's variable attributes
Name Name Name Name
Scope Issue Scope Detailed definition
Description Definition and description Description (part of) -
Description (part of) Interpretation Description (part of) -
Description / Scale Scale Scope or Description Geographical scale
Description / Averaging period - Scope or description Averaging period
Description / Variations and alternatives - Description Variations and alternatives
Description (part of) Linkage to other indicators Description (part of) - R↻

How to read discussions

Fact discussion: .
Opening statement:

Closing statement: Resolution not yet found.

(A closing statement, when resolved, should be updated to the main page.)

Argumentation:

⇤--#1:: . Why should this be non-causal? --Alexandra Kuhn 14:56, 23 March 2007 (EET) (type: truth; paradigms: science: attack)

⇤--#4:: . Because causal links are described in Definition / Causality. --Jouni 00:18, 28 March 2007 (EEST) (type: truth; paradigms: science: attack)
⇤--#7:: . I thought that EVERY link to other variables is causal. --Alexandra Kuhn 17:04, 28 March 2007 (EEST) (type: truth; paradigms: science: attack)
←--#8:: . David's links to other variables seem to be, but WHO "Linkage to other indicators" are associations. David's attribute should be removed from here. --Jouni 11:51, 5 April 2007 (EEST) (type: truth; paradigms: science: defence)
Unit Units Unit Units of measurement
Definition / Causality Not relevant Definition / Causality Links to other variables
Definition / Data R↻

How to read discussions

Fact discussion: .
Opening statement:

Closing statement: Resolution not yet found.

(A closing statement, when resolved, should be updated to the main page.)

Argumentation:

←--#2:: . Why do you allocate data sources etc. to references? --Alexandra Kuhn 14:56, 23 March 2007 (EET) (type: truth; paradigms: science: defence)

←--#5:: . Maybe it should be in Definition / Data. These things may change, as there is not very careful thinking behind. --Jouni 00:18, 28 March 2007 (EEST) (type: truth; paradigms: science: defence)
 Data sources or Related data Definition / Data Data sources, availability and quality
Definition / Formula Computation Definition / Formula Computation algorithm/model
Result (a very first draft of it) R↻

How to read discussions

Fact discussion: .
Opening statement:

Closing statement: Resolution not yet found.

(A closing statement, when resolved, should be updated to the main page.)

Argumentation:

⇤--#3:: . If a result represents the value of a variable, e.g. the actual calculated emissions this is not the same as a worked example. --Alexandra Kuhn 14:56, 23 March 2007 (EET) (type: truth; paradigms: science: attack)

⇤--#6:: . So what is a worked example, then, if it is not the result? I have understood something wrong. --Jouni 00:18, 28 March 2007 (EEST) (type: truth; paradigms: science: attack)
----#9:: . I don't know. I did not suggest it. But I thought it might be an example for calculation. Do we need that? --Alexandra Kuhn 17:02, 28 March 2007 (EEST) (type: truth; paradigms: science: comment)
⇤--#10:: . They are the same concept, but a worked example is a very first draft of what the result could be. You are right, we don't need it as a separate attribute in the Intarese variables --Jouni 11:51, 5 April 2007 (EEST) (type: truth; paradigms: science: attack)
 Not a specific attribute Result (a very first draft of it) Worked example
Discussion - - -
Done by using categories - Done by using categories Type
Done by links to glossary - Done by links to glossary Terms and concepts
Done by argumentation on the Discussion area Specification of data needed Done by argumentation on the Discussion page Data needs
The postition in a causal diagram justifies the existense Justification The postition in a causal diagram justifies the existense -
Not relevant Policy context Not relevant Not relevant
Not relevant Reporting obligations Not relevant Not relevant
Retrieved from "https://en.opasnet.org/index.php?title=Guidance_and_methods_for_indicator_selection_and_specification&oldid=13488"