Selecting indicators in IEHIAS

The text on this page is taken from an equivalent page of the IEHIAS-project.

Assessments of environmental health issues are often complex, and can produce a large array of information – far beyond what many users will need or be able to cope with. For most users, therefore, a more limited set of synoptic information is required, which captures the most important elements of the assessment. These are provided by indicators.

The way in which indicators are selected and defined is vital, for they are the main results from the assessment: what the indicators tell us is what we learn from the assessment, and what is not covered by an indicator might as well not be assessed.

At the same time, it is important to remember that many different stakeholders may have interests in the assessment, and their interests may not all be the same. Different stakeholders may therefore wish to emphasise different aspects of the assessment, and thus demand different indicators.

For this reason, it is important wherever possible to involve stakeholders in the process of selecting indicators.

Indicators can thus take many different forms, and serve different purposes for the stakeholders concerned. Pressures also often occur to apply a large number of indicators, in the attempt to capture every aspect of the assessment and satisfy every interest. This can, of course, be dangerous, for it leads to difficulties in bringing together the results of the assessment, and agreeing on what conclusions to draw.

To ensure that the indicators do justice to the assessment, and meet the needs of users, they need to be selected and constructed with great care. Clear and strict (and agreed) criteria need to be established and agreed at the start; thought and honesty need to be applied in ensuring that the indicators meet these criteria. Ideally, also, the indicators should be tested before they are used, in order to make sure that they are valid and work effectively.

Types of indicator

Indicators may take many different forms – and part of the art of indicator development is to ensure that the right types, and an appropriate range and number, of indicators are selected to describe the issue of interest.

Three different aspects of indicators need to be given particular attention:

Position in causal chain – i.e. where within the chain from sources to impacts the indicators are targetted;
Structure – i.e. how the indicators are constructed and their associated level of complexity;
Perspective – i.e. the value system from which the indicators are defined.

Position in the causal chain

Although the ultimate aim of assessment is to give measures of health effects and their associated impacts, these are not always the only things of concern. If we want to have prior warning of potential health effects, for example, we might want information from earlier in the causal chain - e.g. on exposures or environmental concentrations. If we want to highlight the original causes of health effects, and places where we might take preventative action, then we might look earlier still, at source activities and emissions.

It is therefore useful to recognise three key types of indicator, dependent on their position in the causal chain:

Endpoint indicators, which give a measures of health effect or impact. These include not only direct expressions of health outcome (e.g. in terms of total mortality or morbidity), but also more synoptic measures such as disease- or quality-adjusted life years (DALYs and QALYs) and monetary cost.
Midpoint indicators, representing dose, exposure or environmental concentrations. While these are usually derived from the use of some form of modelling, they may in some cases include indicators derived from biomonitoring.
Source indicators, representing emissions, source intensity or more general measures of socio-economic activity.

If assessments are to give full insight into risks to human health, a suite of indicators may thus be necessary, spanning the causal chain. In the case of an assessment of the health effects of air pollution, for example, these might include measures of:

Road traffic volumes, to represent source activities;
Annual emissions of PM2.5 (or some other marker) to represent releases into the environment;
Mean annual concentrations of PM2.5 to represent environmental conditions;
Estimated mean annual exposures to ambient PM2.5 to represent exposures;
Excess mortality attributable to air pollution, to represent health effects.

DALYs and QALYs

Health impact assessments can be executed by calculating the attributable burden of disease. There are several ways to assess the burden of disease attributable to an (environmental) factor, such as the QALY and the DALY.

Quality Adjusted Life Years, QALYs, capture both the quality and quantity elements of health in one indicator. Essentially, time spent in ill health (measured in years) is multiplied by a weight measuring the relative (un)desirability of the illness state. Thereby a number is obtained which represents the equivalent number of years with full health. QALYs are commonly used for cost-utility analysis and to appraise different forms of health care. To do that, QALYs combine life years gained as a result of these health interventions/health care programs with a judgment about the quality of these life years.

Disability adjusted life years, DALYs, are comparable to QALYs in that they both combine information on quality and quantity of life. However, contrary to QALYs, DALYs give an indication of the (potential) number of healthy life years lost due to premature mortality or morbidity and are estimated for particular diseases, instead of a health state. Morbidity is weighted for the severity of the disorder.

With QALY, the focus is on assessing individual preference for different non-fatal health outcomes that might result from a specific intervention, whereas the DALY was developed primarily to compare relative burdens among different diseases and among different populations. DALYs are suitable for analysing particular disorders or specific factors that influence health and are therefore more often used in the realm of environmental health impact assessment.

Structure

Indicators may vary greatly in their structure.

One important distinction in this context is between simple and compound (or composite) indicators.

Simple indicators, as their name suggests, target a specific element of the system. Examples (relating to an assessment of air pollution) include:

in terms of health outcome, hospital admissions for acute respiratory infections; in terms of exposures (or more strictly environmental concentrations), mean annual PM2.5 concentrations. Compound indicators, in contrast, combine different elements into a single measure. Examples include:

in terms of health outcome, disease adjusted life years (DALYs) attributable to air pollution or the monetary costs of air pollution-related health effects; in terms of exposure, an overall air quality index (e.g. obtained by combining concentrations of particulates, nitrogen oxides, benzene and ozone). Compound indicators clearly provide more information, and fewer indicators may therefore be needed to summarise the results of an assessment. Care is needed, however, in combining the different elements since this almost invariably involves some form of weighting (e.g. on the basis of toxicity or severity). Since these weights cannot usually be defined on the basis of scientific data, they rely on some form of judgement; the quality of these judgements inevitably determines the reliability of the indicators. In addition, it may be difficult for users to unravel a compound indicator and understand which of its different elements is really driving any effect or change.

Perspective

Indicators can be defined from different perspectives, reflecting different worldviews and value systems. Amongst others, indicators may be based on the following perspectives:

Utilitarian – focused on the overall (total or net) impacts across the population as a whole

e.g. average exposure; total mortality; total monetary cost; total disease adjusted life years;

Normative – focused on the degree to which predefined targets and goals are being met

e.g. percentage of population above the air quality standard; excess mortality above the policy target;

Equity – focused on inequalities in exposures or health risks between different, definable sub-populations, typically delimitated in terms of socio-economic status, gender, ethnicity or age

e.g. ratio of attributable mortality rate in low to high socio-economic groups;

Public perception – focused on the acceptability of, or attitudes towards, conditions by the general public

e.g. percentage of people concerned about air quality conditions.

Different stakeholders often have different worldviews, so care is needed in focusing on only one of these perspectives, since this may bias the assessment towards particular stakeholder groups. In many cases, therefore, a mix of indicators, reflecting different perspectives, may be appropriate.

Building good indicators

For many (if not most) users, the indicators are the assessment: it is what they see at the end of the process, and what conditions how they respond. How we choose and construct indicators is therefore crucial in determining the consequences of the assessment.

Clearly the main need, in most cases, is to obtain some overall measure of impact. Aggregated, synoptic indicators of the impacts are consequently essential outputs from most integrated assessments. They can, however, be framed in very different ways (e.g. as the burden of disease or monetary cost), each of which may tell a somewhat different story and reflect different value systems. They also do not tell the whole story, for by aggregating the impacts into a single measure they inevitably blur the detail, and because they focus on outcomes they only hint at the exposures and sources that have gone before. For many assessments, therefore, a number of different indicators may be needed from different points in the causal chain and based on different perespectives, in order to tell users all they need to know, and to avoid unduly biasing the assessment.

Which to choose, and how to design them, are nevertheless difficult decisions. They need to be based on a clear understanding of the issue that is being addressed, the interests of the stakeholders and the purpose of the assessment, in order to ensure that the indicators are useful for the task in hand. The indicators also need to be individually interpretable (and, beneath that, scientifically valid), as well as collectively comprehensive and balanced, if they are to give reliable information.

The figure below provides a framework for indicator selection and design, and lists key criteria that need to be considered in the process. The links under See also, below, provide access to a more detailed set of guidelines for indicator selection, and a worked example.

A framework for indicator selection and design