Purpose and properties of good assessments

<accesscontrol>Members of projects</accesscontrol>

Quality Criteria: properties of good risk assessment

The purpose of a risk assessment is to improve societal decision-making in a particular risk situation.

← The assessment should provide relevant information about the risk situation in a quantitative form.

← In general, the information provided is about predictions on the impacts of possible decisions on some outcomes that have a societal value. However, the scope of the information can be narrowed in a particular case e.g. in the following ways:

The only decision considered is business-as-usual. This results in estimates of the burden of the current situation.
Outcomes considered are not explicitly valuated, but implicitly it is assumed that they relate to something that has intrinsic value. For example, pollutant concentrations can be estimated, although it is the related health impact that has the intrinsic value.

Risk assessments should always be done for a purpose. When the purpose is identified and kept clear in mind and preferably explicated and made public, it helps to guide the process in producing a desired kind of assessment product. The primary purpose is to improve societal decision making by providing good descriptions of chosen parts of reality for the use of the decision-makers. The process of describing the purpose is now considered as an essential and integral part of the method. Proper identification of the purpose of risk assessment crucially affects the assessment process and the content and essence of the final product.

It is probably relatively easy for most people to agree that the ultimate underlying goal of risk assessments should be good societal decisions. This means that a risk assessment should always be designed and carried out in an attempt to progress towards this goal. For each particular risk assessment the situation is naturally different and thus the outputs and outcomes of each assessment are and should always be case-specific and in accordance with the contextual setting of the particular case. The purpose should still be kept clear in mind when planning or carrying out as well as evaluating an assessment or using its outputs. It is thus highly recommended that the purpose of each risk assessment is identified and defined explicitly already in the early phases of the process and also made public if applicable.

The overall purpose of risk assessment can be considered as composing of two different aspects. The general purpose is to describe reality, i.e. to explicate real-world phenomena in a comprehensible and usable form. But mere describing reality without any specific, identified need for the description would probably not make much sense as such. There must also be a certain purpose to undertake such a task, an instrumental purpose, or use purpose, for the outputs of the assessment. It is important to notice these two simultaneously effective components of purpose and the different requirements they bring to risk assessment. The implications of the different aspects of purpose are discussed in more detail e.g. in Evaluating assessment performance.

At this point an important distinction should be made clear. There are real-world phenomena that can be considered as objects whose interactions constitute the reality. On the other hand, when talking about risk assessment, we talk about objects that are descriptions about these real-world objects/phenomena. It should be kept in mind that these objects are not the same, although they are inherently related. The previous is the original issue itself, while the latter is a chunk of information about it. One should also remember that the influential relation between these objects is only one-way - a change in a real-world object/phenomenon, or in the knowledge about it, changes (or should change) the description object, but a change in the description does not change the real-world object/phenomenon.

In pyrkilo method reality is described as variables and assessments (and also as contexts of assessments if seen necessary). An assessment is an object which creation and development is driven by the instrumental use purpose derived from a practical need in the context of the assessment. A variable is an object which creation is also driven by a practical need related to a particular assessment, but which development is driven by the general purpose of describing reality. See Universal products and Variable for more detailed descriptions on important object types and their structure according to pyrkilo method.

It is possible to describe the general properties of good risk assessments. They are based on considerations about the purpose of risk assessments and the relations of risk assessments with their societal context. The general properties also function as the general performance criteria in evaluating risk assessments. These properties and the principles they are based on also have certain implications to risk assessment methodology. D↷

The diagram below illustrates the general properties of good risk assessments as a tree structure. The goal, good risk assessment, is the node on the left of the diagram and the required properties to achieve this goal are then defined and divided moving towards right in the diagram. It is important to notice that the arrows in this diagram describe how particular properties lead to the ultimate objective.

The general properties of good risk assessments consist of three different categories:

Quality of content
Applicability of output
Efficiency of process

Effectiveness

The first two categories, quality of content and applicability of output together form the effectiveness of risk assessment product, so the properties of good risk assessments overall cover both the effectiveness and the efficiency of risk assessment. The effectiveness here means the potential of the assessment to have intended influence on the decision-making processes where the risk assessment outputs are intended to be used and the outcomes these processes. Effectiveness thus also indicates the potential of advancing towards the primary purpose of risk assessment, improved societal decisions. The categories and the properties belonging to both effectiveness and efficiency are describes in more detail below. R↻ R↻

Quality of the content

Quality of content refers to the goodness of the description that is produced in the assessment in describing reality. It consists of the properties called informativeness, calibration and relevance. The point of reference in considering the quality of content for an object is the particular piece of reality that the object is intended to describe. Goodness in terms of quality of content expresses how well does the description match reality and refers to the general purpose of risk assessments to describe reality. Quality of content can be defined for definition and result attributes of variable and assessment objects.

extension of explanations to cover qualitative descriptions

Informativeness means the tightness of spread in a distribution (All results estimates of variables should be considered as distribution estimates of some form rather than point estimates). The tighter the spread, the smaller the variance and the better the informativeness. Informativeness is a property of each individual variable, but the informativeness of each variable is also affected by the informativeness of the variables upstream in the causal chain.

Is the data variance for each variable small?

Calibration means the correctness or exactness of the result estimate of a variable, i.e. how close it is to the real phenomenon it describes. In other words, is the estimate internally valid? Evaluating calibration can be complicated in many situations, but it is necessary to realize it as an important property, when evaluating the goodness of result estimates of an assessment or individual variables.

Do you measure what you want to measure for your variables?

Relevance can be described as the coherence of the assessment, i.e. does the description include the necessary variables and their inter-relations to describe the whole assessed issue sufficiently in relation to the endpoints and use purpose of the assessment. On variable-level relevance can be described e.g. as relevance of certain part of a variable description in relation to its scope, i.e. does the part of description fit within the scope defined for the variable. R↻

Are there any abundant variables or scenarios in the assessment?

Applicability of the assessment output

Applicability refers to the potential of transferring the content of the assessment to those who are intended to use it or who are affected by the use of it (decision-makers and other stakeholders, respectively). Applicability consists of the properties called usability, availability and acceptability. Acceptability can be further defined as acceptability of premises and acceptability of the assessment process. Whereas the properties related to quality of content can often be evaluated more or less objectively, properties related to applicability are evaluated subjectively by different actors, based on their role in the societal context of risk assessment. Applicability is defined in relation to the use purpose of an assessment and can only be defined on assessment-level, not for individual variables.

Usability refers to issues that affect the understanding of the content. These are such as e.g. clarity of presentation, language used etc. Usability is strongly influenced also by the capabilities and other properties of the users and is often not fully controllable by the ones who produce the description. Usability can anyhow be improved especially if the use purpose of the assessment is understood and the intended users and uses are identified and defined. D↷

Can the intented users use the risk assessment output in their own (decision) context?

Availability refers to the openness of access of the intended users to use the product in relation to their needs. The openness is related to issues such as e.g. chosen media of the description, spatial and temporal accessibility and restrictions of access to parts of description content.

Can the intended users access the risk assessment ouput easily?

Acceptability is especially strongly influenced by the role of the acceptor in relation to the assessment process. E.g. the risk assessor community evaluates their acceptance primarily in relation to the quality of the content of the assessment product, the decision-makers evaluate their acceptance primarily in relation to their information needs for decision-making and stakeholders are primarily concerned about the consequences of the decisions and their executions. Different actors may of course represent different groups simultaneously or take different points of view into the issue. This blurring of roles is considered more in mass collaboration and organizing stakeholder involvement. Acceptability from all different perspectives can be divided into acceptability of the premises used in making the assessment and acceptability of the assessment process.

Are the intended users willing to use the assessment ouput?

Efficiency

Whereas the properties related to effectiveness are primarily related to the output of the assessment, Efficiency is a property of the assessment process. Basically efficiency can be described as the goodness of the output (as a function of quality of content & applicability) in relation to the efforts spent in producing the output. Also efficiency can only be defined for assessments, or groups of assessments, not for individual variables.

Intra-assessment efficiency means the efficiency within a certain assessment, i.e the spending of efforts in carrying out a particular single assessment in relation to the output and outcomes produced. The output here refers to the product of the assessment and the outcomes refer to the overall effects of the processes along the chain which the output has influence on.

Inter-assessment efficiency refers to the reduction rate of the marginal efforts needed for each new assessment with the same quality of output when making a series of assessments. This means in practice the ability to avoid doing the same work again if it has been already done in a previous assessment.

The properties of good risk assessment described above can be used as the general performance criteria of risk assessment. The success of risk assessments should thus be evaluated against these goodness properties. As was mentioned above, the ultimate goal should be good societal decisions, but the decision-making process is basically out of reach of risk assessors and there are always several external factors that influence the decisions and their evaluation. Therefore it is more fair for risk assessors if instead the risk assessments are evaluated based on the effectiveness potential of the assessment product and the efficiency of the assessment process.

In general terms it can be said that the properties related to the quality of content are the most crucial ones. Assuring the goodness of the description should thus be the first priority in risk assessment. Anyhow, the goodness of the description has most likely very little significance to the society if the applicability of the assessment remains low. It is therefore also important to explicitly consider all of the aspects of effectiveness when designing or carrying out risk assessments. The requirement for efficiency is mainly related to the practical limitations and inevitably scarce resources for making the assessments. It is necessary to strive for making the best use of the resources.

The performance of risk assessment is tightly related to its purpose. When the purpose of a particular assessment is defined properly, also the performance criteria can be set in more detail. It is also necessary to evaluate the performance against these criteria during the assessment process as well as after the assessment process when the intended effects are to become realized. Evaluating the performance of risk assessments is considered in more detail on Evaluating assessment performance.

It is relatively easy to see that different categories of the properties described above relate to different parts of risk assessment:

quality of content → particular assessment or variable (description of reality)
applicability → use process (intended (instrumental) use purpose)
efficiency → risk assessment process (a technical measure)

Identifying, defining and considering the general properties of good risk assessments also has its implications to the methods that are used in risk assessment. The methods should be such that these properties become adequately covered. Several different kinds of methods are needed to be used and combined:

methods to ensure/enhance the quality of the description content
methods to ensure/enhance the applicability of the risk assessment output
methods to ensure/enhance the efficiency of the process

The relations between general properties of good risk assessments and uncertainty are described in more detail in: Assessing uncertainty