|
|
| (14 intermediate revisions by 4 users not shown) |
| Line 1: |
Line 1: |
| <accesscontrol>Members of projects</accesscontrol>
| | #REDIRECT [[Properties of good assessment]] |
| | |
| It is possible to describe the '''general properties of good risk assessments'''. They are based on considerations about the purpose of risk assessments and the relations of risk assessments with their societal context. The general properties also function as the general performance criteria in evaluating risk assessments. These properties and the principles they are based on also have certain implications to risk assessment methodology. {{disclink|Moderator qualifications}}
| |
| | |
| The diagram below illustrates the general properties of good risk assessments as a tree structure. The goal, good risk assessment, is the node on the left of the diagram and the required properties to achieve this goal are then defined and divided moving towards right in the diagram. It is important to notice that the arrows in this diagram describe how particular properties lead to the ultimate objective.
| |
| | |
| | |
| <center>
| |
| [[Image:General_properties_of_good_RA_graph.PNG]]
| |
| </center>
| |
| | |
| | |
| The general properties of good risk assessments consist of three different categories:
| |
| *Quality of content
| |
| *Applicability of output
| |
| *Efficiency of process
| |
| | |
| The first two categories, quality of content and applicability of output together form the ''effectiveness of risk assessment product'', so the properties of good risk assessments overall cover both the effectiveness and the efficiency of risk assessment. The effectiveness here means the potential of the assessment to have intended influence on the decision-making processes where the risk assessment outputs are intended to be used and the outcomes these processes. Effectiveness thus also indicates the potential of advancing towards the primary purpose of risk assessment, improved societal decisions. The categories and the properties belonging to both effectiveness and efficiency are describes in more detail below. {{reslink|Selection of indicators}} {{reslink|uncertainty}}
| |
| | |
| | |
| '''''Quality of content''''' refers to the goodness of the description that is produced in the assessment in describing reality. It consists of the properties called informativeness, calibration and relevance. The point of reference in considering the quality of content for an object is the particular piece of reality that the object is intended to describe. Goodness in terms of quality of content expresses how well does the description match reality and refers to the general purpose of risk assessments to describe reality. Quality of content can be defined for definition and result attributes of variable and assessment objects.
| |
| | |
| *extension of explanations to cover qualitative descriptions
| |
| | |
| '''Informativeness''' means the ''tightness of spread'' in a distribution (All results estimates of variables should be considered as distribution estimates of some form rather than point estimates). The tighter the spread, the smaller the variance and the better the informativeness. Informativeness is a property of each individual variable, but the informativeness of each variable is also affected by the informativeness of the variables ''upstream'' in the causal chain.
| |
| | |
| '''Calibration''' means the ''correctness'' or ''exactness'' of the result estimate of a variable, i.e. how close it is to the ''real'' phenomenon it describes. Evaluating calibration can be complicated in many situations, but it is necessary to realize it as an important property, when evaluating the goodness of result estimates of an assessment or individual variables.
| |
| | |
| '''Relevance''' can be described as the coherence of the assessment, i.e. does the description include the necessary variables and their inter-relations to describe the whole assessed issue sufficiently in relation to the endpoints and use purpose of the assessment. On variable-level relevance can be described e.g. as relevance of certain part of a variable description in relation to its scope, i.e. does the part of description fit within the scope defined for the variable. {{reslink|Relation of relevance to content quality and applicability}}
| |
| | |
| | |
| '''''Applicability''''' refers to the potential of transferring the content of the assessment to those who are intended to use it or who are affected by the use of it (decision-makers and other stakeholders, respectively). Applicability consists of the properties called usability, availability and acceptability. Acceptability can be further defined as acceptability of premises and acceptability of the assessment process. Whereas the properties related to quality of content can often be evaluated more or less objectively, properties related to applicability are evaluated subjectively by different actors, based on their role in the societal context of risk assessment. Applicability is defined in relation to the use purpose of an assessment and can only be defined on assessment-level, not for individual variables.
| |
| | |
| '''Usability''' refers to issues that affect the understanding of the content. These are such as e.g. clarity of presentation, language used etc. Usability is strongly influenced also by the capabilities and other properties of the users and is often not fully controllable by the ones who produce the description. Usability can anyhow be improved especially if the use purpose of the assessment is understood and the ''intended users and uses'' are identified and defined.
| |
| | |
| '''Availability''' refers to the openness of access of the intended users to use the product in relation to their needs. The openness is related to issues such as e.g. chosen media of the description, spatial and temporal accessibility and restrictions of access to parts of description content.
| |
| | |
| '''Acceptability''' is especially strongly influenced by the role of ''the acceptor'' in relation to the assessment process. E.g. the risk assessor community evaluates their acceptance primarily in relation to the quality of the content of the assessment product, the decision-makers evaluate their acceptance primarily in relation to their information needs for decision-making and stakeholders are primarily concerned about the consequences of the decisions and their executions. Different actors may of course represent different groups simultaneously or take different points of view into the issue. This blurring of roles is considered more in [[Help:Mass collaboration | mass collaboration in risk assessment]] and [[Help:Managing stakeholder involvement|managing stakeholder involvement]]. Acceptability from all different perspectives can be divided into '''acceptability of the premises''' used in making the assessment and '''acceptability of the assessment process'''.
| |
| | |
| | |
| Whereas the properties related to effectiveness are primarily related to the output of the assessment, '''''Efficiency''''' is a property of the assessment process. Basically efficiency can be described as the goodness of the output (as a function of quality of content & applicability) in relation to the efforts spent in producing the output. Also efficiency can only be defined for assessments, or groups of assessments, not for individual variables.
| |
| | |
| '''Intra-assessment efficiency''' means the efficiency within a certain assessment, i.e the spending of efforts in carrying out a particular single assessment in relation to the output and outcomes produced. The output here refers to the product of the assessment and the outcomes refer to the overall effects of the processes along the chain which the output has influence on.
| |
| | |
| '''Inter-assessment efficiency''' refers to the reduction rate of the marginal efforts needed for each new assessment with the same quality of output when making a series of assessments. This means in practice the ability to avoid doing the same work again if it has been already done in a previous assessment.
| |
| | |
| | |
| The properties of good risk assessment described above can be used as the '''general performance criteria of risk assessment'''. The success of risk assessments should thus be evaluated against these goodness properties. As was mentioned above, the ultimate goal should be good societal decisions, but the decision-making process is basically out of reach of risk assessors and there are always several external factors that influence the decisions and their evaluation. Therefore it is more fair for risk assessors if instead the risk assessments are evaluated based on the effectiveness ''potential'' of the assessment product and the efficiency of the assessment process.
| |
| | |
| In general terms it can be said that the properties related to the quality of content are the most crucial ones. Assuring the goodness of the description should thus be the first priority in risk assessment. Anyhow, the goodness of the description has most likely very little significance to the society if the applicability of the assessment remains low. It is therefore also important to explicitly consider all of the aspects of effectiveness when designing or carrying out risk assessments. The requirement for efficiency is mainly related to the practical limitations and inevitably scarce resources for making the assessments. It is necessary to strive for making the best use of the resources.
| |
| | |
| The performance of risk assessment is tightly related to its purpose. When the purpose of a particular assessment is defined properly, also the performance criteria can be set in more detail. It is also necessary to evaluate the performance against these criteria during the assessment process as well as after the assessment process when the intended effects are to become realized. Evaluating the performance of risk assessments is considered in more detail on [[Help:Performance]].
| |
| | |
| | |
| It is relatively easy to see that '''different categories of the properties described above relate to different parts of risk assessment''':
| |
| *quality of content → particular assessment or variable (description of reality)
| |
| *applicability → use process (intended (instrumental) use purpose)
| |
| *efficiency → risk assessment process (a technical measure)
| |
| | |
| | |
| Identifying, defining and considering the general properties of good risk assessments also has its '''implications to the methods that are used in risk assessment'''. The methods should be such that these properties become adequately covered. Several different kinds of methods are needed to be used and combined:
| |
| | |
| #methods to ensure/enhance the quality of the description content
| |
| #methods to ensure/enhance the applicability of the risk assessment output
| |
| #methods to ensure/enhance the efficiency of the process
| |
| | |
| The relations between general properties of good risk assessments and uncertainty are described in more detail in: [[Help:Uncertainty]]
| |
| | |
| [[Category:General assessment framework]]
| |