Talk:Properties of good assessment

Uncertainty

How to read discussions

Fact discussion: .
Opening statement: Closing statement: Resolution not yet found. (A closing statement, when resolved, should be updated to the main page.)
Argumentation: ←--#1:: . Including the diagnosis and communication of uncertainty as criteria for good integrated risk assessment would be in line with W.P.1.5 deliverable 7: Uncertainty report's message: "Experts must be able to systematically diagnose and communicate the uncertainties characterising their assessments" (p.26) "in order to provide an impartial source of facts upon which policy decisions can be based." (p.9) --Sjuurd 17:40, 27 June 2007 (EEST) (type: truth; paradigms: science: defence) ←--#2:: . Diagnosing and communicating uncertainties of the assessment and its parts are important. They are covered in the properties of good risk assessment under the quality of content (informativeness and calibration) and applicability (usability and availability). In essence, the informativeness and calibration describe how well defined, i.e. how uncertain, the contents of the assessment are. Usability and availability describe how well the information, e.g. about the uncertainty, is conveyed to its target. --Mikko 18:16, 27 June 2007 (EEST) (type: truth; paradigms: science: defence) ----#(3):: . Should this then not be explified in the explaining texts on informativeness, calibration, usability, and availability? --Sjuurd II 13:28, 22 February 2008 (EET) (type: truth; paradigms: science: comment)

Evaluation against a standard

How to read discussions

Fact discussion: .
Opening statement: Closing statement: Resolution not yet found. (A closing statement, when resolved, should be updated to the main page.)
Argumentation: ----#1:: . Against what do you compare if there is no such thing as the golden standard? --Alexandra Kuhn 18:02, 17 March 2008 (EET) (type: truth; paradigms: science: comment) ←--#2:: . Evaluation of calibration and informativeness is only possible against an external standard. In expert elicitation, there exists a "golden standard", which is the set of seed questions for which true answers are known. Even if the external standard is not golden (i.e., it can be the personal opinion of an external reviewer), it can be used. Of course, the evaluation can only be as good as is the standard that is used. --Jouni 00:52, 18 March 2008 (EET) (type: truth; paradigms: science: defence)

Moderator qualifications

How to read discussions

Fact discussion: .
Opening statement: Closing statement: Resolution not yet found. (A closing statement, when resolved, should be updated to the main page.)
Argumentation: ←--#1:: . If I've understood right, a good general level understanding is required from a person to be a moderator in a risk assessment. The quality of the assessment is thought to be achieved, basically, through several variable-related, variable-linkage-related, argumentation-based quality aspects. I don't see these aspects adequate SOLELY as themselves for the guarantee of good risk assessments (RAs) with good outcomes. The moderator has to understand complex causes and effects. How could she/he build up a relevant full-chain RA or a relevant RA network, figure out what variables are needed, what would be usefull as addings and what would be all the important uncertainties involved in those and how these uncertainties could be overcome or dealt with if crucial knowledge and comprehensive understanding is lacking? Unfortenately I don't see this could be overcome with expert judgement or guidelines, because comprehensive understanding on the whole network has to be build up by the moderator itself. I see someone has to "run the show". I see expert judgement, argumentation and variable as well as the assessment "rules" as very helpfull means of making it all a lot easier for a moderator that is skilled and experianced enough (specialist or almost such) on the whole issue under assessment. Additionally, an early began discussion between the risk assessor and the risk manager so that they really understand each others is crucial for the final usability of the assessment. I'm sure we agree at least on the latter. --Anna Karjalainen 14:23, 11 June 2007 (EEST) (type: truth; paradigms: science: defence) ⇤--#2:: . A moderator of an assessment or a core group of assessors taking care of moderating an assessment may be necessary to carry out an assessment in practice. The role of the moderator(s) should anyhow be a facilitator rather than a manager. The contents of an assessment, from defining the purpose to issue framing and further to carrying out and finalizing the assessment, should be a synthesized result of the contributions of the participants (assuming broad collaboration here), not a production of the moderator(s). Good real-life examples to prove that this works are still lacking, but there are signs that indicate that the distribution of responsibility can be divided to a diverse group of people even in complex issues and it can produce good quality outputs. --Mikko 15:02, 27 June 2007 (EEST) (type: truth; paradigms: science: attack) ←--#3:: . I see in different assessments the moderator or a core group of moderators (both can be possible and should be conceivable) can be both, either a facilitator or sometimes a "manager". Ideal situation would be of cource, and probably quite attainable, that it would most often be a facilitator, but I'm trying to be also realistic and see that in certain cases the role of the moderator would be quite strong. So I think we should leave space for both possibilities. I agree that the assessment should be a synthesized result that's for sure. I DO NOT suggest that a risk assessment should be exclusively a moderator's product. What I do state is, that IN ORDER TO GUIDE THE ASSESSMENT PROCESS SUCCESFULLY THROUGH, i.e. to be either a facilitator or a "manager" the moderator needs to have (or build on) a fairly good understanding on the assessment under question. So the moderator or the core group of moderators need to be specialists or experts enough, a general level understanding sounds not enough to me. --Anna Karjalainen 09:57, 28 June 2007 (EEST) (type: truth; paradigms: science: defence) ⇤--#(number):: . We must be specific here about the role of this argument. The method itself does not say anything about the qualifications of a moderator, and that is intended. There is no "driver's license" for a moderator that is needed to be allowed to moderate. Anyone can moderate a risk assessment. But, of course, as argument #3 says, a poorly skilled moderator is unlikely to produce good assessments. However, he/she should be allowed to try. One practical reason for this is that it would be very difficult to establish widely accepted quality criteria for moderators, and control for them. --Jouni 15:22, 29 August 2007 (EEST) (type: truth; paradigms: science: attack)

Selection of indicators

How to read discussions

Fact discussion: .
Opening statement: Closing statement: Resolution not yet found. (A closing statement, when resolved, should be updated to the main page.)
Argumentation: ⇤--#1:: . The properties of a good risk assessment is described by trying to categorizise important properties distinctive for good RAs. Why is the used approach for this purpose chosen? is it ment to clarify the goodness-related properties of a RA? I ask this, because some of the presented concepts are not very easily opened up to me. For instance the informativeness: what is stated about it is very much true and understandable as such. However, to my opinion, it would not always be possible to describe variables as distributions or even estimates of distributions in a way that it actually would add up to the infomativeness. I also have difficulties in figuring out what is really ment with the calibration concept. What is correctness and what is the real value? Nature or for instance a individual body is continuously shifting from one state of situation to another, so something that could be thought as real doesn't really actually exist. The term calibration automatically in my mind refers to something much more narrow and measurable than can almost ever be reached in the context it is now used in. The selection criteria for indicators in human health or ecological risk assessments are traditionally viewed as: 1) social relevance, 2) biological and human health relevance, 3) an unambiguous operational definition, 4)accessibility to prediction and measurement, 5) susceptibility to hazardous agents and 6) unresponsiveness to any irrelevant impacts concerning the assessment scope. The last two points are related although they do not automatically exclude each others. To fulfill these requirements is not at all easy, quite far from it, so why make it sound any more complicated than it already is? The above indicator selection criteria may be viewed as conventional, but they still very much fit for use. Could/should these issues be brought up from the stated aspects in the document to your opinion? --Anna Karjalainen 15:15, 11 June 2007 (EEST) (type: truth; paradigms: science: attack) ⇤--#2:: . The properties of good risk assessments are intended for evaluating the performance of the assessments: its outputs and the process. The properties can be applied for the assessment as whole, but the properties related to effectiveness also to parts of the assessment, e.g. individual variables or indicators. The purpose of an assessment always needs to be defined. The purpose then defines what the output, i.e. the assessment product structure, should be like. This then defines how the assessment process needs (or can) be structured. The properties of good risk assessment can then be used in evaluating the performance of these issues, i.e. (1) does the output meet its purpose, (2) was the process efficient. Within the assessment process, the selection of indicators and endpoints should reflect the purpose, including the intended users and uses of the output. The abovementioned criteria for indicator selection can be used in that. --Mikko 15:02, 27 June 2007 (EEST) (type: truth; paradigms: science: attack) ←--#(number):: . There is no point in your reply that I could not actually agree on. I merely pointed out certain issues that I find tricky as concepts. One of those, calibration, was found tricky also by David Briggs just recently. I also pointed out certain issues that to my opinion should be included when figuring out the properties of good RAs. I know, they are actually included in the stated terms, but what harm would it be if the indicator selection criteria would be clarified a bit as stated? I guess it's just up to you and Jouni if you like to take into account these comments. --Anna Karjalainen 10:24, 28 June 2007 (EEST) (type: truth; paradigms: science: defence) ----#(number):: . I think the question about indicator selection criteria goes beyond the point here, because they are not discussed in the properties of good risk assessment. The chosen indicators, whichever they may be, can be evaluated using these properties, but the properties do not explicitly address the selection of indicators. I have no objections for considering the abovementioned criteria in relation the questions about indicator selection, which is considered in more detail in Guidance and methods for indicator selection and specification page. --Mikko 13:03, 28 June 2007 (EEST) (type: truth; paradigms: science: comment)

Relation of relevance to content quality and applicability

How to read discussions

Fact discussion: .
Opening statement: Closing statement: Resolution not yet found. (A closing statement, when resolved, should be updated to the main page.)
Argumentation: ⇤--#1:: . Internal relevance is defined as the internal coherence of the description, i.e. correctness and completeness of the variable relation definitions. External relevance is the correctness and completeness of what is included in assessment in relation to its purpose. The previous is objective in its nature, because its point of reference is reality, but the latter is subjective and case-specific and it refers to the contextually and situationally ever-changing needs set by the intended use of the assessment product. Should external relevance be moved to category: applicability? Or should maybe the definitions of the properties be re-considered and re-specified more accurately? --Mikko 13:31, 24 August 2007 (EEST) (type: truth; paradigms: science: attack) ←--#2:: . I agree: external relevance should be moved to applicability. --Jouni 15:08, 29 August 2007 (EEST) (type: truth; paradigms: science: defence) ⇤--#3:: . I have come to disagree with myself. After serious discussions on relations of properties and uncertainty and targeting properties to information structure it has appeared that relevance is after all a property related only to quality of content. Quality of content is about the issue itself, i.e. about the relation of the description content and the essence of the target of description (particular piece of reality). In other words the point of reference is reality. Applicability is about how well does the description suit the use purpose, i.e. how does it function within a certain context. The point of reference here is the use purpose (or use, user, ...?). Relevance of some information within some description of reality is defined in relation to reality, not use purpose of that description. Therefore, relevance can only be a sub-property of quality of content. To put it briefly: QofC is about the issue itself, applicability is about how does a description of an issue function. Relevance refers to previous. --Mikko 16:31, 15 October 2007 (EEST) (type: truth; paradigms: science: attack) ----#(4):: . This discussion is about two questions: (1) whether internal relevance belongs to the Quality of Content, and (2) whether external relevance belongs to the Quality of Content. This makes a clear discussion difficult, because attacking arguments can relate to both properties or just to a single one. I therefore have opened two seperate discussions below. --Sjuurd II 22:17, 21 February 2008 (EET) (type: truth; paradigms: science: comment)

How to read discussions

Fact discussion: .
Opening statement: Closing statement: Resolution not yet found. (A closing statement, when resolved, should be updated to the main page.)
Argumentation: Add argumentation using attack-, defend- and comment buttons ←--#(1):: . Something is internal relevant, if it relates logically to the rest of the content and it is not abundant. (Abundant paragraphs, variables, scenarios, etc. decrease the quality of the content, because they distrack the reader.) --Sjuurd II 22:48, 21 February 2008 (EET) (type: truth; paradigms: science: defence)

How to read discussions

Fact discussion: .
Opening statement: Closing statement: Resolution not yet found. (A closing statement, when resolved, should be updated to the main page.)
Argumentation: Add argumentation using attack-, defend- and comment buttons ←--#(1):: . The assessment is external relevant, if it addresses the right problems in the assessment context. For example, does an assessment for providing DM information for their decision to build a new nuclear power plant, also address and assess several alternatives to nuclear energy? If assessments address the right problems, decision makers can apply the assessment --Sjuurd II 22:48, 21 February 2008 (EET) (type: truth; paradigms: science: defence)

Usability depends on external relevance and understandability

How to read discussions

Fact discussion: .
Opening statement: Closing statement: Resolution not yet found. (A closing statement, when resolved, should be updated to the main page.)
Argumentation: ←--#(1.1):: . If the assessment output is relevant in the (decision) context, intended users can use it. --Sjuurd II 11:33, 22 February 2008 (EET) (type: truth; paradigms: science: defence) ←--#(1.2):: . AND If the assessment output is understandable for the intented users, they can use it. --Sjuurd II 11:33, 22 February 2008 (EET) (type: truth; paradigms: science: defence)