Assessments - science-based decision support

From Opasnet
Jump to navigation Jump to search



Assessments - science-based decision support is a lecture about assessments as processes and their products that on the one hand are determined by the practical needs of societal decision making (in particular policy making), and on the other hand are bounded by the scientific quest for truth. The lecture, consisting of two parts, discusses how these sometimes apparently conflicting goals influence the making and outcomes of assessments.

Scope

Purpose: To describe how the practical needs of societal decision making are taken account of in assessment (design and execution), and how these needs are addressed by means of science.

Intended audience: Researchers (especially at doctoral student level) in any field of science (mainly natural, not social scientists).

Duration: 1.5 h + 1.5 h

Definition

The lecture is divided into two parts. First part discusses what assessments are about and what purpose do they serve. The second part then discusses what can be done in assessments and how, in order to adequately serve the purpose. The second part is built on the assumption that in between parts I and II, the recipient of this lecture becomes acquainted with the concepts variable and collaboration, in the way they are defined in the context of open assessment.

In order to fully understand this lecture it is recommended to acquaint oneself also with the following lectures:

Objectives, part I:

  • learn the basics of an assessment: question/info need, and the three-part structure: decision - factors - outcome.
  • learn how an information need is transformed into a research question.
  • learn the importance and implications of boundaries in defining an object (assessment, variable)
  • learn the roles of participants and intended users.
  • learn to participate in discussions about a new assessment (Tapas) and its framing.

Objectives, part II:

  • become familiar with the information structure of an Opasnet assessment object
  • learn how solutions to assessment problems are found (sought for)
  • learn how the assessment solution addresses the need and how need influences the solution
  • learn how an assessment consists of a network of individual variables (independent, given causal relations)
  • learn how the information content of an assessment can be analyzed, organized and interpreted for producing the results

Result

Part I: practical need determines the assessment problem

See presentation: File:Assessments -science-based decision support.ppt

Introduction

  • Assessment is a business of creating understanding about reality in relation to a specific need
    • collection, synthesis, interpretation, organization, communication, use of information
    • driven by practical needs, constrained by the scientific quest for truth (yet also facilitated with the means and methods of science)
  • Different views to assessment:
    • practical need vs. science
    • process - product - use

Example: TAPAS - Transportation, air pollution and physical activities: An integrated health risk assessment programme of climate change and urban policies.

  • urban policies (interventions)
  • active transportation
  • climate change
  • other health related outcomes
  • risks: air pollution, UV, crime, accidents
  • benefits: physical activity, social interaction, mental health
  • summarizing effects: DALYs
  • Assessment
    • active transport policy decisions
    • Indicators: GHG emissions, ambient air quality, mental health/quality of life, attributable chronic/acute disese, net health (DALYs)
  • what need is addressed?
  • whose need/concern is it?
  • who is interested in/affected by the issue?
  • who is responsible for the state of affairs or taking action?
  • who can take action?
  • what actions?
  • is someone affected by possible actions about them?
  • what needs to be known in order to take action (right action)?
  • are there conflicts of interest or differences of perception regarding the needs/concerns or actions about them?
  • whose understanding about the issue is most crucial to be increased (e.g. scientists, policy-makers, industrial managers, specific stakeholder groups, common citizens)?
  • what are the issues of interest that we need to know more about?

→ determination of assessment purpose, boundaries and intended users

  • can the assessment problem be formulated into a form of a question or a set of questions
  • TAPAS diagram

Practical need → assessment problem:

  • information gap drives the assessment
    • need to know, but required knowledge/information is missing
    • need to make decision (action/no action, what action) → need to choose among decision options → need to know available decision options → need to know preferability of decision options
  • Various decision makers: e.g. political leaders, industrial managers, individuals
  • Various possible decisions/actions: e.g. policy, investment, consumption
  • → purpose of assessment is to satisfy the information gap of intended users of the assessment
    • What is required in order to know what should be done in order to improve the state of issue of interest?
  • → → the information gap needs to be indicated/identified, analyzed and understood
  • → → → the information gap determines the boundaries for what needs to be assessed
  • purpose, users and boundaries = assessment problem ( or = scope, see Talk:Assessment)
    • what is needed to know, why, who will use the knowledge (why will they acquire the knowledge)?
  • purpose & users give practical requirements for the assessment
  • boundaries define the part of reality which to describe

Looking for solution:

  • What needs to done in order to address the problem?
  • Decision → factor → outcome
  • Indicators: issues of specific interest (can be of any type: outcomes, factors or decisions - typically outcomes)
  • Assessment is about finding solutions to the assessment problem
    • probably also at least as much it is finding out what is not known in order to solve the problem (consider knowledge management vs. ignorance management)
    • → assessments are done according to need, not according to e.g. coincidental data availability


Part II: providing solutions to the assessment problem

See presentations: File:Assessments -science-based decision support part II.ppt | File:Bayesian belief network basics.ppt

Assessments as objects in Opasnet:

  • information structure of assessment objects
    • scope - need → problem
      • purpose, boundaries, users, scenarios, participants
    • definition - means of providing a tentative solution to the problem
      • decision variables, indicators, other variables, indices, analyses
    • result - the tentative solution to the problem
      • results (of indicators and analyses), conclusions
  • assessment is a collection of variables, analyses about them, and their communication
    • decisions, factors and outcomes in a causal network
    • indices: operationalizations of dimensions considered
    • analyses (examples)
      • decision optimization (which decision results in best outcome)
      • other conditioning (intentional deviations from best estimate for any variable result)
        • scenarios are conditionings that are particularly important for the use purpose (thus defined in scope)
      • value of information analysis (where is more understanding most useful?)
      • importance and sensitivity analysis (what factors / change in factors most affect the outcome?)
    • appraisal (by what means are value judgements incorporated in the assessment)
  • assessments as answers to practical questions by means of science
    • the real-world phenomena defined in boundaries need to be described as truthlikely as possible to the extent and detail as relevant according to the use purpose
    • the information in the description should be organized and communicated according to the determined scope
    • practical need provides the questions, science constrains and facilitates how they answered, practical need again determines how the answered are to be conveyed/converged to use

Bayesian belief networks (Patrycja)

Decision optimization, conditioning, scenarios, value of information (Jouni)


Exercises:

Exercise topic Task
Meeting the needs of intended use (primary: WWViews, COP-15) Consider the intended use, its needs and the scope of the assessment e.g. in light of the following questions:
  • Is the assessment addressing right issues and asking the right questions (i.e. is the scope sufficiently defined)?
  • Do you think it has a potential to influence the intended use process (assuming that the assessment succeeds in providing answers its questions adequately)?
  • How should the information be wrapped, organized and communicated to the intended users in order to realize the potential effect (again, assuming that the truth-value of the content is adequate)?
  • What aspects of the information content should be emphasized in communication?

Document your statements, arguments and comments on the discussion page, and make amendments directly to the assessment page as necessary. Also present your findings to other groups including a brief explanation of the identified need and how it is addressed.

Answering the assessment questions - variable network This task is about developing the definition of the case assessment in terms of the variables belonging to the assessment. Explore the list of variables, their types (decision, indicator, other) and the causal diagram representation of the assessment as well as the Analytica model. Given the scope of the assessment, consider e.g. following questions:
  • What variables (descriptions of real-world phenomena) are required to be included in the assessment in order to adequately provide answer the questions (scope) set for the assessment (assuming that the variables are well defined)?
  • Is the given list of variables and the causal diagram/model sufficient in order to answer the questions?
  • Are the variable list and causal diagram/model consistent with each other (variables vs. model nodes)?

Document statements, arguments and comments on the discussion page, and make amendments directly to the assessment page as necessary. Also present your findings to other groups including a brief explanation of the causal network of variables and its relation to the defined scope of the assessment.

Answering the assessment questions - BBN analysis Acquaint yourself with Bayesian belief networks and consider how they can be used in assessments. In particular look into:
  • What is conditioning?
  • What are the similarities and differences between causal diagrams and Bayesian nets?
  • How can Bayesian nets be used for optimizing?

Analyze a simple Uninet model, provided to you by the workshop organizers, covering a part of the case assessment. By conditioning different variables try to find the optimal decision option. You can also update the model with new variable results found in Opasnet and repeat the analysis. You can document statements, arguments and comments on, or make amendments to, related pages in the Opasnet if relevant. Present your findings to other groups including a brief introductory explanation of how Bayesian nets can be used in the case assessment.

Answering the assessment questions - Value of information (VOI) analysis Acquaint yourself with the Value of information method (VOI). In particular look into:
  • What does VOI measure?
  • What is VOI used for?
  • What information must one have (necessary conditions) in order to calculate VOI?

Analyze a simple Analytica model, provided to you by the workshop organizers, covering a part of the case assessment. Calculate VOI for different variables in the model. You can also update the model with new variable results found in Opasnet and repeat the analysis. Present your findings to other groups including a brief introductory explanation of how VOI can be used in the case assessment.

Scenarios Acquaint yourself with the way the concept Scenario is defined in the context of open assessment. Then consider the scenarios set for the case assessment e.g. in light of the following questions:
  • What is the conceptual difference between the base-case and the scenarios?
  • In what ways do the decision scenarios (plausible EU-ETS alterations) differ from the IPCC "scenarios", referred to as "future pathways"?
  • In what ways do the decision scenarios (plausible EU-ETS alterations) from the other decision options assessed?
  • Are the set EU-ETS scenarios (including the base-case) well-chosen and well-defined?
  • How should the scenario set in the case assessment be improved?

Document statements, arguments and comments on the discussion page, and present your findings to other groups.

Retrieved from "https://en.opasnet.org/index.php?title=Assessments_-_science-based_decision_support&oldid=8582"