Talk:DARM DA study exercise Group 1

From Opasnet
Revision as of 13:30, 24 May 2011 by Mikko Pohjola (talk | contribs) (→‎Exercise evaluation)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Exercise evaluation

Relevance 

* internal coherence and logic between parts of the assessment and in relation to the assessment question
** scope
** definition (incl. causal diagram)
** result

←--1: . Clear purpose statement. Expected results from the analysis would clearly contribute to answering the questions. Logical and clear general-level causal diagram. Identification of many relevant variables to be taken account of in assessment. --Mikko Pohjola 14:31, 24 May 2011 (EEST) (type: truth; paradigms: science: defence)

⇤--2: . Treatment of the risk group vaccination scenario is a bit unclear. Other differentiating factors between Finland and Mexico except vaccination/no vaccination not considered. No explication of how would the "better" and "right", as expressed in the assessment questions be valued based on the analysis results. --Mikko Pohjola 14:31, 24 May 2011 (EEST) (type: truth; paradigms: science: attack)

Score: 8/10

Pertinence 

* coherence and logic between the stated intended use and the assessment question
** depends also on the identification of use, which is considered under usability

←--3: . Focuses on the essential question both for the ministry and the common citizen (~random Opasnet user) --Mikko Pohjola 14:31, 24 May 2011 (EEST) (type: truth; paradigms: science: defence)

Score: 5/5

Usability 

* executability of the assessment plan by a "skilled assessor"
* identification of the use/need the assessment intends to address and account of the requirements it poses for the assessment

⇤--4: . Can be turned into an calculative model and quantitative analysis, but requires some interpretation and additional work by a skilled assessor. --Mikko Pohjola 14:31, 24 May 2011 (EEST) (type: truth; paradigms: science: attack)

←--5: . Identification of necessary participant groups (including the primary users!) to be involved in the assessment. --Mikko Pohjola 14:31, 24 May 2011 (EEST) (type: truth; paradigms: science: defence)

Score: 4/5

Bonus points 

* e.g. innovative and clever application of the means provided for students' use or undertaken value adding extra work

←--6: . Extensive and comprehensive background description. --Mikko Pohjola 14:31, 24 May 2011 (EEST) (type: truth; paradigms: science: defence)

Score: 1/3

Total score: 18/20

General step-wise guidelines for making a DA study plan

  1. Familiarize yourself with the given case
  2. Identify and choose an interesting and/or important decision situation related to the case
  3. Formulate a relevant, explicit and answerable question relevant to the decision situation to be analyzed
    • Questions can be for example such as "what would be the impact to X if Y (had) happened?", or "what should be (have been) done to Y if we (had) wanted to maximize X?" (write this under the purpose sub-heading)
    • It is also possible to break the main question down into more than sub-questions, if seen necessary or useful
  4. Describe the (rest of the) scope for your study
    • what is the purpose of the study, e.g. how does this study relate to the case, and how does it aim to improve understanding about thecae
    • what are the boundaries of the study, e.g. in what time point(s) is the question considered, what issues are included in or excluded from consideration in the study, who participate or should be participating in the study, for what use and users the study is intended for etc.
    • what scenarios (counterfactual situations) are considered in the study, e.g. if the order of priority for swine flue vaccination had been different.
  5. Identify the indicator(s), i.e. outcome(s) of interest, that is (are) necessary to be estimated in order to answer the assessment question. Desribe this in the definition section of your plan.
    • For example, if you ask "Would the overall health impacts of swine flu and narcolepsy have been greater if no one in Finland had been given a swine flu vaccine?", you need to come up with an estimate of the actual overall health impacts, and an estimate of the overall health impacts in a counterfactual case (decision option) where no vaccination campaign was organized. (Such an indicator could be called e.g. overall health impacts from swine flu and narcolepsy. Also you would need to consider how the health impacts of narcolepsy and swine flu can be aggregated and/or compared.)
  6. Identify the decision variable(s) according to the decision you are planning to study. Define the different decision options, real or counterfactual, whose impacts you want to analyze and compare against each other. Also write this in the definition section of your plan.
  7. Consider, identify, and describe the variables that connect the decision variable(s) with the indicator(s), i.e. the causal chain. Also consider other variables that have relevant influence on the indicator(s) of your study.
  8. Describe how the decision variables, indicators, and other variables relate to each other (which influences which, which is influenced by which)
    • Describing the indicators as a causal diagram is useful for communicating your ideas (to other group members, other groups, and all others), and it usually also helps to guide the thinking process.
    • In order to turn the plan into an executable model, the variables and the relationships between them need to be defined as mathematical expressions, i.e. how is the result of a variable calculated, how do other variables results influence the result of that variable. In this case no need to open separate pages for each variable, but if by any means possible, try to describe how the indicator results can be calculated for all the decision options with the described network of variables. If you can, also calculate the indicator estimates for each decision option. Write this down in the definition section of your plan.
  9. Whether or not you manage to describe how the indicator results can be calculated, or even calculate them, describe what kind of results you would anticipate from such a study you have planned. Also consider how well does this would answer the assessment question you had formulated, how would this serve the purpose of the study you defined, and would it be useful for the intended users of the study. Write these down in the results section of your plan.
  • EXTRA: If you manage to come up with a comprehensive and explicit enough plan, you may also consider and describe the possibly suitable analyses that could be made for the model based on the plan. You may also briefly discuss what information such analysis/analyses would/could bring.


The thinking and planning does not necessarily progress in exactly this order. Follow the steps iteratively so that you can come back to previous steps and revise them as idea develops, or as is needed to keep up the coherence with other parts of the plan. If you get stuck in one step, leave it, progress to the following steps, and come back to that step later. It is not necessary to define something under every sub-heading. If you think there is nothing to write under some sub-heading, or you are unsure what to write, leave it empty for the time being and think about it later.

--Mikko Pohjola 10:08, 25 March 2011 (EET)

Hints/instructions for preparing DA study presentations

Prepare for approximately 10 minutes of presentation + 10-20 minutes of discussion.

Each group presents their own study plan e.g. as follows:

  • Focus on the essentials:
    • what is the question addressed in the study? → Scope
    • how is the question answered by means of a decision analysis? → Definition
      • (most effort should probably be spent in describing/explaining this)
  • what kind of results does/should such an analysis produce? → Result
    • how do/would the results answer the question? → Result
  • Tie the above into the broader context of the swine flu case, e.g.:
    • how essential is the chosen decision/question in relation to the whole case?
    • what and how significant practical implications would/could the study have?
    • how applicable would the results of such a study be in its intended use?
  • Discuss the main challenges in a) making the plan, b) executing the plan

If you like, you can prepare some (not many!) slides, but the presentation can well be illustrated by showing the exercise page from Opasnet. In any case note that:

  • a causal diagram is an effective way to communicate the DA study structure
  • If possible, also present how the impacts of different decision options on the indicator result(s) can be calculated
  • brief and clear (even if somewhat incomplete) is usually better than lengthy and fuzzy
  • (the discussion points regarding the relationship between the study and the broader context and the challenges can be written down e.g. after the Results section on the exercise page.)
  • The evaluation of the exercise will be done based on what is presented on the group's exercise page in Opasnet

--Mikko Pohjola 14:26, 25 March 2011 (EET)

Retrieved from "https://en.opasnet.org/index.php?title=Talk:DARM_DA_study_exercise_Group_1&oldid=20980"