Decision analysis and risk management 2017/Homework

Follow-up table of the homeworks. Green: work is acceptable. White: there is still work to do. Red: work is overdue. Deadline dates and maximum points available are mentioned in the column headings.

User	HW 1: Open assessment 16 Jun 1 (a) + 1 (b,c,d) points	HW 2: Basic skills of Opasnet 16 Jun 1 point	HW 3: Basic concepts of open assessment and co-creation 16 Jun 1 (a) + 1 (b) points	HW 4: Draft of an assessment plan 16 Jun 2 points	HW 5: Climate policy decisions and actions 16 Jun 2 points	HW 6: Collaboration in climate policy assessment 16 Jun 2 points	HW 7: Structured discussion 16 Jun 2 points	HW 8: Developing a variable page 16 Jun 2 points	HW 9: Evaluation of assessment 16 Jun 2 points	Seminars	Total points (max 17) and final score (max 5)
Amr Ebrahim	HW1a OK Add comments about HW1b-d!		1+1	2	2	2	2			Yes
Edem Agbenowu	1+1		1+1	1.5	1.5	1.5	2			Yes
Ehab Mustafa	1+1		1+1	2	2	2	2			Yes
Jin Qiwen	1+1		not finished
Kaisu Lukkarinen	1+1
Kingsley Aliche
Margaret Arogunyo	1+1		1+1	1			add more arguments			Yes
Nabin Subedi
Noora Rantanen
Tamara Gajst	1+1	HW2	1 + 1	HW4	2	HW6	2	HW8	HW9	Yes
Tarikul Islam
Tine Bizjak	1+1	HW2	1+1	HW4	2	HW6	2	HW8	HW9	Yes
Zahra Shirani	1+1		1+1

Please read the homework assignments carefully and follow the instructions. If there is something unclear, please ask the course organizers (or fellow students) to explain and clarify! NOTE: Write all your homework answers on your own user page.

Also add links to your homework answers in the table above. The evaluation of the homework exercises will be based on the answers found by following the links in the table. Students themselves are responsible for having the correct, complete and up-to-date links to homework answers. if you need help in adding the links to your homework answers to the table, please ask the course organizers (or fellow students) for advice. A convenient way to get help is to come to the exercise sessions.

Please note

• If your Homework says "OK" it means that the given homework is graded as "pass", i.e. at least 1 point. If you want to get better points, you should check and answer lecturers´ comments regarding that homework.
• If there is no "OK" sign, you must revise your work according to the comments in order to make it acceptable.
• You must write homework answers done in groups/pairs to only one place.
• Add link to answers on your own user page if it is located on someone else's user page (do not copy the text on your own user page).

Homework 1a: Open policy practice

Estimated working time: 5 hours.

Purpose: To familiarise yourself to the basic ideas of open policy practice.

Read pages Open policy practice, Knowledge crystal, and Open assessment and browse Assessments are to change the world and Shared information objects in policy support and provide brief answers to three (3) questions from the following question list. You may also want to search from Opasnet. You are free to choose which questions to answer. Write your answers on your own Opasnet user page. Instructions on creating a user account and editing your own user page will be given on first lecture. In case of difficulties in wiki editing, write your answers on a separate document and copy them to your user page later. The questions and answers will be discussed on the second lecture (23 March). A sufficient length for each answers is a few sentences or bullet points. Please do not write lengthy essays, but instead try to identify and briefly describe the main points relevant in each question. The idea of this homework is not to find the right or correct answers, but instead to introduce the conceptual basis of this course to the students.

Questions:

1. What is the main purpose of environmental health assessment?
2. What is shared understanding?
3. What are the main differences between regulatory and academic assessment approaches? Give examples of each.
4. What are co-creation skills?
5. What are the main differences between open assessment and most other assessment approaches?
6. What is benefit-risk assessment?
7. What is open assessment?
8. What different purposes are there for participation in assessment and/or decision making?
9. What are the dimensions of openness?
10. What relevant stakeholder roles are there in environmental health assessment and related decision making
11. What is effectiveness' in the context of environmental health assessment and related decision making?
12. What is the trialogical approach to knowledge creation and learning?
13. What is decision support?
14. What is a pragmatic knowledge service?
15. What is collaboration?
16. What are the properties of good assessment?
17. What is the role of modelling in assessment and policy making?
18. What parts does the open policy practice consist of?
19. What does it mean that the results of assessments can be considered shared information objects?

Homework 1b: Learn the terms in Quizlet

Estimated working time: 5 hours

Purpose: To learn the terms and concepts of open policy practice and see how they are related

Join Quizlet and practice with the sets of terms to learn the concepts:

• Principles of open policy practice
• Properties of good assessment
• Glossary for open policy practice, also the Finnish terms if you can Finnish.
• Categories of interaction
• Dimensions of openness

Homework 1c: Introduction to critical thinking

Estimated working time: 5 hours

Purpose: To learn the basics of critical thinking and argumentation

Join Khan Academy and follow the course of Critical thinking. If you already know this topic well, just do the exercises.

Homework 1d: Introduction to probabilities

Estimated working time: 0.5 - 5 hours

Purpose: To ensure that the basics of probability theory are clear (We assume that the basic statistics have been taught to students participating in this class.)

Go through the contents of the Khan Academy courses probability and random variables and make sure that you refresh your memory on this. Do the exercises, and look at the videos if needed.

There are also some other, more basic material that may be useful: probabilities and combinatorics.

Homework 2: Basic skills of open policy practice

Estimated working time: 5 hours

Basic skills: Mark "yes" when you know how to do this and put a link to the page where you have used the skill. Use these skills as parts of other homeworks.

User	Create a page and type	Upload a file and link it	Use headings, lists, bold, italic	Use internal and external links and templates	Use references	Create a prettytable	Upload data by t2b table and Opasnet Base Uploader	Organise a discussion
Amr Ebrahim	Yes		Yes		Yes			Yes
Edem Agbenowu	Yes	Yes	Yes		Yes	Yes		Yes
Ehab Mustafa	Yes	Yes	Yes	template missing	Yes	Yes		Yes
Jin Qiwen	Yes		Yes	template missing
Kaisu Lukkarinen	Yes
Kingsley Aliche
Margaret Arogunyo	Yes		Yes		Yes	Yes		Yes
Nabin Subedi
Noora Rantanen
Tamara Gajst	Yes	Yes - uploaded the Seminar file	Yes	Yes	Yes	Yes - HW9		Yes
Tarikul Islam	Yes
Tine Bizjak	Yes	Yes	Yes	yes	Yes	yes		Yes
Zahra Shirani	Yes		Yes

Homework 3: Basic concepts of open assessment and co-creation

Estimated working time: 2+5 hours.

Task A: Read a) homeworks 1 and 2, b) Glossary#Terms in open policy practice and c) the introductory pages listed below. If you know Finnish, the Yhtäköyttä report contains a lot of the same material in a more organised way. After reading, write two questions that you think needs clarification. Write the questions on your own user page. The questions will be answered during the next lecture.

Task B: Read the material in a zip file about co-creation, decision support models, and facilitation. It contains the following material (numbers refer to reference numbers in Yhtäköyttä report.

• 11: von Winterfeldt, D (2013). Bridging the gap between science and decision making. PNAS 110:3:14055-14061. [11]
• 45: Aitamurto, T, Landemore, H. (2015) Five design principles for crowdsourced policymaking: Assessing the case of crowdsourced off-road traffic law in Finland. Journal of Social Media for Organizations. 2:1:1-19.
• 46: Force11. FAIR data principles. [41] viitattu 22.2.2017.
• 47: Prahalad, CK, Ramaswamy, V (2004). Co-creation experiences: The next practice in value creation. Journal of Interactive Marketing 18:3:5-14. doi:10.1002/dir.20015 [42]
• 48: Mauser, W, Klepper, G, Rice, M, Schmalzbauer, BS, Hackmann, H, Leemans, R, Current, HM (2013). Transdisciplinary global change research: the co-creation of knowledge for sustainability. Opinion in Environmental Sustainability 5:3–4:420–431. doi:10.1016/j.cosust.2013.07.001
• 49: Franco, LA, Montibeller, G (2010). Facilitated modelling in operational research. European Journal of Operational Research 205:3:489–500. [43]
• 53: Kolbert, E. (2017) Why facts don't change our minds. The New Yorker, 27.2.2017. [47] viitattu 22.2.2017.

Write a short assay on your user page about co-creation in decision support. What is co-creation? What advantage does it bring compared with more traditional decision support processes? What is the role of a facilitator, and what skills do they need?

Homework 4: Draft of an assessment plan

Note! Homework 4 answers will be used as materials in homework 10.

Estimated working time: 8 hours

Task: With your pair, draft a plan of an assessment about the topic agreed on during the lecture. See the correct structure from Assessment. You may copy the structure directly from Template:Assessment structure. Write the draft assessment on either your or your partner's user page (and put a link to it on the other's user page). Choose your specific topic within the broader area of climate change policies in a city. You can consider mitigation (how to reduce greenhouse gas emissions), adaptation (how to prepare for changes caused by climate change) or both. You may choose a specific city on your assessment, or look at some aspect in cities in general.

You are expected to make plans about a good assessment related a topic of your choice (preferably related to climate change policies in cities). Fill in the subheadings in Scope and make plans about the Rationale: what variables or assessment parts you would need to be able to answer the question asked? However, you are NOT expected to come up with results or conclusions (although you can describe what kind of results you might get if the assessment was actually performed).

Homework 5: Climate policy decisions and actions

Estimated working time: 6 hours

Consider that you are given an assignment to assess the direct or indirect health impacts caused by a climate (adaptation) strategy or program. One of the first things in getting started with the assessment is to discuss, identify and explicate the decisions and options related to the assessment problem. In pairs choose one climate (adaptation) strategy/program from the material list below and identify and write out answers to the following questions based on the material. Use your own reasoning and knowledge or other sources (e.g. Google search) as complementary where the material is incomplete or inconclusive.

Write your answers on either group member's user page (other member adds a link to the answers on his/her user page).

Questions:

• What are the aims/goals of the strategy/program, i.e. what are the desired impacts and outcomes striven for?
  • Who are those that benefit if the aims/goals of the strategy/program are reached?
• What are the actions that are needed/intended to take in order to progress towards the aims/goals?
  • Who are those that actually realize these actions?
• What are the decisions that are needed to make in order to enable/promote the actions?
  • Who are the decision makers?
• What direct or indirect health impacts, positive or negative, these decisions and actions (may) have?
  • Where and how do these impacts take place, who are those that face these health impacts in practice?The community,the citizens,
  • Are the health impacts big or small in relation to other impacts (e.g. economical, social, climate, other environmental, ...)?
  • Do the intended policies result in win-win, win-lose, lose-win, or lose-lose situations with regard to health and other impacts?
• Formulate a plausible and meaningful specific assessment question that takes account of (some of) the aspects considered in above questions.
• Extra question: In what ways your answers do or do not represent "shared understanding"? (The climate program/strategy can be considered a compilation of contributions by many experts and attempting to reflect the views and needs of different decision makers and stakeholders).

Materials:

• Klimaprogramm-Bayern-2020 klimaprogramm2020_en_05_2009_ba Bavarian Climate Programme 2020 [1](margaret)
• Summary_Ludwigsburg_LEAP_Final_EN Integrated Climate Protection and Energy Strategy for Ludwigsburg
• National_Climate_Change_Strategy_of_Hungary_2008 National Climate Change Strategy of Hungary 2008 (mitigation, adaptation)
• YTV_climate_strategy_2030 Climate Strategy 2030 of Helsinki Metropolitan Area
• 11_2012_Helsinki_Metropolitan_Area_Climate_Change_Adaptation_Strategy Climate Adaptation Strategy of Helsinki Metropolitan Area
• NCCS-2012-Publication National Climate Change Strategy of Singapore 2012 (mitigation, adaptation)
• 2011_09_06 KORT_Jaarversl_RCI_over10_EN DEFINITIEF Rotterdam Climate Initiative RCI (Amr, Ehab)
• RCP_ENG_def Rotterdam Climate Proof Adaptation Programme 2010 (Amr, Ehab)
• Cover+Adaptation+to+climate+change+in+Switzerland Adaptation to climate change in Switzerland (Tamara, Tine)
• Ireland_at_Risk_2 Ireland at Risk. Critical Infrastructure Adaptation for Climate Changeedem
• klimatilpasningsstrategi_uk_web Danish Strategy for Adaptation to a Changing Climate
• Climate adaptation materials in different European languages
• op_fi:Helsingin ilmastonmuutos_-tiekartta

Homework 6: Collaboration in climate policy assessment

Estimated working time: 6 hours

This exercise continues from homework 5. With the same pair, using the same material, and building on your homework 5 answers, identify and write out your answers to the following questions. Narrow your scrutiny down to e.g. one or two decisions/actions/goals if needed. Base your answers on the climate program/strategy paper you have chosen, but also apply your own reasoning, other DARM course materials etc., particularly on the second set of questions.

Write your answers on either group member's user page (other member adds a link to the answers on his/her user page).

Homework 6, part A: Questions about identifying roles and participation:

• Who are the relevant participants of the assessment?
• What roles the different participants (may) take in the assessment?
• What kind of relevant knowledge they (may) have regarding the assessment?
• What needs and aims do they represent in the assessment?

Homework 6, part B: Consider also the following questions about facilitating collaboration:

• How could the relevant participants be involved in the assessment in an effective way?
• How can the quality of an assessment be assured if anyone can participate?
• How can you prevent malevolent contributions where the purpose is to vandalise the process?
• How can you make the outcome converge to a conclusion, because all issues are uncertain and controversial?
• How can you ensure that the outcomes are useful for the users?

Homework 7: Structured discussion

Estimated working time: 20 hours

Main message:

Question: What are the evaluation criteria for structured discussion (homework 7)? Answer: Evaluation of arguments: Each argument is evaluated either A (very good), B (good), or C (irrelevant). When you have written at least one A argument and at least three B arguments, you get grade 2. When you have written at least four B arguments you get grade 1.5. If you have written at least two B arguments you get grade 1. Argument with C is a slight dis-merit and may affect borderline situations. B argument is the default. A arguments differ from B arguments by having an important, unique aspect, good referencing, and/or clever use of hierarchy.

The objective of this homework is to learn to organise existing written material into a structured discussion with a main statement and related arguments. In addition, students should learn to develop and use own arguments within a structured discussion. For examples, see

• Talk:Environmental impact assessment directive
• Talk:Assessment of the health impacts of H1N1 vaccination
• op_fi:Keskustelu:Pneumokokkirokotteen turvallisuus
• Talk:Climate change policies in Helsinki

Your task is to initiate and participate in structured discussions on page Talk:Congestion charge according to the instructions on page Discussion. Come up with one original statement for a discussion based your own ideas, by talking to Helsinki city representatives (jari.viinanen(at)hel.fi, mira.jarkko(at)hel.fi) or the material below.

Articles in Wikipedia

• Congestion pricing
• London congestion charge
• Ruuhkamaksu

Studies about congestion charges in Helsinki (and related discussion) ----#: . If you know Finnish, you should first look at these texts to make their arguments available to others in English. --Jouni (talk) 10:43, 10 April 2017 (UTC) (type: truth; paradigms: science: comment)

• Helsingin kaupunki. Ruuhkamaksut tehokkain keino parantaa Helsingin ilmanlaatua nopeasti. (12.01.2017) [2]
• HSL (11.2.2016): Tiemaksut varmistaisivat Helsingin seudun kestävän kasvun [3]
  • Helsingin Sanomat [4]
  • Helsingin Uutiset [5]
  • Kauppalehti-blogi [6]
• LVM. (2011) Helsingin seudun ruuhkamaksu. Jatkoselvitys. Liikenne- ja viestintäministeriön julkaisuja 5/2011. [7]
• LVM. (2007). Joukkoliikenteen houkuttelevuuden ja käytön lisääminen eri liikkujaryhmissä. Liikenne- ja viestintäministeriön julkaisuja 63/2007. [8]
  • Talouselämä-uutiskommentti [9]
• LVM. Tienkäyttömaksujärjestelmät. Esiselvitys. Liikenne- ja viestintäministeriön julkaisuja 17/2006. [10]

Scientific articles about congestion charge and health

• doi:10.1126/science.aaf3420
• doi:10.1016/j.jth.2015.08.002
• doi:10.1080/09640568.2014.912615
• doi:10.1016/j.tra.2015.03.004
• doi:10.1016/j.tranpol.2015.01.002
• doi:10.1016/j.atmosenv.2008.09.015
• doi:10.1080/13547500902965252
• doi:10.5172/impp.453.10.2-3.269
• doi:10.1016/j.amepre.2006.03.030
• doi:10.1136/jech.2003.012385

Build the content to your discussion based on different materials you can find from the climate reports in homework 5, from the Internet, and from the city representatives. Note that you can and should also participate in discussions launched by other students.

As facilitators, you should pay attention to get as many different opinions documented as possible. So, jump into a role of a stakeholder and try to think what he/she would say. Possible roles include:

• A national authority giving environmental permissions.
• A taxi company.
• A department store inside or outside a planned congestion charge zone.
• A nature conservationist.
• A local politician interested in both nature and local economy.
• A citizen.

Note that you are allowed to:

• Contradict your own arguments.
• Update and improve statements if they are too vague or poorly written. However, be careful not to push the existing argumentation out of context. Instead of making large changes to a statement, start a new discussion with your new statement.
• Add your signature to other people's arguments if you agree with them. Note that the first name is assumed to be the original author, so don't put your name first.
• Clarify other people's arguments, if you do it carefully and do not change the meaning.
• Copy arguments from one discussion to another, if they are relevant. But instead of copying large blocks, make references to the other discussion instead.

Homework 8: Participate in an assessment

Estimated working time: 44 hours

• Everyone participates in making an open assessment. The topic is Congestion charge to reduce car traffic. This relates to a larger topic Climate change policies in Helsinki. The topic shares some similarities with Climate change policies and health in Kuopio and Climate change policies in Basel, although congestion charge was not discussed in those assessments.
• Based on structured discussions in Homework 7, every pair selects a sub-topic (or research question) and develops at least one variable page that asks that question and answers it. Before starting a variable page, suggestionss about relevant questions should be discussed on the assessment page (Congestion charge). This is because the relevance of a specific question depends on other questions asked, and some issues link causally together nicely when the questions are formulated in a coherent manner.
• The task is to produce a shared understanding of the assessment topic, i.e. congestion charges in Helsinki. The shared understanding is produced using the structure of shared understanding. In practice:
  • Each main subtopic is described on its own page in Opasnet.
  • Find enough relevant information for a plausible answer, synthesise the information on the page, and quantitate it.
  • The connections of the pages and other items are described on three tables: a) Items, b) Relations, and c) Evaluations. Page Voting age has an example about how to use the tables.
• The data should be documented well enough to convince a critical reader that this is a good answer to the question.
• Other examples of shared understanding (op_fi:Jaetun ymmärryksen menetelmä) in Finnish:
  • op_fi:Keskipitkän aikavälin ilmastopolitiikan suunnitelma,
  • op_fi:Energiarenessanssi,
  • op_fi:Pietarsaaren rokotuskeskustelu

Homework 9: Evaluation of assessment

Estimated working time: 8 hours.

In this exercise you are asked to look into and evaluate one homework 4 draft assessment (other than your own) and one real-life assessment performed in Opasnet (listed below).

• Helsinki energy decision 2015
• Climate change policies and health in Kuopio
• Climate change policies in Basel
• Pneumococcal vaccine
• Water guide
• Fukushima nuclear accident
• Risk assessment on Hämeenkyrö municipal solid waste incinerator
• Comparative risk assessment of dioxin and fine particles
• Benefit-risk assessment of fish consumption for Beneris
• Emission assessment of small-scale energy production in the Helsinki metropolitan area
• Assessment of building policies' effect on dampness and asthma in Europe
• Assessment of the health impacts of H1N1 vaccination
• Benefit-risk assessment of cinnamon
• INTEGRATED ENVIRONMENTAL AND HEALTH IMPACT ASSESSMENT OF WASTE MANAGEMENT IN LAZIO (ITALY)
• The health risks and benefits of cycling in urban environments compared with car use: health impact assessment study
• Environmental impact assessment directive
• Assessment on impacts of emission trading on city-level (ET-CL)
• Gasbus - health impacts of Helsinki bus traffic
• Biofuel assessments
• Assessments in Finnish:
  • op_fi:Pahtavaaran kaivos
  • op_fi:Silakan hyöty-riskiarvio
  • op_fi:Rauman sataman laajennuksen vaikutus terveyteen
  • op_fi:Talvivaaran kaivoksen terveysvaikutukset

The work is based on instructions and tables on page Open policy practice#Evaluation and management. Find the assessments by the two users below you on the user/homework list on top of this page (the last on the list shall pick the first two users on the list and the second last on the list shall pick the last and the first user).

This exercise is intended to be done individually. However, co-operation between students is recommended.

First characterize the draft assessments according to the Knowledge-policy interaction and Dimensions of openness frameworks. The things to consider in the characterization are listed and explained in the tables in Open policy practice#Evaluation and management.

In order to identify the last point in framework for characterising settings (Table 3.), mode of interaction that the draft assessment builds on, characterize the dimensions of openness in the assessment explained in Table 4. (Open policy practice#Dimensions of openness). The example categories for interaction mentioned in Table 3 are explained in Table 5 (Open policy practice#Categories of interaction).

Second, evaluate the assessment drafts according to the (slightly modified) Open policy practice#Properties of good decision support framework. Base your evaluation on the characterization you have made. The things to consider in the evaluation are listed and explained in Table 2. For each attribute (i.e. an aspect to consider) give a numerical evaluation on a 1-5 scale (1 = poor, 5 = excellent). Also briefly write down your reasoning for each numerical evaluation. If something seems completely missing or not possible to evaluate, the numerical evaluation is 0 (also write down your reasoning why the particular aspect of the draft assessment deserves an evaluation of 0).

Evaluation of assessments is not only something to be done after an assessment has been completed. Instead, evaluation should be seen as a means to guide the making of assessments towards their aims while they are still happening. Therefore, the third task of this exercise is to formulate suggestions for developing/improving the draft assessment. Write your suggestions as comments/arguments to the user pages where the draft assessment descriptions are. Also point out where the information in the draft assessment is/was missing or insufficient for characterization or evaluation.

Homework 4 answers will be used as materials in this exercise. It is recommended that you attempt to do this exercise only starting on the deadline of Homework 4.

Links to some examples of using the above mentioned evaluation frameworks:

• Openness in participation, assessment, and policy making upon issues of environment and environmental health: a review of literature and recent project results (Dimensions of openness)
• Assessment of impacts to environment and health in influencing manufacturing and public policy (Knowledge-policy interaction.)
• Evaluating effectiveness of open assessments on alternative biofuel sources (Dimensions of openness and Properties of good assessment.)
• op_fi:Puijon metsien käyttösuunnitelman päätöksenteko (All methods. In Finnish)

Extra homework (not evaluated): Structure of pages and objects and R code

----#: . In the next course, we need a homework where there is one variable and some related (pre-known) data. The task is to go through the data, evaluate its applicability, transform it into a format that better answers the question, and discuss different interpretations. The purpose is to produce a probability distribution as an answer to the question. This variable might be a part of the training assessment. --Jouni (talk) 14:47, 19 May 2015 (UTC) (type: truth; paradigms: science: comment)

Estimated working time: 8 hours

The objective of this homework is that you learn to see what different parts of a page are and how they are related to each other and to other pages. Especially, an objective is to understand the role of R code in this system. You should learn to identify key things from a code and understand their use and connections to other parts of an assessment. These skill are then needed in Homework 9 when we actually perform an assessment.

With your pair, select and reserve three pages (by adding your usernames beside the page link) from the list below. At least two of them have to contain t2b tables and R code. Go through the content by doing all of the key tasks below, if possible. Also look at the additional questions and answer at least some of them. Write your answers to the page by using the comment, defend (when things are OK), and attack (when things are not OK) buttons. If you can, improve the content or suggest tasks for improvement.

In addition, select three other pages from the list such that another pair has already done the work. Read the content and their comments, and agree or disagree with them. Try to improve the content further.

Key tasks

• Check that the page has a correct page type and change when needed. Check that the page has all subheadings that belong to the page type. Add, if missing.
• Categorise the page to relevant categories.
• Organise the content into the right subheadings. Especially, look what is Data and what is Answer.
• Check and update the Dependencies. Also check that the Answers in dependency pages are coherent with this page.
• Make rcodes that a) creates the ovariable (under Calculations) and b) gets the latest ovariable and prints basic results (under Answer).
• Test any existing code and report its functionalities on the page.
• Write or update a summary (one paragraph in the very beginning explaining the main points of the text) on the page. If the content is too unclear to write a good summary, write down clarification questions to the moderator of that page.
• If you have problems with any previous steps, describe them on the relevant point on the page.

Additional questions

• Does the page have a correct page type?
• Does the page have a question? Is it clear and unambiguous?
• Does the page have an answer to the question? Does it actually give an answer to what is asked?
• With variables, is the answer given as a link to a model run with calculated results? If yes,
  • Does the model run have a clear result table?
  • Does the model run have a clear result graph?
  • Is it clear where the code that was used to run the results is?
• In method pages: based on the guidance in the answer, is it possible to actually use the method in an assessment?
• In method pages: What data is required to be able to use the method? Are the requirements listed under "Inputs"?
• Are there data on the page that is needed to answer the question? Are it in machine-readable format (i.e., in t2b table or directly stored in the database)?
  • Are the data under Rationale/Data subheading, (or in methods under Rationale/Inputs)?
• Is there data or text that is NOT needed to justify the answer? Would that data be in better place on another page with a different question? What would that question be?
• If the data is needed but is not used in the Answer, update it or suggest tasks to update it.
• Are there external variables whose values need to be known to be able to estimate this object? If yes,
  • Are these listed under Rationale/Dependencies?
  • Are there equations (as text) for calculating this object based on the dependencies under Rationale/Formula (or Rationale/Calculations)
• Is there an R code that implements the object?
  • With variables, is the code under Rationale/Calculations?
  • With methods, is there a code under Rationale/Calculations that defines the method object?
  • With methods, is there a code under Answer that describes how the method object is used??
  • If there are dependencies and formula, does the code take them in to produce an ovariable?
  • If there are data, does the code take them in to produce an ovariable?
  • When you run the code, does it crash (i.e. produce an error message) before completion? When and why (use show code and show messages and errors to understand what's going on)?
  • Are there several different codes on the page? Are their purposes clear?
  • Does the page use other pages (objects) in calculations? Are these connections listed explicitly as links under the R code?
• Does the page have an evaluation (edistymisluokitus) in either a separate box in the beginning, or in the metadata box?
• Does the page have other subheadings (See also, References, Related files, Keywords)?
  • Are there links to other related pages? Are relevant links missing?
• Is the page categorised to relevant categories?
• With encyclopedia pages: is the content detailed enough so that one or more variables or methods could be made based on it? Does such page(s) exist? Are these pages linked to each other?
• Does the page explain its links to other pages? Is it clear how the page could be used as a part of an assessment?
• Do you find other pages that actually have duplicate content? Is some content outdated (based on e.g. version history?)? Suggest how pages should be updated, deleted, or merged.
• Do you find errors or mistakes on the page?
• Is the text clear?
• Write or update a summary (one paragraph in the very beginning explaining the main points of the text) on the page. If the content is too unclear to write a good summary, write down clarification questions to the moderator of that page.
• Is the text properly referenced?
• Are there discussions on the Talk page? If yes,
  • Have they been linked to from the main page?
  • Have the current resolutions been incorporated in the main page?

Pages with R code

• Buildings in Basel
• Exposure to PM2.5 in Finland
• OpasnetUtils/Drafts
• Energy use of buildings
• Emission factors for burning processes
• Building model
• Health impact assessment
• Disease risk
• ERFs of environmental pollutants
• Burden of disease in Finland
• Climate change policies and health in Kuopio
• Building stock in Kuopio
• Exposure to PM2.5 in Finland
• Population of Kuopio
• ERF of indoor dampness on respiratory health effects
• Concentration-response to PM2.5
• Climate change policies in Kuopio
• Energy balance
• Energy balance in Kuopio
• Energy balance in Stuttgart
• Energy balance in Suzhou
• Energy transformations
• Greenhouse gas emissions in Kuopio
• Emission factors for burning processes
• Energy consumption of heating of buildings in Kuopio
• Fuels used by Haapaniemi energy plant
• op_fi:Luikonlahden rikastamon ympäristöterveysriskien arviointi
• op_fi:Väestön kohdekohtainen ympäristöperäisen haitta-ainealtistumisen ja terveysriskin arviointi
• op_fi:Kuljetuksen päästöt
• op_fi:Kaivoksen kuljetusten pölypäästö
• op_fi:Metallimalmin murskausprosessin pölypäästöt
• op_fi:Rikastekuljetukset
• op_fi:Metallimalmin hihnakuljetuksen pölypäästöt
• op_fi:Kohdekohtaisen Minera-arvioinnin mallisivu
• op_fi:Järvisedimenttien metallipitoisuudet
• op_fi:Energiantuotannon päästökertoimet/Sähköntuotanto
• op_fi:Talvivaaran kaivoksen terveysvaikutukset
• op_fi:Väestön kohdekohtainen ympäristöperäisen haitta-ainealtistumisen arviointi
• op_fi:Pneumokokki

Pages without R code

• Haapaniemi energy plant in Kuopio
• Energy consumption of heating of buildings in Kuopio
• Energy consumption and GHG emissions in Kuopio by sector
• Effect of urban land use change on ambient air temperature
• HI:Residential floorspace in Europe
• Climate change policies in Thessaloniki
• Greenhouse gas emissions in Rotterdam
• op_fi:Ympäristöterveydelliset viite- ja raja-arvot
• op_fi:Haitta-aineiden imeytyminen iholta elimistöön

Seminar: Lessons learned

Estimated preparing time: 9 hours for the three topics.

Workshop seminars 16th May

For the first presentations on Tuesday 16th, all topics will be about the group works that have been done during 15th and 16th May. Each presenting group will take a slightly different angle to the Congestion charge case. The topics and groups are described in the table. For each topic, consider two different points of view:

• Substantive: What were the key pieces of information related to our topic? Where did we find them? What was left out and why?
• Operative: How did our method work? Did it actually help us synthesise necessary information? Could other people understand our information product, i.e. did the method produce good information objects (compare to properties of good assessment)?

Times and topics for the seminar presentations

Time	Presenter	Topic	Presentation
14.00-14.20	Amr, Ehab	Variables in the causal chain of the assessment.	If you have presentation slides, put a link to them here.
14.20-14.40	Tamara, Tine	Organising text and discussion into structured arguments.
14.40-15.00	Edem, Margaret	Describing variables and statements in tables of Items, Relations, and Evaluations.
15.00-15.20	Jin, Kaisu?	Shared understanding and open assessment as a method for understanding a complex policy topic
15.20-15.40		Linking discussions into variables in the causal diagram
15.40-16.00	Jouni	General discussion

Final seminar 29th May

Each group has a slot of 30 min to present their topic and discuss it. A recommendation is to aim at 20 min of presentation and 10 min of discussion. Each group has a different topic, but each topic is about one of the homeworks. Note that each group may have different presentations related to the work the group has done: about presenting an assessment (HW4-6), structured discussion or evaluation (HW7, 9) and assessment (HW8). Remember that the audience has not read the report or assessment of your topic. So, in the presentation first describe the main purpose and content of your topic/material.

When preparing your presentation, focus on three things:

1. Describe the substantive content of your topic. What did you learn about it, what conclusions were made based on the material and the work?
2. Describe how the content relates to a wider perspective, namely an assessment or a decision process. What additional value did this topic bring? Did it change conclusions? Was it important in increasing understanding, or some other way?
3. How was it incorporated into the bigger picture? What methods were used to incorporate it? How were the methods used, and did the methods work for their purpose?

Times and topics for the seminar presentations

Time	Presenter	Topic	Presentation
times to be confirmed, each ca 30 min	Amr, Ehab	HW4	City of Rotterdam
	Tamara, Tine	HW5	DARM HW5 Lessons learned
	Edem, Margaret	HW6	SEMINAR PRESENTATION
	Jin, Kaisu	HW9
	Jouni	General discussion, Course evaluation

Course evaluation

During the last lecture, course feedback will be collected and discussed. Please answer to the following questions (based on properties of good assessment).

• Quality of content
  • Informativeness: Was the course informative? Did you learn new things, and things that you would not have (easily) learned from other courses?
  • Calibration: Was the information given truthful and scientifically correct? Were there topics that were not covered with enough rigour?
  • Coherence: Did the course content reflect the course description? Were some parts over- or underemphasises? Was everything addressed? Was something unnecessary?
• Applicability
  • Relevance: Correspondence between output and its intended use. Did the course teach you things that are useful in decision support work? Did the course teach you things that you are likely to use in your future job? What would you add to or leave out of the course?
  • Availability: Is the information provided by the course available to you when and where you need it in the future? Can you use the web tools in your future work?
  • Usability: Potential of the information of the course to trigger understanding in its user(s). Can you perceive and internalize the information provided, i.e. can you use this information? Did your understanding increase about the course topics?
  • Acceptability: Potential of the course being accepted by its users. Fundamentally a matter of its making and delivery, not its information content. How was the course organised? Is the course, and the way it is obtained and delivered for use, acceptable?
• Efficiency
  • Intra-course efficiency: Resource expenditure of participating in the course. How much effort is spent in the participation? Could some parts be learned with less effort? Would some parts have needed more effort? How much did you spend time on each homework?
  • Inter-course efficiency: Resource expenditure of the course as a part of your studies as a whole. Did the course help you learn more easily topics of the same area?