From open assessment to shared understanding: practical experiences

From Opasnet
Jump to: navigation, search


From open assessment to shared understanding: practical experiences is a manuscript of a scientific article. The main point is to offer a comprehensive summary of the methods developed at THL/environmental health to support informed societal decision making, and evaluate their use and usability in practical examples in 2006-2017. Manuscript is to be submitted to BMC Public Health as a technical advance article [3].

Title page

  • Title: From open assessment to shared understanding: practical experiences
  • List full names, institutional addresses and emails for all authors
    • Jouni T. Tuomisto1, Mikko Pohjola2, Teemu Rintala3.
      1. National Institute for Health and Welfare, P.O.Box 95, 70701 Kuopio, Finland
      2. Santasport, Rovaniemi, Finland
      3. Aalto University, Espoo, Finland
    • emails: jouni.tuomisto[]thl.fi, mikko.pohjola[]santasport.fi, teemu.rintala.a[]gmail.com
    • Corresponding author: Jouni Tuomisto

TO DO

  • A specific link should be available to mean the object itself rather than its description. http://en.opasnet.org/entity/...?
  • All terms and principles should be described in Opasnet in their own pages. Use italics to refer to these pages.
  • Upload Tuomisto 1999 thesis to Julkari. And Paakkila 1999

Letter to the Editor

Madam/Sir,

Please find enclosed a manuscript that describes a decision support methodology called open policy practice. It has been developed in National Institute for Health and Welfare (THL, Finland) during the last 15 years. Each assessment, case study, and method has been openly described and also typically published in scientific journals. However, this is the first comprehensive summary of the methodology as a whole (since 2007) and thus gives a valuable overview and a rationale for several methodological choices we have made.

We believe that open policy assessment is able to provide useful approaches to several contemporary problems related to unoptimal use of scientific information in policy making. Thus, it has a potential for becoming a highly cited reference in the science-policy interface.

Sincerely,

Jouni Tuomisto

Abstract

Background

Evidence-based decision making and better use of scientific information in societal decisions has been an area of development for decades but is still topical. A decision support work can be viewed from the perspective of information collection, synthesis, and flow between decision makers, experts, and stakeholders.

Methods

We give an overview of and describe practical experiences from the open policy practice method that has been developed in National Institute for Health and Welfare for more than a decade. Open assessments are online collaborative efforts to produce information for decision makers by utilising e.g. quantitative models, structured discussions, and knowledge crystals. Knowledge crystal is a web page that has a specific resarch question, and an answer is continually updated based on all available information. Shared understanding is used to motivate decision makers and stakeholders to common dialogue and inform about conclusions and remaining disagreements.

Results

The methods and online tools work as expected, as demonstrated by the numerous assessments and policy support processes conducted. The approach improves the availability of information and especially of relevant details. Experts are ambivalent about the acceptability of openness: it is an important scientific principle, but it goes against many current publishing practices. However, co-creation and openness are megatrends that are changing decision making and the society at large. Against many experts' fears, open participation has not caused problems in performing high-quality assessments. On the contrary, a key problem is to motivate more people, including experts and decision makers, to participate and share their views.

Conclusions

Shared understanding has proved to be a useful concept that guides policy processes toward more collaborative approach, whose purpose is wider understanding rather than winning. There is potential for merging open policy practice with other open science and open decision process tools. Active facilitation, community building and improving the user-friendliness of the tools were identified as key solutions for improving usability of the method in the future.

Keywords
environmental health, decision support, open assessment, open policy practice, shared understanding, modelling, online tools, policy, method development, evaluation, knowledge crystal

Background

In this article, we describe and evaluate open policy practice, a set of methods and tools for improving science-based policy making. They have been developed in the National Institute for Health and Welfare (THL, located in Finland) for more than 16 years especially to improve environmental health assessmentsa.

Science-based decision support has been a hot and evolving topic for a long time, and its importance is not diminishing any time soon. The area is complex, and all the key players – decision makers, experts, and citizens or other stakeholders – all have different views on the process, their own roles in it, and how information should be used in the process. For example, researchers often think of information as a way to find the truth, while politicians see information as one of the tools to promote political agendas ultimately based on values.[1] Therefore, any successful method should provide functionalities for each of the key groups.

In the 1970's, the focus was on scientific knowledge and an idea that political ambitions should be separated from objective assessments especially in the US. Since the 1980's, risk assessment has been a key method to assess human risks of environmental and occupational chemicals[2]. National Research Council specifically developed a process that could be used by all federal US agencies. Although it was generic in this sense, it typically focused on a single chemical at a time and thus provided guidance for administrative permissions such as pesticide approval, and it was not easily used for complex policy issues such as guidance for healthy city planning.

This shortcoming was tackled in another report that acknowledged this complexity and offered deliberation with stakeholders as a solution, in addition to scientific analysis[3]. However, despite these intentions, practical assessments have found it difficult to successfully perform deliberation on a routine basis[4]. On the contrary, citizens often complain that even if they have been formally heard during a process, they have not been listend to and their concerns have not affected decisions made.

In the early 2000's, several important books and articles were published about mass collaboration[5], wisdom of crowds[6], crowdsourcing in the government[7], and co-creation[8]. A common idea of the authors was that voluntary, self-organised groups had knowledge and capabilities that could be much more effectively harnessed in the society than what was happening at the time.

These ideas were seen as potentially important for environmental health assessment in THL (at that time National Public Health Institute, KTL), and they were adopted in the work of the Centre of Excellence for Environmental Health Risk Analysis (2002-2007). A technical milestone was achieved in January 2006 when we launched our own wiki site Opasnet for environmental health assessments, inspired by the success of Wikipedia. This enabled the intertwining of both theoretical and practical work to improve assessment methods and test openness and co-creation as elementary parts of the previously closed expert work[9]. This research soon led to a summary report about the new methods and tools developed to facilitate assessments[10].

The main ideas of our approach was to facilitate both the scientific work about policy-related facts and policy support about finding out what could and should - and should not - be done and why. We identified three critical needs for development in the scientific enterprise, namely a) data sharing, b) criticism, and c) common platform for discussion, data collection, and modelling. For decision making, we identified needs to explicate better a) decision makers' values and objectives, b) connections between scientific and other relevant issues, and c) disagreements between individuals and their potential resolutions. Decision making was essentially seen as an art of balancing knowledge with values in a coherent and explicit way.

In this article, we will give the first comprehensive, peer-reviewed description about the methodology and tools of open policy practice. Case studies have been published along the way, and the key methods have been described in each article. Also, all methods and tools have been developed online and the full material has been available for interested readers since each piece was first written. However, there has not been a systematic description since a 2007 report[10], and a lot of development has taken place since.

We will also take a step back and critically evaluate the methods used during the last 17 years since the start of the Centre of Excellence.

Finally, we will discuss some of the main lessons learned and give guidance for further work to improve the use of scientific information in societal decision making.

Methods

Insight networks

Figure 1. Insight network about dioxins, Baltic fish, and health as described in the BONUS GOHERR project[11]. Decisions are shown as red rectangles, decision makers and stakeholders as yellow hexagons, decision objectives as yellow diamonds, and substantive issues as blue nodes. The relations are written on the diagram as predicates of sentences where the subject is at the tail of the arrow and the object is at the tip of the arrow. For other insight networks, see Appendix 1.

Policy issues are complex, which makes it difficult to understand, analyse and discuss them effectively. The first innovation that we implemented to tackle this difficulty was insight networkb. Insight networks are based on an idea that regardless of the decision at hand, core issues always include some actions based on the decision, and these actions have causal effects on some objectives. In an insight network, actions, objectives, and other issues are depicted with nodes, and their relations are depicted with arrows (aka edges). For example, a potential dioxin-related decision is to clean up emissions from waste incineration. The logic of such a decision can be described as a chain or network of causally dependent issues as shown in Figure 1: Reduced dioxin emissions to air improve air quality and dioxin deposition into the Baltic Sea; this has a favourable effect on concentrations in the Baltic herring; this reduces human exposures to dioxins via fish; and this helps to achieve an ultimate objective of reduced health risks from dioxin.

Causal modelling as such is an old idea, and there are various methods developed for it, both qualitative and quantitative. However, the additional ideas with insight networks were that a) they can effectively be used in clarifying one's ideas and then communicating them especially in complex cases, and b) also all non-causal issues can and should be linked to the causal core in some way, if they are relevant to the decision. In other words, a participant in a policy discussion should be able to make a reasonable connection between what they are saying and some node in an insight network developed for that policy issue. If they are not able make such link, their point is probably irrelevant.

The first implementations of insight networks were about toxicology of dioxins[12] and restoration of a closed asbestos mine area[13]c. In the early cases, the main purpose was to give structure to discussion and assessment rather than to be a backbone for quantitative models. In later implementations, such as in the composite traffic assessment[14] or BONUS GOHERR project[11], diagrams have been used for both purposes.

For a description of the current notation, see Appendix 2.

Knowledge crystals

Although an insight network provides a method to illustrate and analyse a complex decision situation, it offers little help in describing quantitative nuances within the nodes or arrows, such as functional or probabilistic relations or estimates. There are tools with both graphical and modelling functionalities, e.g. Hugin (Hugin Expert A/S, Aalborg, Denmark) for Bayesian belief networks and Analytica® (Lumina Decision Systems Inc, Los Gatos, CA, USA) for Monte Carlo simulation. However, commercial tools are typically designed for a single desktop user rather than for open co-creation. In addition, they have limited possibilities for adding non-causal nodes and links or free-format discussions about the topics.

There was a clear need for theoretical and practical development in this area, so we combined three open source softwares into a single web-workspace named Opasnet for discussions, modelling and data storage of scientific and policy issues: Mediawiki, R statistical software, and MongoDB database. The workspace will be described in more detail in the next section and appendix 3. Here we will first look at how the information is structured in this system.

The second major innovation was the concept of a knowledge crystal. Its purpose is to offer a versatile information structure for a node in an insight network. It should be able to handle any topic and to systematically describe all causal and non-causal relations to other nodes, whether they are quantitative or qualitative. It should contain mathematics, discussions, illustrations, or other information as necessary. Also, it should handle both facts and values, and withstand misconceptions and fuzzy thinking as well. Its main structure should be universal and understandable for both a human and a computer. It should be manageable using scientific practices, notably criticism and openness, i.e. in a way that anyone can read and contribute to its content. And finally, it should be easy enough for an interested non-expert to find it online and to understand and use its main message.

After some experiments, we identified a few critical features a knowledge crystal should have to fulfil its objectives. First, a knowledge crystal is a web page with a permanent identifier or URL. Second, it has an explicit topic, which is described in the format of a research question. Importantly, the topic does not change over time (in practice, adjustments to the wording are allowed especially if the knowledge crystal is not yet established and used in several different assessments). This makes it possible for a user to come later to the same page and find an up-to-date version of the same topic.

Third, the purpose of a page is to give an informed answer to the question presented. The answer is expected to change as new information becomes available, and anyone is allowed to bring in new relevant information as long as certain rules of co-creation are followed. In a sense, the answer of a knowledge crystal is never final but it is always usable.

A standardised structure is especially relevant for the answer of a knowledge crystal, because it enables its direct use in assessment models or internet applications. So even though the content is updated as knowledge increases, the answer remains in the same, computer-readable format. So far, such interpretations of particular research topics have been rare outside Opasnet: open data contains little or no interpretation, and scientific reviews or articles are neither updatable nor machine-readable (at least not until artificial intelligence develops further).

Fourth, an answer is based on information, reasoning, and discussion presented on the page under the heading rationale. The purpose of rationale is to contain anything that is needed to convince a critical rational reader about the validity of the answer. It is also the place for new information and discussions that may change the answer.

Table 1. The attributes of a knowledge crystal.
Attribute Description
Name An identifier for the knowledge crystal. Each Opasnet page has a permanent, unique identifier or URL (e.g. http://en.opasnet.org/w/index.php?curid=4004) and a few practical ways to refer to the knowledge crystal: the name of the page for humans (e.g. Mercury concentrations in fish in Finland), and the page identifier for modelling code in computers (e.g. Op_en4004).
Question Gives the research question that is to be answered. It defines the scope of the knowledge crystal. When possible, the question should be defined in a way that it has relevance in many different situations, i.e. makes the page reusable. (For example, a page about mercury concentrations can be used in several assessments related to fish consumption and health.)
Answer Presents an understandable and useful answer to the question. It is the current best synthesis of all available data. Typically it has a descriptive easy-to-read summary and a detailed quantitative result published as open data. An answer may contain several competing hypotheses, if they all hold against scientific critique. This way, it may include an accurate description of the uncertainty of the answer, often in a probabilistic way.
Rationale Contains any information that is necessary to convince a critical rational reader that the answer is credible and usable. It presents the reader the information required to derive the answer and explains how it is formed. It may have different sub-attributes depending on the page type, some examples are listed below. Rationale may also contain lengthy discussions about relevant details.
  • Data tell about direct observations (or expert judgements) about the topic.
  • Dependencies tell what we know about how upstream knowledge crystals (i.e. causal parents) affect the answer. Dependencies may describe functional or probabilistic relationships.
  • Calculations are an operationalisation of how to calculate or derive the answer. It uses algebra, computer code, or other explicit methods if possible. Typically it is R code that produces and stores the necessary parts of a model to Opasnet.
Other In addition to attributes, it is practical to have clarifying subheadings on a knowledge crystal page. These include: See also, Keywords, References, Related files

It is useful to compare a knowledge crystal to a scientific article, which is organised around a single dataset or an analysis and is expected to stay permanently unchanged after publication. Further, articles offer little room for deliberation about the interpretation or meaning of the results after a manuscript is submitted: reviewer comments are often not published, and further discussion about an article is rare and mainly occurs only if serious problems are found. Indeed, the current scientific publishing system is poor in correcting errors via deliberation[15].

In contrast, knowledge crystal is designed to support continuous discussion about the science (or valuations, depending on the topic) backing up conclusions, thus hopefully leading to improved understanding of the topic.

This process is similar to that in Wikipedia, but the information structure is different, as Wikipedia articles describe issues rather than answer specific questions. Another difference is that Wikipedia relies on established sources such as textbooks rather than interprets original results. There is a clear reason for this difference: knowledge crystals are about some details of a decision or an assessment, and there is no established textbook knowledge about most of the information needed.

There are different kinds of knowledge crystals for different uses. Variables contain substantive topics such as emissions of a pollutant, food consumption or other behaviour of an individual, or disease burden in a population (for examples, see Figure 1.) Assessments describe the information needs of particular decision situations and work processes designed to answer those needs. They also may describe whole models (consisting of variables) for simulating impacts of a decision. Methods describe specific procedures to organise or analyse information. The question of a method typically starts with "How to..." For a list of all knowledge crystal types used in Opasnet web-workspace, see Appendix 2.

Knowledge crystals are designed to be modular and reusable. This is important for the efficiency of the work. Variables are used in several assessments where possible, and methods are used to standardise and facilitate the work across assessments. For this reason, all software in Opasnet are widely used open source solutions. An R package OpasnetUtils was developed to contain the most important methods, functions, and information structures needed to use knowledge crystals in modelling (for details, see Appendix 3).

Opasnet web-workspace

In 2006, we set an objective to develop an assessment system that has a solid theoretial basis and coherent practices that are supported by functional tools. We assumed that the different parts need to be developed side by side so that theories are tested with practical work and assessment practices and tools are evaluated against the theoretical framework. This lead our team to simultaneously develop information structures, assessment methods, and evaluation criteria; perform assessment case studies; and build software libraries and online tools.

The first part of our web-workspace, namely a wiki, was launched in January, 2006. Its name Opasnet is a short version of Open Assessors' Network. The purpose was to test and learn co-creation among environmental health experts and also start opening the assessment process to interested stakeholders. There were several reasons to choose Mediawiki platform. First, it is the same as used by Wikipedia, so a large group of people had already tested and developed many novel practices and we could directly learn from them. Second, open source code and wide use elsewhere would help our practices spread to new communities because the installation and learning costs were presumably low. Third, Mediawiki has necessary features to implement good research practices, such as talk pages to clearly separate content and discussions about the content, and automatic and full version control. And fourth, the maintenance and development of the software itself seemed to be certain for several years.

We launched several wiki instances for different projects, as participating researchers didn't want to write to a website that was visible to other projects as well. However, this caused extra maintenance burden and confusion with no real added value, so we soon started to discourage against this practice. Instead, we moved to a system with three wiki instances. English Opasnet (en.opasnet.org) contains all international projects and most scientific information. Finnish Opasnet (fi.opasnet.org) contains mostly project material for Finnish projects and pages targeted for Finnish audiences. Heande (short for Health, the Environment, and Everything) is a password-protected project wiki, which contains information that can not be published (yet) for a reason or another. We used a lot of effort to encourage researchers to write directly to open wikis, but most were hesitant to do so at the time (and many still are).

In the beginning, Opasnet was mainly used to document project content. All environmental health assessments were performed using commercial software, notably Analytica®. However, it was clear that the contents of all assessment models to the very detail should be published and opened up for public scrutiny. Although it is possible to upload model files for people to download and examine, this does not happen in practice if an interested reader has to obtain, install, and learn the software first. So, it was necessary to switch to an open source modelling software that enabled online work with model code.

The statistical software R was chosen, as it was widely used, it had object-oriented approach (thus supporting modularity) and it enabled complex modelling with fairly simple code, thanks to hundreds of packages that volunteers had written and shared to enhance the functionalities of R. There was no inherent support for R in Mediawiki, so we had to write our own interface. As a result, we could write R code directly to a wiki page, save, and run it by clicking a button. The output of a code would appear as a separate web page, or embedded on the wiki page with the code. Resulting objects could also be stored to the server and fetched later by another code. This made it possible to run complex models online without installing anything on your own computer (except a web browser). It also enabled version control and archival of the model data, code, and results.

We also developed an R package OpasnetUtils (available from CRAN repository cran.r-project.org) to support knowledge crystals and decision support models. It has a special object type for knowledge crystals (called ovariable) that implements the functionalities described above. For example, an answer can be calculated based on data or functional dependencies; an ovariable "understands" its own dependencies and is able to fetch its causal parents to the model from separate Opasnet pages; a model can be adjusted afterwards by implementing one or more decision options to relevant parts of the model, and this is done on an Opasnet page with no changes to the model code; and if input values are uncertain, it automatically propagates uncertainties through the model using Monte Carlo simulation.

The modelling functionalities created a need to store data to the web-workspace. A database called Opasnet Base was created using MongoDB no-sql softwared. It has several user interfaces.

A user can write a table directly on a wiki page, and the content will be uploaded to the database in a structured format. This is often used to give parameter values to variables in assessment models. A benefit is that the data is located in an intuitive place, typically under the Rationale subheading on a knowledge crystal page.

Another interface is especially used to upload large datasets (some population datasets contain ca. 10 million rows) to Opasnet Base. It is noteworthy that each dataset must be linked to a single wiki page, which contains all the necessary descriptions and metadata about the data. All datasets are also downloadable to R for calculations irrespective of whether R is run from an Opasnet page or from user's own computer.

This data structure facilitates coherent practices of daily work with little or no extra effort needed to link datasets to relevant topics and document and archive them. Functionalities are deliberately organised in a way that all assessment-related work and research can be performed using the same tools.

Open assessment

Open assessment is a method for performing impact assessments using insight networks, knowledge crystals, and open online assessment tools. Here we use "assessment of impacts" for ex ante consideration about what will happen if a particular decision is made, and "evaluation of outcomes" for ex post consideration about what did happen after a decision was implemented. Open assessments are typically performed before a decision is made (although in Carbon Neutral Helsinki 2035 both approaches are used; see below). The focus is necessarily on expert knowledge and how to organise that, although prioritisation is only possible if the objectives and valuations of the decision maker are known.

As a research topic, open assessment attempts to answer this research question: "How can factual information and value judgements be organised for improving societal decision making in a situation where open participation is allowed?" This question was in our minds when we developed many of the ideas presented in this article. As can be seen, openness, participation, and values are taken as given premises. In the early 2000's, this was far from common practice, although these ideas had been proposed before we started to develop practices based on them[3].

The main focus since the beginning was to think about information and information flows, rather than jurisdictions, political processes, or hierarchies. So, we deliberately ignored questions like what kinds of scientific committees are needed to support relevant high-quality advice; or how expert opinions should be included in the work of e.g. a government, parliament, or municipality council. The idea was rather that if the information production process is completely open, it can include information from any committee or individual as long as the quality aspect is successfully resolved. And if all useful information related to a decision can be synthesised and made available to everyone, then any kind of decision-making body could use that information. Generic approach was chosen to be helpful irrespective of administrative structures and governance processes.

Of course, this does not mean that any kind of organisation or individual is equally prone or capable of using assessment information. It simply means that we considered it as a separate question. Having said that, there was also a thought that if a good assessment is able to produce some clear and unequivocal conclusion that the whole public can see and understand, it will become much harder for any decision maker to deviate from that conclusion.

Principles in open assessment

Guidance exists about crowdsourced policymaking[16], and similar ideas have been utilised in open assessment. This section describes the principles that we have identified important for guiding the information work. For a full list of principles, see Table 2.

Intentionality is a key starting point for an open assessment. The purpose of intentionality is to inform a decision process about decision makers' values. Depending on intentions, different aspects and topics become relevant and important, and that should be reflected in the assessment work. Often also other values than those of the decision maker should be analysed: conclusions may be sensitive to initial values, or ignoring political opposition's views may cause trouble at a later stage. There are models for describing facts and values in a coherent dual system, and the use of such methods is encouraged[17]. Unfortunately, it is often not done, because it requires extra effort, and also it may be tactically useful for a decision maker not to conceal all their values.

Criticism based on observations and rationality is a central idea in the scientific method, and therefore it is also part of open assessment. Most critique is verbal or written discussion between participants, focussing on particular, often detailed topics. We have found some useful information structures for criticism, notably structured discussions that can target any part of an assessment (scope, data, premises, analyses, structure, results etc). Technically, links are added to a criticised text to guide an interested user to a discussion page where a structured critique is presented. If the critique is successful, the text is updated accordingly.

The principles of causality, criticism, permanent resource locations, openness, and reuse together rule out many common working environments and information structures. Therefore, we have developed and implemented knowledge crystals that strictly follow those principles. Knowledge crystals have also properties that are not directly derived from other principles, and therefore we consider the use of knowledge crystals as a principle of its own. The principles mentioned have been built into the structure and functionalities of knowledge crystals in aim to make it easy to obey them. Making and visualising insight networks based on knowledge crystals puts emphasis on causalities, and publishing them on open web-workspaces complies with openness automatically. Knowledge crystals with their machine-readable results promote reuse of information. Although principles are partly built into the system, they also require understanding and effort from the participants. The currently used system, Opasnet, has open structure and although it guides a user to apply the principles, it does not force them.

Table 2. Principles in open assessment.
Principle Description
Intentionality The decision maker explicates their objectives and decision options under consideration. All that is done aims to offer better understanding about impacts of the decision related to the objectives of the decision maker. Thus, the participation of the decision maker in the decision support process is crucial.
Causality The focus is on understanding and describing the causal relations between the decision options and the intended outcomes. The aim is to predict what impacts will likely occur if a particular decision option is chosen.
Criticism All information presented can be criticised based on relevance and accordance to observations. The aim is to reject ideas, hypotheses -- and ultimately decision options -- that do not hold against critique. Criticism has a central role in the scientific method, and here we apply it in practical situations, because rejecting poor statements is much easier and more efficient than trying to prove statements true.
Permanent resource locations Information is organised around topics (described as research questions), and each topic has a permanent location where it can be found even if the content develops in time. In practice, these locations are webpages with permanent URLs.
Openness All work and all information is openly available to anyone interested for reading and contributing all the time. If there are exceptions, these must be publicly justified. Openness is crucial because a priori it is impossible to know who may have important factual information or value judgements about the topic.
Reuse All information is produced in a format that can easily be used for other purposes by other people. Open data principles are used when possible[18]. For example, some formats such as PDF files are not easily reusable.
Use of knowledge crystals All information is openly shared using a systematic structure (notably question, answer, and rationale) and permanent locations in a common workspace where all participants can work. Knowledge crystals are used for this. The structure of an assessment and its data is based on substance (i.e. causal, logical and other substantive connections between issues). Objectives determine the information needs, which are then used to define research questions to be answered in the assessment. The assessment work is collaboration aiming to answer these questions in a way that holds against critique. Thus, knowledge crystals are practical information structures that comply with other principles of open assessment.

Properties of good assessment

There is a need to evaluate the assessment work before, during, and after it is done[19]. First we take a brief look at what makes a good assessment and what criteria could be used (see Table 3)[20].

Quality of content refers to the output of an assessment, typically a report, model or summary presentation. Its quality is obviously an important property. If the facts are plain wrong, it is more likely to misguide than lead to good decisions. But it is more than that. Informativeness and calibration describe how large the remaining uncertainties are and how close the answers probably are to the truth (compared with some golden standard). In some statistical texts, similar concepts have been called precision and accuracy, respectively, although with assessments they should be understood in a flexible rather than strictly statistical sense.[21] Coherence means that the answers given are those to the questions asked. Although not always easy to evaluate, coherence is an important property to keep in mind because lack of it is common: politicians are skillful in answering other questions than asked, and researchers tend to do the same if the research funding is not sufficient to answer the actual, hard question.

Applicability is a large evaluation area. It looks at properties that affect how well the assessment work can and will be applied to support decisions. It is independent of the quality of content, i.e. despite high quality, an assessment may have very poor applicability. The opposite may also be true, as sometimes faulty assessments are actively used to promote policies. However, usability typically decreases rapidly if the target audience evaluates an assessment to be of poor quality.

As coherence evaluates whether the assessment question was answered, relevance asks whether a good question was asked in the first place. Understanding what the right question actually is is surprisingly hard, and often its identification requires lots of discussion and deliberation between different groups, including decision makers and experts. Typically there is always too little time available for such discussions, and online forums may potentially help in this.

Availability is more technical property and describes how easily a user can find the information when needed. A typical problem is that a potential user does not know that a piece of information exists even if it could be easily accessed.

Usability may differ from user to user, depending on e.g. background knowledge, interest, or time available to learn the content.

Acceptability is a very complex issue and most easily detectable when it fails. A common situation is that stakeholders feel that they have not been properly heard and therefore any output from an assessment process is perceived faulty. Also doubts about the credibility of the assessor fall into this category.

Efficiency evaluates resource use when performing an assessment. Money and time are two common measures for this. Often it is most useful to evaluate efficiency before an assessment is started. Is it realistic to produce new important information given the resources and schedule available? If more (less) resources were available, what added (lost) value would occur? Another aspect in efficiency is that if assessments are done openly, reuse of information becomes easier and the marginal cost and time of a new assessment decrease.

All properties of good assessment, not just efficiency, are meant to guide planning, execution, and evaluation of the whole assessment work. If they are kept in mind always, they can improve daily work.

Table 3. Properties of good assessment.
Category Property Description Question
Quality of content Informativeness Specificity of information, e.g. tightness of spread for a distribution. How many possible worlds does the answer rule out? How few possible interpretations are there for the answer?
Calibration Exactness or correctness of information. In practice often in comparison to some other estimate or a golden standard. How close is the answer to reality or real value?
Coherence Correspondence between questions and answers. Also between sets of questions and answers. How completely does the answer address the assessment question? Is everything addressed? Is something unnecessary?
Applicability Relevance Correspondence between output and its intended use. How well does the information provided by the assessment serve the needs of the users? Is the assessment question good?
Availability Accessibility of the output to users in terms of e.g. time, location, extent of information, extent of users. Is the information provided by the assessment available when, where and to whom is needed?
Usability Potential of the information in the output to trigger understanding in its users about what it describes. Can the users perceive and internalise the information provided by the assessment? Does users' understanding increase about the assessed issue?
Acceptability Potential of the output being accepted by its users. Fundamentally a matter of its making and delivery, not its information content. Is the assessment result (output), and the way it is obtained and delivered for use, perceived as acceptable by the users?
Efficiency Intra-assessment efficiency Resource expenditure of producing the assessment output. How much effort is spent in the making of an assessment?
Inter-assessment efficiency Resource expenditure of producing assessment outputs in a series of assessments. If another (somewhat similar) assessment was made, how much (less) effort would be needed?

Open policy practice

Figure 2. Information flows in open policy practice. Open assessments and web-workspaces have an important role as information hubs. They collect relevant information for particular decision processes and organise and synthesise it into useful formats for especially decision makers but also for anyone. The information hub works more effectively if all stakeholders contribute to one place, or alternatively facilitators collect their contributions there.

During Intarese project (2005-2011), it became more and more clear to us based on literature and own practical experience that assessments themselves were not enough to convey the information to decision processes. The scientific and political realms are based on different premises and objectives, and we identified a need to evaluate when the information does not flow well and what are typical problems with it. So, new theoretical work was done on decision processes, roles of assessments and information in them, and guidance for participants (Figure 2).

Open policy practice is a method to support societal decision making in an open society, and it is the overarching concept covering all methods, tools, practices, and terms presented in this article[22]. One part of open policy practice is open assessment, which focuses on producing relevant information for decision making. Open policy practice is a larger concept and it is especially focused on promoting the use of information in the decision making process. It gives practical guidance for the whole decision process from initiation to decision support to actual decision making to implementation and finally to evaluation of outcomes. Our aim was that it is applicable to all kinds of societal decision situations in any administrative area or discipline.

Figure 3. The four parts of open policy practice. The timeline goes roughly from left to right, but all work should be seen as iterative processes. Shared understanding as the main objective is in the middle, expert-driven information production is on the top as part of execution. Co-creationis used in all parts of the work, therefore it is an underlying object.

Open policy practice is divided into four main parts, which are briefly described here (see also Figure 3).

  • Shared understanding (a structured documentation of different facts, values, and disagreements related to a decision situation) is the main objective of open policy practice. It is a product that guides the decision and also is basis for evaluation of outcomes. For more details, see the next section.
  • The execution of decision support. It is mostly about collecting, organising and synthesising scientific knowledge and values in order to inform the decision maker to help them reach their objectives. In practice, most of open assessment work happens in this part. It also contains communication, co-creation, and execution of the decision process itself and the integration of these two processes.
  • Evaluation and management of the work (of decision support and decision making). It focusses on looking at what is being done, whether the work produces the intended knowledge and helps to reach the objectives, and what needs to be changed. It continues all the way through the decision process (before, during, and after the actual execution).
  • Co-creation. Information has to be collected, organised, and synthesised; and facilitators need to motivate and help people to share their information. This requires specific skills and work that are typically available among neither experts nor decision makers. It also contains specific practices and methods, such as motivating participation, facilitating discussions, clarifying and organising argumentation, moderating contents, using probabilities and expert judgement for describing uncertainties, or developing insight networks or quantitative models. Sometimes the skills needed are called interactional expertise.

When we thought about the decision making process from planning to implementation and evaluation, we realised that the properties of good assessment (see previous section) can be easily adjusted to this wider context. Target object in this wider context is not an assessment report but a shared understanding about the decision options to be chosen and implemented, or rejected. The evaluation criteria are valid in both contexts. The adjusted list of criteria is presented on Opasnet page Open policy practice. We have also developed other evaluation criteria for important aspects of open policy practice; the most important ones are described below.

Open assessment (the main part of execution) has been discussed already. Evaluation and management is an integral part of making sure that execution is performed well. Therefore, we present that next, and later co-creation and shared understanding.

Evaluation and management

We developed several criteria to evaluate and describe the decision support and decision making work. Their purpose is to help participants focus on the most important parts of open policy practice.

Settings of assessments

All too often a decision making process or an assessment is launched without clear understanding, what should be done and why. An assessment may even be launched in a hope that it will somehow reveal what the objectives or other important things are. Settings of assessments are a part of evaluation and management (Table 4). They try to help in explicating these things so that useful decision support can be provided[23]. Also the subattributes of an assessment scope help in this:

  • Question: What is the actual research question?
  • Boundaries: What are the temporal, geographical, and other limits within which the question is considered?
  • Decisions and scenarios: What decisions and options will be assessed and what scenarios will be considered?
  • Timing: What is the schedule of the assessment work?
  • Participants: Who are the people who will or should contribute to the assessment?
  • Users and intended use: Who is going to use the final assessment report and for what purpose?
Table 4. Important settings for environmental health (and other) assessments and related public policy.
Attribute Guiding questions Example categories
Impacts
  • Which impacts are addressed in assessment?
  • Which impacts are most significant?
  • Which impacts are most relevant for the intended use?
Environment, health, cost, equity
Causes
  • Which causes of impacts are recognized in assessment?
  • Which causes of impacts are most significant?
  • Which causes of impacts are most relevant for the intended use?
Production, consumption, transport, heating, power production, everyday life
Problem owner
  • Who has the interest, responsibility and/or means to assess the issue?
  • Who actually conducts the assessment?
  • Who has the interest, responsibility and/or power to make decisions and take actions upon the issue?
  • Who are affected by the impacts?
Policy maker, industry, business, expert, consumer, public
Target users
  • Who are the intended users of assessment results?
  • Who needs the assessment results?
  • Who can make use of the assessment results?
Policy maker, industry, business, expert, consumer, public
Interaction
  • What is the degree of openness in assessment (and management)? (See Table 5.)
  • How does assessment interact with the intended use of its results? (See Table 6.)
  • How does assessment interact with other actors in its context?
Isolated, informing, participatory, joint, shared

Interaction and openness

In open assessment, the method itself is designed to facilitate openness in all its dimensions. The dimensions of openness help to identify if and how the work deviates from the ideal of openness, so that the work can be improved in this respect (Table 5)[24]. However, it is important to notice that currently there is a large fraction of decision makers and experts who are not comfortable with open practices and openness as a key principle, and they would like to have a closed process. Dimensions of openness do not give direct tools to convince them. But it identifies issues where openness makes a difference and increases understanding about why there is a difference. This hopefully also leads to wider acceptance of openness.

Table 5. Dimensions of openness.
Dimension Description
Scope of participation Who are allowed to participate in the process?
Access to information What information about the issue is made available to participants?
Timing of openness When are participants invited or allowed to participate?
Scope of contribution To which aspects of the issue are participants invited or allowed to contribute?
Impact of contribution How much are participant contributions allowed to have influence on the outcomes? In other words, how much weight is given to participant contributions?

Openness can also be examined based on how intensive it is and what kind of collaboration is aimed at between decision makers, experts, and stakeholders[4][25]. Different approaches are described in Table 6. During the last decades, the trend has been from isolated to more open approaches, but all categories are still in use.

Table 6. Categories of interaction within the knowledge-policy interaction framework.
Category Description
Isolated Assessment and use of assessment results are strictly separated. Results are provided to intended use, but users and stakeholders shall not interfere with making of the assessment.
Informing Assessments are designed and conducted according to specified needs of intended use. Users and limited groups of stakeholders may have a minor role in providing information to assessment, but mainly serve as recipients of assessment results.
Participatory Broader inclusion of participants is emphasized. Participation is, however, treated as an add-on alongside the actual processes of assessment and/or use of assessment results.
Joint Involvement of and exchange of summary-level information among multiple actors in scoping, management, communication and follow-up of assessment. On the level of assessment practice, actions by different actors in different roles (assessor, manager, stakeholder) remain separate.
Shared Different actors involved in assessment retain their roles and responsibilities, but engage in open collaboration upon determining assessment questions to address and finding answers to them as well as implementing them in practice.

Co-creation

Co-creation, or knowledge production by self-organised groups, is a discipline in itself[8], but here we only raise a few key points about facilitation and structured discussion.

Facilitation is an art of helping people participate in co-creation processes using hearings, workshops, online questionnaires, wikis, and other tools. In addition to practical tools, facilitation implements principles that have been seen to motivate participation. Two are worth mentioning here.

  • Grouping: Facilitation methods are used to promote the participants' feeling of being an important member of a group that has a meaningful purpose.
  • Respect: Contributions are systematically documented and their merit evaluated so that each participant receives the respect they deserve based on their contributions.

Structured discussions are synthesised and reorganised natural discussions, where the purpose is to highlight key statements and argumentations that lead to acceptance or rejectance of these statements. Discussions can be organised according to pragma-dialectical argumentation rules[26][27], so that arguments form a hierarchical thread pointing to a main statement or statements. Attack arguments are used to invalidate arguments by showing that they are either untrue or irrelevant in their context; defend arguments are used to protect from attacks; and comments are used to clarify issues. For an example, see Figure A1-4 in Appendix 1 and links thereof. In Opasnet, such hierarchical structures are built based on what people say or write by using argument templates. Although handwork is required, it has clear added value for a reader, because even a lengthy discussion can be summarised into short statements after resolutions are found. Hierarchical threads are efficient, because a reader does not need to read further if they agree with the topmost arguments. On the other hand, any thread can be individually scrutinised to the last detail if needed.

Shared understanding

Figure 4. Shared understanding is produced in collaboration by decision makers (managers), experts (assessors), and stakeholders. Each group brings in their own knowledge and concerns.

Shared understanding is a situation where all participants' views about a particular topic have been described and documented well enough so that people can know what facts, opinions, reasonings, and values exist; and what agreements and disagreements exist and why.

Shared understanding is always about a particular topic and produced by a particular group of participants. With another group it could be different, but with increasing number of participants, it should approach shared understanding of the whole society. Each participant should agree that the written description correctly contains their own thinking about the topic. Participants should even be able to correctly explain what other thoughts there are and how they differ from their own. Ideally any participant can learn, understand, and explain any thought represented in the group. Importantly, there is no need to agree on things, just to agree on what the disagreements are about. Therefore, shared understanding is not the same as consensus or agreement.

Shared understanding has potentially several purposes that all aim to improve the quality of societal decisions. It helps people understand complex policy issues. It helps people see their own thoughts from a wider perspective and thus increase acceptance of decisions. It improves trust in decision makers; but it may also deteriorate trust if the actions of a decision maker are not understandable based on shared understanding. It dissects each difficult detail into separate discussions and then collects statements into an overview; this helps to allocate the time resources of participants efficiently. It improves awareness of new ideas. It releases the full potential of the public to prepare, inform, and make decisions. How well these purposes have been fulfilled in practice in assessments will be discussed in Results.

Test of shared understanding can be used to evaluate how well shared understanding has been achieved. In a successful case, all participants of a decision process will give positive answers to the questions in Table 7. In a way, shared understanding is a metric for evaluating how well decision makers have embraced the knowledge base of the decision situation.

Table 7. Test of shared understanding.
Question Who is asked?
Is all relevant and important information described? All participants of the decision processes.
Are all relevant and important value judgements described? (Those of all participants, not just decision makers.)
Are the decision maker's decision criteria described?
Is the decision maker's rationale from the criteria to the decision described?

Shared understanding may have different levels of ambition. On an easy level, shared understanding is taken as general guidance and an attitude towards other people's opinions. Main points and disagreements are summarised in writing, so that an outsider is able to understand the overall picture.

On an ambitious level, the idea of documenting all opinions and their reasonings is taken literally. Participants' views are actively elicited and tested to see whether a facilitator is able to reproduce their thought processes. The objective here is to document the thinking in such a detailed way that a participant's response can be anticipated from the description they have given. The purpose is to enable effective participation via documentation, without a need to be present in any particular hearing or other meeting.

Written documentation with an available and usable structure is crucial in spreading shared understanding among those who were not involved in discussions. Online tools such as wikis are needed especially in complex topics, among large groups, or if the ambition level is high.

Good assessment models are able to quickly and easily incorporate new information or scenarios, so that they can be run again and again and learn from these changes. In a similar way, a comprehensive shared understanding can incorporate new information from the participants. A user should be able to quickly update the knowledge base, change the point of view, or reanalysise how the situation would look like with alternative valuations. Such level of sophistication necessarily requires a few concepts that have not yet been described.

Value profile

Value profile is a list of values, preferences, and choices made and documented by a participant. Voting advice applications produce a kind of value profiles. The candidates answer questions about their values, worldviews, or decisions they would do if elected. The public can then answer the same questions and analyse which candidates share their values. Nowadays, such applications are routinely developed by all major media houses for every national election in Finland. However, these tools are not used to collect value profiles from the public between elections although such information could be used in decision support. Value profiles are mydata, i.e. data about which an individual themself may decide who is allowed to see and use it. This requires trusted and secure information systems. Although some ideas of piloting exist, we are not aware of systems that would collect value profile data for actual policy support between elections.

Archetype

Archetype is an internally coherent set of values that is supported by a fuzzy set of people. Fuzzy set means that the people in that set may support the archetype as a whole or some specific values in it only partially, with a degree ranging from zero to one. So, an individual can explicate their own values by saying that they are equal to those of an archetype, possibly with some exceptions. For example, political parties can develop archetypes to describe their political agendas and programs. In this way, people and groups may easily document value profiles in a shared understanding system. Practical experience of archetypes is limited, so little is known about how practical and accepted they would be. However, writing policy papers is established routine, and the main innovation with value profiles and archetypes is to be systematic and explicit about the details of value structures among the participants rather than providing something that is unfamiliar to people.

Archetypes aim to save effort in gathering value data from the public, as not everyone needs to answer all possible questions, when archetypes are used. It also increases security when there is no need to handle individual people's potentially sensitive answers but open aggregated data about archetype values instead.

The use of archetypes is based on a premise that although their potential number is very large, most of people's thinking can be covered with a fairly small amount of archetypes and paradigms. As a comparison, there are usually from two to a dozen political parties in a democratic country rather than hundreds. There is also research on human values showing that they can be systematically evaluated using a fairly small amount (4, 10, or 19) of different dimensions[28].

In other words, it seems possible to build a system that efficiently covers a large share of what people have to say about policy issues by using a manageable number of archetypes. If this is true, resources can be used to get the critical issues precise and informative, and then apply these information objects in numerous practical cases.

Paradigm

Paradigms are collections of rules to mimic inferences that participants would make from data in the system. For example, scientific paradigm has rules about criticism and a requirement that statements must be backed up by data or references. Participants are free to develop paradigms with any rules of their choosing, as long as they can be documented and operationalised within the system. For example, a paradigm may state that when in conflict, priority is given to the opinion presented by a particular authority. Hybrid paradigms are also allowed. For example, a political party may follow the scientific paradigm in most cases but when economic assessments are ambiguous, the party will choose an interpretation that emphasises the importance of an economically active state (or alternatively market approach with a passive state).

Although we have used wording such as "make inferences in the system", this approach does not need to rely on artificial intelligence. Rather, numerous contributors who co-create content and make inferences based on the rules described in paradigms, are the "system" we refer to. Parts of the work described here may be automated in the future, but the current system would be mostly based on information work by humans.

By now it seems clear that information in a description of shared understanding is very complex and often very large. So, a new research question emerges: how can all this information be written down and organised in such a way that it can easily be used and searched by both a human and a computer? A descriptive book would be too long for busy decision makers and unintelligible for computers. An encyclopedia would miss relevant links between items. A computer model would be a black box for citizens.

Open policy ontology

There is a need to describe all the information structures and policy content in a systematic, coherent, and unambiguous way. So, there was a need for an open policy ontology. We developed the first version recently and tested it with indicator production and Carbon Neutral Helsinki 2035 action plan[29].

World Wide Web Consortium has developed the concepts of open linked data and resource description framework (RDF)[30]. These were used as the foundations for ontology development.

Ontologies are based on vocabularies with specified terms and meanings. Also the relations of terms are explicit. Resource description framework is based on the idea of triples, which have three parts: subject, predicate (or relation), and object. These can be thought as sentences: an item (subject) is related to (predicate) another item or value (object), thus forming a claim. Claims can further be specified using qualifiers and backed up by references. Insight networks can be documented as triples, and a set of triples using this ontology can be visualised as diagrams of insight network. Triple databases enable wide, decentralised linking of various sources and information. There are open source solutions for creating such databases, such as Wikibase that also has an interface to wiki platform. There is also a query language SPARQL facilitating the management for and search of the data in RDF format. Thus, there are established open source tools for managing triple data and thus insight network content. However, for smaller needs we have used R.

The current version of the open policy ontology (see Appendix 2) focusses on describing all information objects and terms described above, and making sure that there is a relevant item type or relation to every critical piece of information that is described in an insight network, open assessment, or shared understanding. "Critical piece of information" means something that is worth describing as a separate node, so that it can be more easily found, understood, and used. A node itself may contain large amounts of information and data, but for the purpose of producing shared understanding about a particular decision, there is no need to highlight the node's internal data on an insight network. What is critical and what not is a matter of practicality and does not have an objective solution, as shown by examples below.

Figure 1 shows that the EU Parliament and Ministry Council have a role in fishing regulation, but this graphical representation of the triple does not attempt to tell anything else about these large and complex organisations, except that they have a role related to bioaccumulation of dioxin (shown as "stakeholder" node); that they make a decision about herring fishing intensity (shown as "decides" arrow); and that a link is provided to the web page of that organisation. Also, in Figure A1-1 in Appendix 1 there are two scientific articles listed. The content of these articles is not described, because already with little effort (connecting an article identifier, a topic, and a link to the article) the availability and usability of the article increases a lot. In contrast, a further step of structuring the article content into the insight network would take much more work and give less added value. The effort may become worth it, if someone identifies an important new statement in the article.

Thus, the art of open policy practice is to learn to identify important pieces of relevant information (such as scientific facts, publications, discussions etc.) and to add that information into a proper place in an insight network by using open policy ontology and a reasonable amount of work. A key to success is to identify the right level of detail to describe in the system for the purpose of informing decision makers, stakeholders, and others. An ontology helps to do this in a way that is understandable for both humans and computers.

For a full description of the current vocabulary in the ontology, see Appendix 2 and Figure A1-3 in Appendix 1.

Results and evaluation

Table 8. Some environmental health assessments performed using open assessment. More assessments can be found on Opasnet page Category:Assessments. References give links to both an assessment page and a scientific publication as applicable.
Topic Assessment Year Project
Vaccine effectiveness and safety Assessment of the health impacts of H1N1 vaccination[31] 2011 In-house, collaboration with Decision Analysis and Risk Management course
Tendering process for pneumococcal conjugate vaccine[32] 2014 In-house, collaboration with the National Vaccination Expert Group
Energy production, air pollution and climate change Helsinki energy decision[33] 2015 In-house, collaboration with city of Helsinki
Climate change policies and health in Kuopio[34] 2014 Urgenche, collaboration with city of Kuopio
Climate change policies in Basel[35] 2015 Urgenche, collaboration with city of Basel
Availability of raw material for biodiesel production[20] 2012 Jatropha, collaboration with Neste Oil
Health impacts of small scale wood burning[36] 2011 Bioher, Claih
Carbon neutral Helsinki 2035 action plan[29] 2018 In-house, collaboration with city of Helsinki
Health, climate, and economic effects of traffic Gasbus - health impacts of Helsinki bus traffic[37] 2004 Collaboration with Helsinki Metropolitan Area
Cost-benefit assessment on composite traffic in Helsinki[14] 2005 In-house
Risks and benefits of fish consumption Benefit-risk assessment of Baltic herring in Finland[38] 2015 Collaboration with Finnish Food Safety Authority
Benefit-risk assessment of methyl mercury and omega-3 fatty acids in fish[39] 2009 Beneris
Benefit-risk assessment of fish consumption for Beneris[40] 2008 Beneris
Benefit-risk assessment on farmed salmon[41] 2004 In-house
Benefit-risk assessment of Baltic herring and salmon intake[11] 2018 BONUS GOHERR
Dioxins, fine particles TCDD: A challenge to mechanistic toxicology[12] 1999 EC ENV4-CT96-0336
Comparative risk assessment of dioxin and fine particles[42] 2007 Beneris
Plant-based food supplements Compound intake estimator[43] 2014 Plantlibra
Health and ecological risks of mining Paakkila asbestos mine[13] 1999 In-house
Model for site-specific health and ecological assessments in mines[44] 2013 Minera
Risks of water from mine areas [45] 2018 Kaveri
Drinking water safety Water guide[46] 2013 Conpat
Organisational assessments Analysis and discussion about research strategies or organisational changes within THL 2017 In-house
Transport and communication strategy in digital Finland[47] 2014 Collaboration with the Ministry of Transport and Communications of Finland
Information use in government or municipality decision support Case studies: Assessment of immigrants' added value; Real-time co-editing, Fact-checking, Information design[48] 2016 Yhtäköyttä, collaboration with Prime Miniter's Office
Evaluation of forest strategy process for Puijo, Kuopio[49] 2012 In-house
Indicator development Environmental health indicators in Finland[50] 2018 Ympäristöterveysindikaattori
Indicators for the social affairs and health sector in Finland[51] 2018 In-house

The methods described above have been used in several research projects (see the funding declaration) and health assessments (some mentioned in Table 8) since 2004. They have also been taught on international Kuopio Risk Assessment Workshops for doctoral students in 2007, 2008, and 2009 and on a Master's course Decision Analysis and Risk Management (6 credit points), organised by the University of Eastern Finland (previously University of Kuopio) in 2011, 2013, 2015, and 2017.

As methods and tools were developed side by side with practical assessment work, there is extensive experience about some parts of the method. Some newer parts (e.g. value profiles and paradigms) have so far only been tested and found conceptually sound ideas; for example, we have implemented value profiles and paradigms on structured discussions and produced shared understanding description that has a structure that is readable by both humans and machines.

This evaluation is based on the experience accumulated during the scientific, expert, and teaching work. We will follow the properties of good assessment and apply them to the method itself.

Quality of content

Scientific quality is an important property of an assessment report, and the common methods should be used: careful evaluation of data to be used, state-of-art analysis methods, and source criticism are crucial parts of good work. Open assessments are not different than other assessments in this respect. Also, open policy practice does not restrict the use of any previous methods that are necessary for successful decision support. Therefore, it is safe to say that the quality of the produced information is potentially at least the same as with some other method. We have noticed in numerous cases that the structures offered for e.g. assessments or knowledge crystals help in organising and understanding the content, thus facilitating work and improving content.

Criticism is emphasised in open policy practice as an integral part of quality control. Giving critique is made as easy as possible. Surprisingly, we still see fairly little of it in practical assessments. There seem to be several reasons: experts have little time to actually read other people's assessments and give detailed comments; people are reluctant to interfere with other people's work even within a joint project, so they rather keep to a strict division of tasks; there are no rewards or incentives for giving critique; and if an assessment spans several web pages, it is not clear when and how to contribute.

We have seen lack of criticism even in the vaccine-related assessment in 2014 with a potentially emotive topic. With active facilitation we were able to get comments and critique from both drug industry and vaccine citizen organisations, and they were all very matter-of-fact. This was interesting, as the same topics cause outrage in social media, but we did not see that on structured assessments. Interestingly, one of the most common objections and fears against open assessment is that outside contributions will be ill-informed and malevolent. They have never been in our assessments.

Lack of contributions limits the amount of new views and ideas that potentially could be identified with open participation. However, even if we do not see all potential of criticism, we don't think that the lack of open critique would hamper the quality of assessments, because all the common practices of e.g. experts' source-checking are still in place. Actually, the current practices in research and assessment are even worse in respect of open criticism: it rarely happens. Pre-publishing peer review is almost the only time when scientific work is criticised by people outside the authoring research group, and those are typically not open. A minute fraction of published works are criticised openly in journals; a poor work is simply left alone and forgotten. Interestingly, some administrative processes follow scientific principles better than many research processes: for example, environmental impact assessment has a compulsory process for open criticism at both design and result phases[52].

Fortunately, there are some active platforms for scientific discussion and criticism, such as pre-publishing physics forum ArXiv.org, and similar platforms are emerging on other disciplines. Such practice suits to continually updating knowledge crystals much better than what is typically done in most research areas. Open discussion is increasing, and knowledge crystals are one way to facilitate this positive trend.

Relevance

A major issue with relevance is that the communication between decision makers and experts is not optimal, and therefore experts don't have good knowledge about what questions need answers and how the information should be provided. Also, decision makers prefer information that supports views that have already been selected on political grounds[1].

In open policy practice, the questions and other structures aim to explicate relevant questions, so that experts get a clear picture about what information is expected and needed. They also focus discussions and assessment work to questions that have been identified as relevant. On the other hand, shared understanding and other explicit documents of relevant information make it harder for a decision maker to pick only favourable pieces of information, at least if there is political pressure. The pneumococcal vaccine assessment (see Table 8) had clear political pressure, and it would have been difficult for the decision maker to deviate from the conclusions of the assessment. However, typically assessments are taken as just one piece of information rather than a normative guidance; this clearly depends on political culture, how strong the commitment to evidence-based decision making is, and how well an assessment succeeds in incorporating all relevant views.

Shared understanding and detailed assessments challenge participants to clarify what they mean and what is found relevant. For example, the composite traffic and biodiesel assessments were already quite developed, when discrepancies between objectives, data, and draft conclusions forced rethinking about the purposes of the assessments. In both cases, the main research questions were adjusted, and more relevant assessments and conclusions were produced. Political questions and arguments are also expected to clarify in a similar way when more of it is incorporated into systematic scrutiny.

Availability

A typical problem with availability is that a piece of information is designed for a specific user group and made available in a place targeted to that group. In contrast, open policy practice is designed with an idea that anyone can be a reader or contributor in a user group, so by default everything is made openly available on the Internet. To our experience, such approach works well in practice as long as there are seamless links to also repositories for non-publishable data and contributors know the open source tools such as wiki and R. Such openness is, however, a major perceived problem to many experts; this issue is discussed more under acceptability.

Another problem is that even if users find an assessment page, they are unsure about its status and whether some information is still missing. This is because many pages are work in progress and not finalised for end users. We have tried to clarify this by adding status declarations on the tops of pages. Declaring drafts as drafts has also helped experts who are uncomfortable in showing their own work before it is fully complete.

The information structure using knowledge crystals and one topic per page has proven a good one. Usually it is easy to find the page of interest even if there is only a vague idea of its name or specific content. Also, it is mostly straightforward to see which information belongs to which page.

There are some comments about not being able to find pages in Opasnet, but to our experience these problems are surpassed by benefits for people being able to easily find detailed material with e.g. search engines without prior knowledge about Opasnet. The material seems to be fairly well received, as there were 52000 visits to and 90000 pageviews of the Finnish and English Opasnet websites in 2017. The most interesting topics seemed to be ecological and health impacts of mining and drinking water safety. Also the pneumococcus vaccine assessment, Helsinki energy decision, and transport and communication strategy in digital Finland were popular pages when they were prepared.

Lack of participation among decision makers, stakeholders, and experts outside the assessment team is a constant problem in making existing information available. Assessments are still seen as a separate part from the decision process, and the idea that scientific assessments could contain value judgements from the public is unprecedented. The closest resemblance are environmental impact assessments, where the law requires public hearings, but many people are sceptic about the influence of comments given.

Participation is far from its full potential also in Wikipedia, where only a few percent of readers ever contribute, and the fraction of active contributors is even smaller[53].

Experts are a special group of interest, as they possess vast amounts of relevant information that is not readily available to decision makers. Yet, we have found that experts are not easily motivated to policy support work.

This emphasises the need for facilitation and active invitations for people to express their views. Importantly, direct online participation is not an objective as such but one way among others to collect views and ideas. It is also more important to cover all major ideas than to represent every person individually. In any case, there is a clear need to inform people about new possibilities for participation in societal decisions. We have found it very useful to simply provide links to ongoing projects to many different kinds of user groups across and outside organisations.

Open policy practice was designed so that it could incorporate any relevant information from experts and stakeholders. In practice, it is not used for that purpose, because it does not have an established role in decision process. For many interest groups, non-public lobbying, demonstrations and even spreading faulty information are more attractive ways of influencing the outcome of a decision. All these methods deviate from the ideal of evidence-based policy, and therefore further studies are needed specifically on how this information is better incorporated into and made available via shared understanding and whether that improves decision processes. If shared understanding is able to offer acceptable solutions to disagreeing parties, it reduces the need to use political force, but so far we have too little experience on that to make conclusions. We also don't yet know whether quiet people or marginal groups have better visibility with such a web-workspace, but in theory, their capabilities are better.

We have found it very useful that the version control and archival process is supported by Opasnet workspace. Many experts are reluctant to make their text available if other people can edit it, but this fear is typically alleviated when they learn that the original version can always be restored from the archive. Availability is also improved as information is produced in a proper format for archiving, backups are produced automatically, and it is easy to produce a snapshot of a final assessment. There is no need to copy information from one repository to another, but it is also easy to store final assessments in external open data repositories.

Usability

We have found it very useful to structure pages so that it starts with a summary, then describes the research question and gives a more detailed answer, and finally provides a user with relevant and increasingly detailed information in the rationale. On the other hand, some people have found this structure confusing as they don't expect to see all the details of an assessment. This emphasises the need to publish easy summaries in other places as well, e.g. in blogs, newsletters, or policy briefs, which are more familiar formats for readers. Indeed, there are lots of online open-source and other tools available to improve the usability of information production and consumption. Some of these are listed in Appendix 4.

A strength of shared understanding is in its capability of clarifying complex issues and eliciting implicit valuations and reasonings. It facilitates rational discussion about a decision, and it can also be used for creating policital pressure against options that are not well substantiated. For example, the shared understanding about the research strategy of THL (see Table 8.) was produced mostly based on critical discussions about valuations and reasonings rather than scientific analysis of decision optimisation. The process was well received and many participants found the discussions and syntheses illuminating.

Boundary object is a concept for managing information work within a heterogeneous group of participants[54]. As people come from different disciplines, they see things differently and use different words to describe things. Boundary objects are common words or concepts that are similar enough across disciplines so that they help understanding but allow specific interpretations within disciplines or by individuals. Several dioxin-related knowledge crystals were successfully used as boundary objects in BONUS GOHERR project to produce shared understanding among authorities, fishers, and researchers from public health, marine biology, and social sciences.

Shared understanding can incorporate complex ideas from detailed discussions and also quantitative data from surveys. Often many different information sources and analyses are needed to produce a full picture. Indeed, open policy practice method was developed using many approaches simultaneously: information structures, online tools, and testing with actual cases and different pracices. This seemed to be a viable strategy. However, even more emphasis should have been put to usability and user friendliness of tools, and building user community.

One of the criteria for development of open policy practice was that it should be versatile and usable for any expert or policy work. However, the implementation of the method can start gradually, like Prime Minister's Office started to use useful concepts and tested online editing but were reluctant to deeply commit to open assessments. Also, data visualisations showed immediate utility and were seen as an important way to communicate expert knowledge. In contrast, the full open policy practice method contains many parts that are clearly against the current practices and therefore need thorough testing and evaluation before they become usable in decision making processes. For example, the discussion about what parts of a law making process could be opened to the public has only started. The current default is that little is opened except some hearings that are required by law.

To our experience, the slow development of the user community is partly but not mainly due to non-optimal usability of tools: both Mediawiki and especially R require some learning before they can be effectively used. All students in the courses have been able to learn and effectively operate in Opasnet, so the learning challenge can be overcome if users are motivated. A bigger problem has been to motivate people to change their practices about how to perform or participate in assessments. Many experts have excelled in using different tools and processes, and switching to another platform (e.g. from Microsoft Excel or SAS to R) is a major investment that is not made lightly.

Acceptability

A central theme related to acceptability is openness. Although it is a guiding principle in science, it is actually in conflict with many current practices. For example, it is common to hide expert work until it has been finalised and published, preferably in a peer-reviewed journal. Therefore, a demand to work openly and describe all reasoning and data already from the beginning is often seen as an unreasonable requirement, and it is a major reason for lack of participation by experts. This observation has raised two opposite conclusions: either that openness should be promoted actively in all research and expert work, including decision support; or that openness as an objective is unnecessary and hinders expert work.

We have heard several objections against openness. People are concerned that expertise is not given proper weight, if open participation is allowed. People fear that strong lobbying groups hijack the process. People fear that self-organised groups produce low-quality information or even malevolent dis-information. Experts often demand the final say as the ultimate quality criteria, rather than trusting that data, reasoning, and criticism would do a better job. In brief, experts commonly think that it is simply easier and more efficient to produce high-quality information in closed groups.

An example of this is the website Integrated Environmental Health Impact Assessment System (IEHIAS) that was created by two EU-funded projects, Intarese and Heimtsa. During the projects there was discussion about whether the website should be open (and integrated with Opasnet) or closed (and run separately) when it was built. Clear majority of researchers wanted to work in a closed system and especially retain control over their pages until they were finalised. But at the end of the projects, all content was unanimously published with an open license, and a few years later when the maintenance funding ended, all content was moved from IEHIAS website to Opasnet. In hindsight, developing two different websites with two different platforms and then finally merging them took a lot of extra resources and gave little added value.

The openness of scientific publishing is increasing and many resarch funders demand publishing of data, so this megatrend in the scientific society is changing common practices. It has already been widely acknowledged that the current mainstream of proprietary (as contrast to open access) scientific publishing is a hindrance to spreading ideas and ultimately science[55]. In addition, Wikipedia has demonstrated that self-organised groups can indeed produce high-quality content[56]. Our own experience is the same, and we have not seen hijacking, malevolent behaviour or low-quality junk contributions. We have, however, seen some robots producing unrelated advertisement material on Opasnet pages, but that is easy to identify and remove, and it has not become a problem.

All opinions are critically evaluated in open policy practice, and some of them will be found unsubstantiated. A decision maker is more likely to ignore them if the problems are identified and explicated. However, shared understanding does not contain an idea that proponents of unsubstantiated thoughts should be forced or pressured to reject them, it merely points out these discrepancies and thus nudges participants away from them. It also aims to inform the public so that it can put political pressure against poor ideas. Indeed, one of the ideas behind the method is that it should be good at identifying poor ideas rather than having power to distinguish the best decision option among other good ones.

Sometimes people tend to stick to their original ideas. This is seen in politics, where a key purpose of information is to justify existing opinions rather than adjust them by learning[1]; and as acquiescence, i.e. situations where people know that their choice is irrational but they choose it anyway[57]. Shared understanding may help in this, as the content of an opinion and the reasons to reject it become more explicit and external, and thus the opinion may be more easily affected by a person themself or by others. Research should be performed on this particular topic.

Shared understanding has been a well accepted idea among decision makers in Finland. This was observed in collaboration with Prime Minister's Office of Finland, when they soon adopted the light version of the term and started to use it in their own discussions and publications[58][59]. Sometimes it is better to aim to understanding rather than consensus, and this idea was well accepted.

As mentioned before, accumulation of scientific merit is a key motivator, and established processes, such as journals and scientific committees fulfil this purpose. To maximise the societal impact, an expert should possibly write to Wikipedia rather than a scientific specialist journal. However, such activity gives little merit. Indeed, only 7 % of people contributing to Wikipedia do it for professional reasons[60]. Open assessments seem to suffer a similar problem as non-established scientific endeavour.

Because presenting also controversial and unpopular ideas is a prerequisite for a complete shared understanding, it is very important that such activity is encouraged and respected. A community producing shared understanding should cherish such attitude and promote contributions even if they are incompatible with scientific or another paradigm. This is challenging to both someone presenting such opinions and someone else personally against the presented opinion. The situation requires that all parties have faith in the process and its capability to produce fair conclusions. Therefore, acceptability of open, editable pages and contributions to such pages should be promoted and motivated as a part of method development.

Efficiency

Assessment work is typically mostly needed for collecting and processing data, and making syntheses. The resources needed to do a single assessment online vs. offline are not that different. R has lots of packages that increase efficiency because they are openly shared. Code can be shared, but a spreadsheet model would be understandable by a larger group. However, we have noticed that although spreadsheet models could be shared, in practice they have typically been designed for single users, and model interfaces are poor or non-existent. This is a major problem for efficiency. Also, modelling, co-creation, and/or archiving functionalities are more developed and thus efficient in R. Irrespective of platform, model documentation is key to ensure availability and usability of assessments and thus efficiency.

We have put effort in developing reusable modules and data sources for environmental modelling. The work has been successful in producing building blocks that have been used in several assessments about e.g. fish consumption and thus reduced the marginal costs of a new assessment about a related topic (see Table 8). However, the use of these modelling blocks has not been large outside THL, so there is much more potential for inter-assessment efficiency. Again, the main challenge is about building a community for assessments and decision support and also include people who can write code and model.

An interesting new development in collaborative modelling is Carbon Neutral Helsinki 2035 action plan, which has stated already in the beginning that they will implement open policy practice and use open models for impact assessment and monitoring of the action plan. The planning phase of the action plan already showed potential in creating a community of authorities and experts. The implementation phase, which started in December 2018, aims to involve also citizens, decision-makers, and other municipalities. It is the largest case study so far testing the methods presented in this article. If successful, it will combine existing and produce new climate emission models for municipalities, thus increasing efficiency and capabilities of municipalities in Finland and abroad.

A major issue in spreading practices to new users is that there is a lot to learn before these tools can be used. Even experts with substantive knowledge need time to familiarise with the modelling language used unless they know it already. This has lead to a situation where the lack of common modelling language hampers cooperation of experts on a detailed level. The most common solution that we have seen is strict division of tasks, specified data interfaces, and sequential work. In practice, one group produces data about e.g. emissions of a pollutant, another group takes that data and calculates exposures, and a third group estimates health impacts after the first two steps are finalised. Detailed understanding of and contributions to other groups' work and models remain low or non-existent. On the other hand, most researchers are happy in their own niche and don't expect that other experts could or should learn the details of their work. Consequently, the need for shared tools is often considered low.

Due to these reasons, facilitation is necessary to increase understanding across disciplines and reuse of tools among experts. It is also necessary to improve the readability of the content, help non-experts to understand complex issues, and keep discussion on focus. All this increases the usability of open assessments and makes the next assessment easier.

Jupyter notebooks aim to alleviate these problems (jupyter.org). It is an open source system where data, code, results, graphs, and text can be combined into executable online articles. They can be used to produce knowledge crystals or other information objects. They may contain either R or python code, thus being more versatile than Opasnet. Their popularity is also increasing rapidly among researchers, and therefore they may become an important tool in increasing inter-assessment efficiency.

So far, archetypes and paradigms have not been used to explicitly predict citizens' views on societal decisions and then used those as decision criteria. Interestingly, there is a new political movement (Liike Nyt https://liikenyt.fi/) in Finland that claims that their member of parliament will vote whatever a public online discussion concludes. However, they are - at least so far - not using information tools such as archetypes or paradigms to synthesise public discussions.

The popularity of voting advice applications demonstrates that there is a societal need for value analysis and aggregation. There are straightforward and fairly successful analyses classifying people into clusters along two dimensions based on their answers in these applications: conservativeness vs liberalism, and left-wing vs right-wing. These simple demonstrations imply that at least some major political clusters can be identified and used as archetypes based on such data. With more nuanced data, a set of archetypes describing common and important values in the population can probably be developed. We hypothesize that some of them have potential to increase in popularity and form kind of virtual parties that represent population's key values, while archetypes about less important values will not raise in a similar way.

Also, we hypothesise that only a few major paradigms will emerge, and those are ones whose applicability is wide and independent of the discipline. Scientific paradigm is expected to be one of them, and it will be interesting to see what else emerges. People commonly reason against some unintuitive rules of the scientific method (e.g. they try to prove a hypothesis right rather than wrong) but it is not clear whether this will cause a need to develop a paradigm for an alternative approach. It is even not clear whether people are willing to accept the idea that there could be different, competing rules for reasoning in a single assessment or decision support process.

Discussion

The experience about open policy practice demonstrates that the method works as expected if the participants are committed to the methods, practices, and tools. However, there have been less participants in most open assessments that what was hoped for, and the number of experts or decision makers who actively read from or contribute to Opasnet website has remained lower than expected, although 90000 pageloads per year is still a fair amount. This is partly due to insufficient marketing, as reader numbers have gone clearly up with assessments that have gained media coverage and public interest (e.g. the transport and communication strategy).

The principles behind open policy practice are not unique; on the contrary, they have mostly been borrowed from good practices of various disciplines. Many principles from the original collection[10] have increased in popularity. Openness in science is a current megatrend, and its importance has been accepted much more widely than what was the case in 2006 when Opasnet was launched. Not surprisingly, there are several other websites and organisations that promote one or more of the same principles. Some of them are described here.

Open Science Framework is a project that aims to increase reproducibility in science by developing structured protocols for reproducing research studies, documenting study designs and results online, and producing open source software and preprint services to support this[61]. The Framework maintains a web-workspace for documenting research as it unfolds rather than only afterwards in articles.

Omidyar Network is an organisation that gives grants to non-profit organisations and also invest in startups that promote e.g. governance and citizen engagement[62]. As an example, they support tools to improve discussion online with annotations[63], an objective similar to with structured discussions.

ArXiv.org is a famous example of preprint servers offering a place for publishing and discussing manuscripts before peer review[64]. Such websites, as well as open access journals, have increased during recent years as the importance of availability of scientific information has been understood. Using open data storages (ida.fairdata.fi) for research results are often required by research funders. Also governments have been active in opening data and statistics to wide use (data.gov.uk). Governance practices have been developed towards openness and inclusiveness, promoted by international initiatives such as Open Government Partnership (www.opengovpartnership.org). In brief, facilities for openness and inclusiveness in science and governance are increasing rapidly, and with the current rate, the practices will change radically in the next ten years.

Also non-research domains show increasing openness. Shared online tools such as Google Drive (drive.google.com), Slack (slack.com), and others have familiarised people with online collaboration and idea that information is accessible from anywhere. Open platforms for deliberation of decisions are available (otakantaa.fi, kansalaisaloite.fi), and sharing of code is routinely done via large platforms (github.com, cran.r-project.org). The working environment has changed much faster than the practices in research and societal decision making.

As an extreme, a successful hedge fund Bridgewater Associates implements radical openness and continuous criticism of all ideas presented by its workers rather than letting organisational status determine who is heard[65]. In a sense, they are implementing the scientific method in much more rigorous way than what is typically done in science.

All in all, despite challenges in practical implementation, the principles found in open policy practice have gained popularity and produced success stories in many areas, and they are more timely than before.

Despite all this progress, the websites and tools mentioned above do not offer a place for open topic-wise scientific information production and discussion that would also support decision making. This could be achieved by merging the functionalities of e.g. Opasnet, Open Science Framework, open data repositories, and discussion forums. Even if different tasks would happen on separate websites, they could form an integrated system (by using e.g. permanent resource locations) to be used by decision makers, experts, stakeholders, and machines. Resource description framework and ontologies could be helpful in organising such a complex system.

To keep expert and decision making practices up to the recent progress, there is a need for tools and also training designed to facilitate a change. New practices could also be promoted by developing ways to give scientific - or political - merit and recognition more directly based on participation in co-creation online. The current publication counts and impact factors - or votes - are very indirect measures of societal or scientific importance of the information or policies produced.

In this article, we have demonstrated methods and practices that have already been successfully used in decision support. However, there are many parts that have been thought as important parts in open policy practice but that have not yet been extensively tested. There is still a lot to learn about using co-created information in decision making. However, experiences so far have demonstrated that decision making can be more evidence-based than what it is today, and several tools promoting this change are available to us. Open policy practice may not be able to choose the best alternative among good ones, but it is more effective in identifying and rejecting poor alternatives, which is often more important. This is expected to reduce the influence of a single leader or decision maker, resulting in more stable and predictable policies.

Conclusions

In conclusion, we have demontrated that open policy practice works technically as expected. Open assessments can be performed openly online. They do not fail due to reasons many people think they will, namely low quality contributions, malevolent attacks or chaos caused by too many uninformed participants; these phenomena are very rare. Shared understanding has proved to be a useful concept that guides policy processes toward more collaborative approach, whose purpose is wider understanding rather than winning.

However, open policy practice has not been adopted in expert work or decision support as expected. Key hindrances have been that it offers little added efficiency or quality of content for a single task by an expert or a decision maker, although its impacts on the overall process are positive. The increased availability, acceptability, and inter-assessment efficiency have not been recognised by the scientific or policy community.

Active facilitation, community building and improving the user-friendliness of the tools were identified as key solutions in improving usability of the method in the future.

List of abbreviations

  • THL: National Institute for Health and Welfare (government research institute in Finland)
  • IEHIAS: Integrated Environmental Health Impact Assessment System (a website)
  • RDF: resource description framework

Declarations

  • Ethics approval and consent to participate: Not applicable
  • Consent for publication: Not applicable
  • Availability of data and materials: The datasets generated and/or analysed during the current study are available in the Opasnet repository, http://en.opasnet.org/w/Open_policy_practice
  • Competing interests: The authors declare that they have no competing interests.
  • Funding: This work resulted from the BONUS GOHERR project (Integrated governance of Baltic herring and salmon stocks involving stakeholders, 2015-2018) that was supported by BONUS (Art 185), funded jointly by the EU, the Academy of Finland and and the Swedish Research Council for Environment, Agricultural Sciences and Spatial Planning. Previous funders of the work: Centre of Excellence for Environmental Risk Analysis 2002-2007 (Academy of Finland), Beneris 2006-2009 (EU FP6 Food-CT-2006-022936), Intarese 2005-2011 (EU FP6 Integrated project in Global Change and Ecosystems, project number 018385), Heimtsa 2007-2011 EU FP6 (Global Change and Ecosystems project number GOCE-CT-2006-036913-2), Plantlibra 2010-2014 (EU FP7-KBBE-2009-3 project 245199), Urgenche 2011-2014 (EU FP7 Call FP7-ENV-2010 Project ID 265114), Finmerac 2006-2008 (Finnish Funding Agency for Innovation TEKES), Minera 2010-2013 (European Regional Development Fund), Scud 2005-2010 (Academy of Finland, grant 108571), Bioher 2008-2011 (Academy of Finland, grant 124306), Claih 2009-2012 (Academy of Finland, grant 129341), Yhtäköyttä 2015-2016 (Prime Minister's Office, Finland), Ympäristöterveysindikaattori 2018 (Ministry of Social Affairs and Health).
  • Authors' contributions: JT and MP jointly developed the open assessment method and open policy practice. JT launched Opasnet web-workspace and supervised its development. TR developed OpasnetUtils software package from an original idea by JT and implemented several assessment models. All authors participated in several assessments and discussions about methods. JT wrote the first manuscript draft based on materials from MP, and TR. All authors read and approved the final manuscript.
  • Acknowledgements: We thank Einari Happonen and Juha Villman for their work in developing Opasnet; and Arja Asikainen, John S. Evans, Alexanda Gens, Patrycja Gradowska, Päivi Haapasaari, Sonja-Maria Ignatius, Suvi Ignatius, Matti Jantunen, Anne Knol, Sami Majaniemi, Päivi Meriläinen, Kaisa Mäkelä, Raimo Muurinen, Jussi Nissilä, Juha Pekkanen, Mia Pihlajamäki, Teemu Ropponen, Kalle Ruokolainen, Simo Sarkki, Marko Tainio, Peter Tattersall, Hanna Tuomisto, Jouko Tuomisto, Matleena Tuomisto, and Pieta Tuomisto for crucial and inspiring discussions about methods and their implementation, and promoting these ideas on several forums.

Endnotes

a This paper has its foundations on environmental health, but the idea of decision support necessarily looks at aspects seen relevant from the point of view of the decision maker, not from that of an expert in a particular field. Therefore, this article and also the method described are deliberately taking a wide view and covering all areas of expertise. However, all practical case studies have their main expertise needs in public health, and often specifically in environmental health. b Whenever this article presents a term in italic (e.g. open assessment), it indicates that there is a page on the Opasnet web-workspace describing that term and that it can be accessed using a respective link (e.g. http://en.opasnet.org/w/Open_assessment). c Insight network was originally called pyrkilo (and at some point also extended causal diagram). The word and concept pyrkilo was coined in 1997. It is Finnish and a free translation is "an object or process that tends to produce or aims at producing certain kinds of products." The reasoning for using the word was that pyrkilo diagrams tend to improve understanding and thus decisions. The first wiki website was also called Pyrkilo, but the name was soon changed to Opasnet. d The database consists of two parts: MongoDB contains the actual data, and a related MySQL database contains metadata about all tables. The first version of the database used only MySQL but it was not optimal for data with no predefined structure.

References and notes

  1. 1.0 1.1 1.2 Jussila H. Päätöksenteon tukena vai hyllyssä pölyttymässä? Sosiaalipoliittisen tutkimustiedon käyttö eduskuntatyössä. [Supporting decision making or sitting on a shelf? The use of sociopolitical research information in the Finnish Parliament.] Helsinki: Sosiaali- ja terveysturvan tutkimuksia 121; 2012. http://hdl.handle.net/10138/35919. Accessed 4 Jan 2019. (in Finnish)
  2. National Research Council. Risk Assessment in the Federal Government: Managing the Process. Washington DC: National Academy Press; 1983.
  3. 3.0 3.1 National Research Council. Understanding risk. Informing decisions in a democratic society. Washington DC: National Academy Press; 1996.
  4. 4.0 4.1 Pohjola MV, Leino O, Kollanus V, Tuomisto JT, Gunnlaugsdóttir H, Holm F, Kalogeras N, Luteijn JM, Magnússon SH, Odekerken G, Tijhuis MJ, Ueland O, White BC, Verhagen H. State of the art in benefit-risk analysis: Environmental health. Food Chem Toxicol. 2012;50:40-55.
  5. Tapscott D, Williams AD. Wikinomics. How mass collaboration changes everything. USA: Portfolio; 2006. ISBN 1591841380
  6. Surowiecki J. The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. USA: Doubleday; Anchor; 2004. ISBN 9780385503860
  7. Noveck, BS. Wiki Government - How Technology Can Make Government Better, Democracy Stronger, and Citizens More Powerful. Brookings Institution Press; 2010. ISBN 9780815702757
  8. 8.0 8.1 Co-creation: Mauser W, Klepper G, Rice M, Schmalzbauer BS, Hackmann H, Leemans R, Current HM. Transdisciplinary global change research: the co-creation of knowledge for sustainability. Opinion in Environmental Sustainability 2013;5:420–431; doi:10.1016/j.cosust.2013.07.001
  9. Pohjola M. Assessments are to change the world. Prerequisites for effective environmental health assessment. Helsinki: National Institute for Health and Welfare Research 105; 2013. http://urn.fi/URN:ISBN:978-952-245-883-4
  10. 10.0 10.1 10.2 Tuomisto JT, Pohjola M, editors. Open Risk Assessment. A new way of providing scientific information for decision-making. Helsinki: Publications of the National Public Health Institute B18; 2007. http://urn.fi/URN:ISBN:978-951-740-736-6.
  11. 11.0 11.1 11.2 Assessment: http://en.opasnet.org/w/Goherr_assessment. Accessed 4 Jan 2019.
  12. 12.0 12.1 Tuomisto JT. TCDD: a challenge to mechanistic toxicology [Dissertation]. Kuopio: National Public Health Institute A7; 1999.
  13. 13.0 13.1 Tuomisto JT, Pekkanen J, Alm S, Kurttio P, Venäläinen R, Juuti S et al. Deliberation process by an explicit factor-effect-value network (Pyrkilo): Paakkila asbestos mine case, Finland. Epidemiol 1999;10(4):S114.
  14. 14.0 14.1 Tuomisto JT; Tainio M. An economic way of reducing health, environmental, and other pressures of urban traffic: a decision analysis on trip aggregation. BMC PUBLIC HEALTH 2005;5:123. http://biomedcentral.com/1471-2458/5/123/abstract Assessment: http://en.opasnet.org/w/Cost-benefit_assessment_on_composite_traffic_in_Helsinki. Accessed 4 Jan 2019.
  15. Allison DB, Brown AW, George BJ, Kaiser KA. Reproducibility: A tragedy of errors. Nature 2016;530:27–29. doi:10.1038/530027a
  16. Aitamurto T, Landemore H. Five design principles for crowdsourced policymaking: Assessing the case of crowdsourced off-road traffic law in Finland. Journal of Social Media for Organizations. 2015;2:1:1-19.
  17. von Winterfeldt D. Bridging the gap between science and decision making. PNAS 2013;110:3:14055-14061. http://www.pnas.org/content/110/Supplement_3/14055.full
  18. Open Knowledge International. The Open Definition. http://opendefinition.org/. Accessed 4 Jan 2019.
  19. Pohjola MV, Pohjola P, Tainio M, Tuomisto JT. Perspectives to Performance of Environment and Health Assessments and Models—From Outputs to Outcomes? (Review). Int. J. Environ. Res. Public Health 2013;10:2621-2642 doi:10.3390/ijerph10072621
  20. 20.0 20.1 Sandström V, Tuomisto JT, Majaniemi S, Rintala T, Pohjola MV. Evaluating effectiveness of open assessments on alternative biofuel sources. Sustainability: Science, Practice & Policy 2014;10;1. doi:10.1080/15487733.2014.11908132 Assessment: http://en.opasnet.org/w/Biofuel_assessments. Accessed 4 Jan 2019.
  21. Cooke RM. Experts in Uncertainty: Opinion and Subjective Probability in Science. New York: Oxford University Press; 1991.
  22. Tuomisto JT, Pohjola M, Pohjola P. Avoin päätöksentekokäytäntö voisi parantaa tiedon hyödyntämistä. [Open policy practice could improve knowledge use.] Yhteiskuntapolitiikka 2014;1:66-75. http://urn.fi/URN:NBN:fi-fe2014031821621 (in Finnish)
  23. Pohjola MV. Assessment of impacts to health, safety, and environment in the context of materials processing and related public policy. In: Bassim N, editor. Comprehensive Materials Processing Vol. 8. Elsevier Ltd; 2014. pp 151–162. doi:10.1016/B978-0-08-096532-1.00814-1
  24. Pohjola MV, Tuomisto JT. Openness in participation, assessment, and policy making upon issues of environment and environmental health: a review of literature and recent project results. Environmental Health 2011;10:58 http://www.ehjournal.net/content/10/1/58.
  25. van Kerkhoff L, Lebel L. Linking knowledge and action for sustainable development. Annu. Rev. Environ. Resour. 2006. 31:445-477. doi:10.1146/annurev.energy.31.102405.170850
  26. Eemeren FH van, Grootendorst R. A systematic theory of argumentation: The pragma-dialectical approach. Cambridge: Cambridge University Press; 2004.
  27. Eemeren FH van. Reasonableness and effectiveness in argumentative discourse. Fifty contributions to the development of pragma-dialectics. Springer International Publishing Switzerland, 2015. ISBN 978-3-319-20954-8. doi:10.1007/978-3-319-20955-5
  28. Schwartz SH, Cieciuch J, Vecchione M, Davidov E, Fischer R, Beierlein C, Ramos A, Verkasalo M, Lönnqvist J-E. Refining the theory of basic individual values. Journal of Personality and Social Psychology. 2012: 103; 663–688. doi: 10.1037/a0029393.
  29. 29.0 29.1 Esitys Hiilineutraali Helsinki 2035 -toimenpideohjelmaksi. [Suggestion for an action plan for carbon neutral Helsinki 2035] Publications of the central administration of city of Helsinki 4/2018. [in Finnish] https://www.hel.fi/static/liitteet/kaupunkiymparisto/julkaisut/julkaisut/HNH-2035-toimenpideohjelma.pdf Assessment: http://fi.opasnet.org/fi/Hiilineutraali_Helsinki_2035. Accessed 4 Jan 2019.
  30. W3C. Resource Description Framework (RDF). https://www.w3.org/RDF/. Accessed 6 Sep 2018.
  31. Assessment: http://en.opasnet.org/w/Assessment_of_the_health_impacts_of_H1N1_vaccination. Accessed 4 Jan 2019.
  32. Assessment: http://en.opasnet.org/w/Tendering_process_for_pneumococcal_conjugate_vaccine. Accessed 4 Jan 2019.
  33. Tuomisto JT, Rintala J, Ordén P, Tuomisto HM, Rintala T. Helsingin energiapäätös 2015. Avoin arviointi terveys-, ilmasto- ja muista vaikutuksista. [Helsinki energy decision 2015. An open assessment on health, climate, and other impacts]. Helsinki: National Institute for Health and Welfare. Discussionpaper 24; 2015. http://urn.fi/URN:ISBN:978-952-302-544-8 Assessment: http://en.opasnet.org/w/Helsinki_energy_decision_2015. Accessed 4 Jan 2019.
  34. Asikainen A, Pärjälä E, Jantunen M, Tuomisto JT, Sabel CE. Effects of Local Greenhouse Gas Abatement Strategies on Air Pollutant Emissions and on Health in Kuopio, Finland. Climate 2017;5(2):43; doi:10.3390/cli5020043 Assessment: http://en.opasnet.org/w/Climate_change_policies_and_health_in_Kuopio. Accessed 4 Jan 2019.
  35. Tuomisto JT, Niittynen M, Pärjälä E, Asikainen A, Perez L, Trüeb S, Jantunen M, Künzli N, Sabel CE. Building-related health impacts in European and Chinese cities: a scalable assessment method. Environmental Health 2015;14:93. doi:10.1186/s12940-015-0082-z Assessment: http://en.opasnet.org/w/Climate_change_policies_in_Basel. Accessed 4 Jan 2019.
  36. Taimisto P, Tainio M, Karvosenoja N, Kupiainen K, Porvari P, Karppinen A, Kangas L, Kukkonen J, Tuomisto JT. Evaluation of intake fractions for different subpopulations due to primary fine particulate matter (PM2.5) emitted from domestic wood combustion and traffic in Finland. Air Quality Atmosphere and Health 2011;4:3-4:199-209. doi:10.1007/s11869-011-0138-3 Assessment: http://en.opasnet.org/w/BIOHER_assessment. Accessed 4 Jan 2019.
  37. Tainio M, Tuomisto JT, Hanninen O, Aarnio P, Koistinen, KJ, Jantunen MJ, Pekkanen J. Health effects caused by primary fine particulate matter (PM2.5) emitted from buses in the Helsinki metropolitan area, Finland. RISK ANALYSIS 2005;25:1:151-160. Assessment: http://en.opasnet.org/w/Gasbus_-_health_impacts_of_Helsinki_bus_traffic. Accessed 4 Jan 2019.
  38. Tuomisto JT, Niittynen M, Turunen A, Ung-Lanki S, Kiviranta H, Harjunpää H, Vuorinen PJ, Rokka M, Ritvanen T, Hallikainen A. Itämeren silakka ravintona – Hyöty-haitta-analyysi. [Baltic herring as food - a benefit-risk analysis] ISBN 978-952-225-141-1. Helsinki: Eviran tutkimuksia 1; 2015 (in Finnish). Assessment: http://fi.opasnet.org/fi/Silakan_hy%C3%B6ty-riskiarvio. Accessed 4 Jan 2019.
  39. Leino O, Karjalainen AK, Tuomisto JT. Effects of docosahexaenoic acid and methylmercury on child's brain development due to consumption of fish by Finnish mother during pregnancy: A probabilistic modeling approach. Food Chem Toxicol. 2013;54:50-8. doi:10.1016/j.fct.2011.06.052. Assessment: http://en.opasnet.org/w/Benefit-risk_assessment_of_methyl_mercury_and_omega-3_fatty_acids_in_fish. Accessed 4 Jan 2019.
  40. Gradowska PL. Food Benefit-Risk Assessment with Bayesian Belief Networks and Multivariable Exposure-Response. Delft: Delft University of Technology (doctoral dissertation); 2013. https://repository.tudelft.nl/islandora/object/uuid:9ced4cb2-9809-4b58-af25-34e458e8ea23/datastream/OBJ Assessment: http://en.opasnet.org/w/Benefit-risk_assessment_of_fish_consumption_for_Beneris. Accessed 4 Jan 2019.
  41. Tuomisto JT, Tuomisto J, Tainio M, Niittynen M, Verkasalo P, Vartiainen T et al. Risk-benefit analysis of eating farmed salmon. Science 2004;305(5683):476. Assessment: http://en.opasnet.org/w/Benefit-risk_assessment_on_farmed_salmon. Accessed 4 Jan 2019.
  42. Leino O, Tainio M, Tuomisto JT. Comparative risk analysis of dioxins in fish and fine particles from heavy-duty vehicles. Risk Anal. 2008;28(1):127-40. Assessment: http://en.opasnet.org/w/Comparative_risk_assessment_of_dioxin_and_fine_particles. Accessed 4 Jan 2019.
  43. Assessment: http://en.opasnet.org/w/Compound_intake_estimator. Accessed 4 Jan 2019.
  44. Kauppila T, Komulainen H, Makkonen S, Tuomisto JT, editors. Metallikaivosalueiden ympäristöriskinarviointiosaamisen kehittäminen: MINERA-hankkeen loppuraportti. [Summary: Improving Environmental Risk Assessments for Metal Mines: Final Report of the MINERA Project.] Helsinki: Geology Survey Finland, Research Report 199; 2013. 223 p. ISBN 978-952-217-231-0. Assessment: http://fi.opasnet.org/fi/Minera-malli. Accessed 4 Jan 2019.
  45. Assessment: http://fi.opasnet.org/fi/Kaivosvesien_riskit_(KAVERI-malli). Accessed 4 Jan 2019.
  46. Assessment: http://en.opasnet.org/w/Water_guide. Accessed 4 Jan 2019.
  47. Liikenne ja viestintä digitaalisessa Suomessa. Liikenne- ja viestintäministeriön tulevaisuuskatsaus 2014 [Transport and and communication in digital Finland] Helsinki: Ministry of Transport and Communication; 2014. http://urn.fi/URN:ISBN:978-952-243-420-3 Assessment: http://fi.opasnet.org/fi/Liikenne_ja_viestint%C3%A4_digitaalisessa_Suomessa_2020. Accessed 4 Jan 2019.
  48. Tuomisto JT, Muurinen R, Paavola J-M, Asikainen A, Ropponen T, Nissilä J. Tiedon sitominen päätöksentekoon. [Binding knowledge to decision making] Helsinki: Publications of the Government's analysis, assessment and research activities 39; 2017. ISBN 978-952-287-386-6 http://tietokayttoon.fi/julkaisu?pubid=19001. Assessment: http://fi.opasnet.org/fi/Maahanmuuttoarviointi. Accessed 4 Jan 2019.
  49. Kajanus M, Ollikainen T, Partanen J, Vänskä I. Kävijätutkimukseen perustuva Puijon virkistysmetsien hoito- ja käyttösuunnitelma. [Forest strategy for recreational forests at Puijo, Kuopio, based on visitor study.] (in Finnish) Kuopion kaupunki, Metsätoimisto; 2010. http://fi.opasnet.org/fi-opwiki/images/8/8a/Puijo-loppuraportti.pdf. Assessment: http://fi.opasnet.org/fi/Puijon_metsien_k%C3%A4ytt%C3%B6suunnitelman_p%C3%A4%C3%A4t%C3%B6ksenteko Accessed 4 Jan 2019.
  50. Jouni T. Tuomisto, Arja Asikainen, Antti Korhonen, Heli Lehtomäki. Teemasivu ympäristöterveys [Portal: Environmental health]. A website, THL, 2018. [1]
  51. Eeva Ketola, Annaleena Okuloff et al. Indicators for social and health phenomena. Website, THL, 2018. [2]
  52. European Parliament. Directive 2014/52/EU of the European Parliament and of the Council of 16 April 2014 amending Directive 2011/92/EU on the assessment of the effects of certain public and private projects on the environment Text with EEA relevance. https://eur-lex.europa.eu/eli/dir/2014/52/oj Accessed 4 Jan 2019.
  53. Wikipedia: Wikipedians. https://en.wikipedia.org/wiki/Wikipedia:Wikipedians. Accessed 4 Jan 2019
  54. Star SL, Griesemer JR. Institutional Ecology, 'Translations' and Boundary Objects: Amateurs and Professionals in Berkeley's Museum of Vertebrate Zoology, 1907-39. Social Studies of Science, 1989; 19 387-420.
  55. Eysenbach G. Citation Advantage of Open Access Articles. PLoS Biol 2006: 4; e157. doi: 10.1371/journal.pbio.0040157
  56. Giles J. Internet encyclopaedias go head to head. Nature 2005;438:900–901 doi:10.1038/438900a
  57. Walco DK, Risen, JL. The Empirical Case for Acquiescing to Intuition. PSYCHOLOGICAL SCIENCE 2017;28:12:1807-1820. doi:10.1177/0956797617723377
  58. Dufva M, Halonen M, Kari M, Koivisto T, Koivisto R, Myllyoja J. Kohti jaettua ymmärrystä työn tulevaisuudesta [Toward a shared understanding of the future of work]. Helsinki: Prime Minister's Office: Publications of the Govenrment's analysis, assessment and research activities 33; 2017. (in Finnish) http://tietokayttoon.fi/julkaisu?pubid=18301. Accessed 4 Jan 2019.
  59. Oksanen K. Valtioneuvoston tulevaisuusselonteon 1. osa. Jaettu ymmärrys työn murroksesta [Government Report on the Future Part 1. A shared understanding of the transformation of work] Prime Minister’s Office Publications 13a; 2017. (in Finnish) http://urn.fi/URN:ISBN:978-952-287-432-0. Accessed 4 Jan 2019.
  60. Pande M. Wikipedia editors do it for fun: First results of our 2011 editor survey. 2011. https://blog.wikimedia.org/2011/06/10/wikipedia-editors-do-it-for-fun-first-results-of-our-2011-editor-survey/. Accessed 4 Jan 2019.
  61. Open Science Framework. https://osf.io/. Accessed 4 Jan 2019.
  62. Omidyar Network. A world of positive returns. http://www.omidyar.com. Accessed 4 Jan 2019.
  63. Hypothesis. Annotate the web, with anyone, anywhere. https://web.hypothes.is/. Accessed 4 Jan 2019.
  64. Cornell University Library. arXiv.org. https://arxiv.org/. Accessed 4 Jan 2019.
  65. Dalio R. Principles: Life and work. New York: Simon & Shuster; 2017. ISBN 9781501124020

Figures and tables

Move them here for submission.

Appendix 1: Examples of insight networks

Appendix 2: Open policy ontology

Shared understanding aims at producing a description of different views, opinions, and facts related to a specific topic such as a decision process. The open policy ontology describes the information structures that are needed to document shared understanding of a complex decision situation. The purpose of the structure is to help people identify hidden premises, beliefs, and values and explicate possible discrepancies. This is expected to produce better understanding among participants.

The basic structure of a shared understanding is a network of items and relations between them. This network uses Resource description framework, which is an ontology standard used to describe many Internet contents. Items and relations (aka properties) are collectively called resources. Each item is typically of one of the types mentioned below. This information is documented using property instance of (e.g. Goherr assessment is instance of assessment).

Items are written descriptions of the actual things (people, tasks, publications, or phenomena), and on this page we discuss these descriptions rather than the real things. Different item types have different levels of standardisation and internal structure. For example, knowledge crystals are web pages that always have headings question, answer and rationale, and the information is organised under those headings. Some other items describe e.g. statements that are free-text descriptions about how a particular thing is or should be (according to a participant), and yet some others are metadata about publications. A common feature is that all items contain information that is relevant for a decision.

In the open policy ontology, each item may have lengthy texts, graphs, analyses or even models inside them. However, the focus here is on how the items are related to each other. The actual content is often referred to as one key sentence only (description). Each item also has a unique identifier URI that is used for automatic handling of data.

The most important items are knowledge crystals and they are described here.

  • Assessment describes a particular decision situation and focuses on estimating impacts of different options. Its purpose is to support the making of that decision. Unlike other knowledge crystals, assessments typically have a defined start and end dates and they are closed after the decision is made. They also have contextually and situationally defined goals`to be able to better serve the needs of the decision makers of the decision.
  • Variable answers a particular factual or ethical question that is typically needed in one or more assessments. The answer of a variable is continually updated as new information arises, but its question remains constant in time. Variable is the basic building block of describing reality, i.e. how things are or should be. In R, variables are typically implemented using ovariable objects from OpasnetUtils package.
  • Method tells how to systematically implement a particular information task. Method is the basic building block for describing the assessment work (not reality, like variables). In practice, methods are "how-to-do" descriptions about how information should be produced, collected, analysed, or synthesised in an assessment. Typically, methods contain a software code or another algorithm to actually perform the method easily. In R, methods are typically ovariables that require some context-specific upstream information about dependencies before it can be calculated.

There are also other important classes of items:

  • Publication is any documentation that contains useful information related to a decision. Publications that are commonly used in Opasnet include encyclopedia article, lecture, nugget, and study. Other publications in Opasnet are typically uploaded as files.
    • Encyclopedia article is an object that describes a topic rather than answers a specific research question. They do not have a predefined attribute structure.
    • Lecture: Lecture contains a piece of information that is to be mediated to a defined audience and with a defined learning objective. It can also be description of a process during which the audience learns, instead of being a passive recipient of information.
    • Nugget is an object that is not editable by other people than a dedicated author (group) and is not expected to be updated once finalised. They do not have a predefined attribute structure.
    • Study describes a research study and its answers, i.e. observational or other data obtained in the study. The research questions are described as the question of the information object, and the study methods are described as the rationale of the object. Unlike in an article, discussion may be missing, and unlike in a variable, the answer and rationale of the study are more or less fixed after the work is done; this is because the interpretations of the results typically happen elsewhere, like in variables for which the study contains useful information.
  • Discussion is a hierarchically structured documentation of a discussion about a defined statement or statements.
  • Stakeholder page is used to describe a person or group that is relevant for a decision or decision process; they may be an actor that has an active role in decision making or is a target of impacts. Contributors in Opasnet are described on their own user pages; other stakeholders may have their page on the main namespace.
  • Process describes elements of a decision process.
  • Action describes what, who and when should act to e.g. perform an assessment, make a decision, or implement policies.

Relations show different kinds of connections between items.

  • Causal link tells that the subject may change the object (e.g. affects, increases, decreases, prevents).
  • Participatory link describes a stakeholder's particular role related to the object (participates, negotiates, decides).
  • Operational link tells that the subject has some kind of practical relation to the object (executes, offers, tells).
  • Evaluative link tells that the subject shows preference or relevance about the object (has truthlikeness, value, popularity, finds important).
  • Referential link tells that the subject is used as a reference of a kind for the object (makes relevant; associates to; has reference, tag, category).
  • Argumentative link occurs between statements that defend or attack each other (attack, defend, comment).
  • Property link connects an evaluative (acceptability, usability), a logical (opposite, inverse) or set theory (has subclass, has part) property to the subject.

Item types

This ontology is specifically about decision making, and therefore actions (and decisions to act) are handled explicitly. However, any natural, social, ethical or other phenomena may relate to a decision and therefore the vocabulary has to be very generic.

Class English name Finnish name Description
resource resurssi All items and relations are resources
resource item asia Relevant pieces of information related policy making. Sometimes also refers to the real-life things that the information is about. Items are shown as nodes in insight networks.
resource relation relaatio Information about how items are connected to each other. Relations are shown as edges in insight networks.
item substance ilmiö Items about a substantive topic or phenomenon itself: What issues relate to a decision? What causal connections exist between issues? What scientific knowledge exist about the issues? What actions can be chosen? What are the impacts of these actions? What are the objectives and how can they be reached? What values and preferences exist?
item stakeholder sidosryhmä Items about people or organisations who have a particular role in a policy process, either as actors or targets of impacts: Who participates in a policy process? Who should participate? Who has necessary skills for contributing? Who has the authority to decide? Who is affected by a decision?
item process prosessi Items about doing or happening in relation with a topic, especially information about how a decision will be made): What will be decided? When will it be decided? How is the decision prepared? What political realities and restrictions exist?
item action toiminta Items about organising decision support, decision making, implementation, and evaluation: What tasks are needed to collect and organise necessary information? When do these tasks need to be done? Who is responsible of what? How is information work organised? Tasks are also important afterwards to distribute merit and evaluate the process: Who did what? How did information evolve? Where did data come from?
item information object tieto-olio A specified structure containing information about substance, stakeholders, processes, methods, or actions.
information object knowledge crystal tietokide information object with a standardised structure and contribution rules
knowledge crystal assessment arviointi Describes a decision situation and typically provides relevant information to decision makers before the decision is made (or sometimes after the decision about its implementation or success). It is mostly about the knowledge work, i.e. tasks for decision support.
knowledge crystal variable muuttuja Describes a real-world topic that is relevant for the decision situation. It is about the substance of the topic.
knowledge crystal method metodi Describes how information should be managed or analysed so that it will answer the policy-relevant questions asked. How to perform information work? What methods are available for a task? How to participate in a decision process? How to use statistical and other methods and tools? How to motivate participation? How to measure merit of contributions?
information object discussion part keskustelun osa Information object that is used to organise discussions into a specified structure. The purpose of the structure is to help validation of statements and facilitate machine learning.
information object discussion keskustelu Discussion, or structured argumentation, describes arguments about a particular statement and a synthesis about an acceptable statement. In a way, discussion is (a documentation of) a process of analysing the validity of a statement.
discussion fact discussion faktakeskustelu Discussion that can be resolved based on scientific knowledge.
discussion value discussion arvokeskustelu Discussion that can be resolved based on ethical knowledge.
discussion part statement väite Proposition claiming that something is true or ethically good. A statement may be developed in a discussion by adding and organising related argumentation (according to pragma-dialectics), or by organising premises and inference rules (according to Perelman).
statement value statement arvoväite Proposition claiming that something is ethically good, better than something else, prioritised over something, or how things should be.
statement fact statement faktaväite Proposition claiming how things are or that something is true.
value statement true value statement tosi arvoväite A statement that has not been successfully invalidated.
value statement false value statement epätosi arvoväite A statement that has been successfully invalidated.
fact statement true fact statement tosi faktaväite
fact statement false fact statement epätosi faktaväite
statement true statement tosi väite
statement false statement epätosi väite
statement opening statement avausväite A statement that is the basis for a structured discussion, a priori statement.
statement closing statement lopetusväite A statement that is the resolution of a structured discussion, a posteriori statement. Closing statement becomes an opening statement when the discussion is opened again.
opening statement fact opening statement avausfaktaväite
closing statement fact closing statement lopetusfaktaväite
opening statement value opening statement avausarvoväite
closing statement value closing stetement lopetusarvoväite
discussion part argument argumentti A statement that has also contains a relation to its target as an integral part. Due to this relation, arguments appear inside discussions and target directly or indirectly the opening statement.
discussion part argumentation väittely Hierarchical list of arguments related to a particular statement.
information object knowledge crystal part tietokideosa This is shown separately to illustrate that the objects are actually linked by has part rather than has subclass relation.
knowledge crystal part question kysymys A research question asked in a knowledge crystal. The purpose of a knowledge crystal is to answer the question.
knowledge crystal part answer vastaus An answer or set of answers to the question of a knowledge crystal, based on any relevant information and inference rules.
knowledge crystal part rationale perustelut Any data, discussions, calculations or other information needed to convince a critical rational reader that the answer of a knowledge crystal is good.
knowledge crystal part answer part vastausosa This is shown separately to illustrate that the objects are actually linked by has part rather than has subclass relation.
answer part result tulos The actual, often numerical result to the question, conditional on relevant indices.
answer part index indeksi A list of possible values for a descriptor. Typically used in describing the result of an ovariable.
knowledge crystal part ovariable ovariable A practical implementation of a knowledge crystal in modelling code. Ovariable takes in relevant information about data and dependencies and calculates the result. Typically implemented in R using OpasnetUtils package and ovariable object type.
ovariable key ovariable avainovariable An ovariable that is shown on an insight network even if some parts are hidden due to practical reasons.
information object publication julkaisu Any published report, book, web page or similar permanent piece of information that can be unambiguously referenced.
publication nugget tiedomuru An object that is not editable by other people than a dedicated author (group).
substance topic aihe A description of an area of interest. It defines boundaries of a content rather than defines the content itself, which is done by statements. When the information structure is improved, a topic often develops into a question of a knowledge cryatal, while a statement develops into an answer of a variable.
priority objective tavoite A desired outcome of a decision. In shared understanding description, it is a topic (or variable) that has value statements attached to it.
substance risk factor riskitekijä
substance indicator indikaattori Piece of information that describes a particular substantive item in a practical and often standard way.
indicator risk indicator riski-indikaattori Indicator about (health) risk or outcome
information object data tietoaineisto
information object graph kuvaaja Graphical representation of a piece of information. Typically is related to an information object with describes relation.
work data work tietotyö
work data use tiedon käyttö
substance priority prioriteetti
substance expense kustannus
substance health impact terveysvaikutus
stakeholder decision maker päättäjä
stakeholder public officer virkamies
stakeholder assessor arvioija
stakeholder expert asiantuntija
stakeholder citizen kansalainen
stakeholder agent toimija
action task toimenpide action to be taken when the option has been selected
action decision päätös action to be taken when the option is yet to be selected. Describes a particular event where a decision maker chooses among defined alternatives. This may also be a part of an assessment under heading Decisions and scenarios.
action work työ continuous actions of the same kind and typically independent of the decision at hand. If the decision changes work routines, the action to make this change happen is called task.
work prevention ennaltaehkäisy trying to prevent something
work treatment hoito trying to fix something when something has already happened
work support tuki work that aids in the completion of the selected option, in whatever way
method open policy practice avoin päätöksentekokäytäntö framework for planning, making, and implementing decisions
method open assessment avoin arviointi method answering this question: How can factual and value information be organised for supporting societal decision making when open participation is allowed?
method analysis analyysi
method reporting raportointi
method measurement mittaus
publication study tutkimus
publication encyclopedia article ensyklopedia-artikkeli An object that describes a topic rather than answers a specific research question.
publication lecture luento Contains a piece of information that is to be mediated to a defined audience and with a defined learning objective.
method procedure toimintamalli
method principle periaate a short generic guidance for information work to ensure that the work is done properly. They especially apply to the execution phase.
principle intentionality tavoitteellisuus The decision maker explicates their objectives and decision options under consideration. All that is done aims to offer better understanding about impacts of the decision related to the objectives of the decision maker. Thus, the participation of the decision maker in the decision support process is crucial.
principle causality syysuhteiden kuvaus The focus is on understanding and describing the causal relations between the decision options and the intended outcomes. The aim is to predict what impacts will likely occur if a particular decision option is chosen.
principle criticism kritiikki All information presented can be criticised based on relevance and accordance to observations. The aim is to reject ideas, hypotheses -- and ultimately decision options -- that do not hold against critique. Criticism has a central role in the scientific method, and here we apply it in practical situations, because rejecting poor statements is much easier and more efficient than trying to prove statements true.
principle permanent resource locations kohteellisuus Information is organised around topics (described as research questions), and each topic has a permanent location where it can be found even if the content develops in time. In practice, these locations are webpages with permanent URLs.
principle openness avoimuus All work and all information is openly available to anyone interested for reading and contributing all the time. If there are exceptions, these must be publicly justified. Openness is crucial because a priori it is impossible to know who may have important factual information or value judgements about the topic.
principle reuse uusiokäyttö All information is produced in a format that can easily be used for other purposes by other people. Open data principles are used when possible. For example, some formats such as PDF files are not easily reusable.
principle use of knowledge crystals tietokiteiden käyttö All information is openly shared using a systematic structure (notably question, answer, and rationale) and permanent locations in a common workspace where all participants can work. Knowledge crystals are used for this. The structure of an assessment and its data is based on substance (i.e. causal, logical and other substantive connections between issues). Objectives determine the information needs, which are then used to define research questions to be answered in the assessment. The assessment work is collaboration aiming to answer these questions in a way that holds against critique. Thus, knowledge crystals are practical information structures that comply with other principles of open assessment.
principle grouping ryhmäytyminen Facilitation methods are used to promote the participants' feeling of being an important member of a group that has a meaningful purpose.
principle respect arvostus Contributions are systematically documented and their merit evaluated so that each participant receives the respect they deserve based on their contributions.
objective expense objective kustannustavoite
process step jakso one of sequential time intervals when a particular kind of work is done. In the next step, the nature of the work changes.
step decision support valmistelu the first step in a decision process. Helps in collecting necessary information for making a decision.
step decision making päätöksenteko the second step in a decision process. When the decision makes actually chooses between options.
step implementation toimeenpano the third step in a decision process. When the chosen option is put in action.
process phase vaihe one part of a decision work process where focus is on particular issues or methods. Typically phases overlap temporally.
phase shared understanding jaettu ymmärrys documenting of all relevant views, facts, values, and opinions about a decision situation in such a way that agreements and disagreements can be understood
phase execution toteutus production of necessary information for a decision at hand
phase evaluation and management seuranta ja ohjaus ensuring that all work related to a decision will be, is, and has been done properly
phase co-creation and facilitation yhteenvetämisen taito helping people to participate, contribute, and become motivated about the decision work

Relation types

Class English name Finnish name English inverse Finnish inverse Description
relation participatory link osallisuuslinkki The subject is a stakeholder that has a particular role related to an object
relation operational link toimintolinkki The subject has some kind of practical relation to the object (a fairly wide class)
relation evaluative link arvostuslinkki The subject shows preference of relevance about the object
relation referential link viitelinkki The subject is used as a reference of a kind for the object
relation argumentative link argumentaatiolinkki The subject is used as an argument to criticise the object.
relation causal link syylinkki The subject has causal effect on the object (or vice versa in the case of an inverse relation)
relation property link ominaisuuslinkki The object describes a defined property of the subject.
causal link negative causal link negatiivinen syylinkki The subject reduces or diminishes the object.
causal link positive causal link positiivinen syylinkki The subject increases or enhances the object.
negative causal link decreases vähentää is decreased by vähentyy
positive causal link increases lisää is increased by lisääntyy
negative causal link worsens huonontaa is worsened by huonontuu
positive causal link improves parantaa is improved by parantuu
negative causal link prevents estää is prevented by estyy
positive causal link enhances edistää is enhanced by edistyy
negative causal link impairs heikentää is impaired by heikentyy
positive causal link sustains ylläpitää is sustained by ylläpitäytyy
causal link affects vaikuttaa is affected by vaikuttuu
causal link indirectly affects vaikuttaa epäsuorasti indirectly affected by vaikuttuu epäsuorasti
causal link cause of syy caused by johtuu Wikidata property P1542
causal link immediate cause of välitön syy immediately caused by johtuu välittömästi Wikidata property P1536
causal link contributing factor of vaikuttava tekijä Wikidata property P1537
participatory link performs toteuttaa performer toteuttajana who does a task?
participatory link decides päättää decider päätäjänä
participatory link asks kysyy asker kysyjänä
participatory link participates osallistuu participant osallistujana
participatory link accepts hyväksyy accepted by hyväksyjänä
participatory link develops kehittää developed by kehittäjänä
participatory link proposes ehdottaa proposed by ehdottajana
participatory link answers vastaa answered by vastaajana
participatory link responsible for vastuussa responsibility of vastuullisena
participatory link negotiates neuvottelee negotiated by neuvottelijana
participatory link recommends suosittelee recommended by suosittelijana
participatory link controls kontrolloi controlled by kontrolloijana
participatory link claims väittää claimed by väittäjänä
participatory link owns omistaa owned by omistajana
participatory link does tekee done by tekijänä
participatory link maintains ylläpitää maintained by ylläpitäjänä
participatory link oversees valvoo overseen by valvojana
operational link has option omistaa vaihtoehdon option for vaihtoehtona
operational link has index omistaa indeksin index for indeksinä
operational link tells kertoo told by kertojana
operational link describes kuvaa described by kuvaajana
operational link maps kartoittaa mapped by kartjoittajana
operational link contains data sisältää dataa data contained in data sisältyy
operational link data for on datana gets data from saa datansa
operational link uses käyttää is used by on käytettävänä an input (object) for a process (subject)
operational link produces tuottaa is produced by tuottajana Object is an output of a process produced by a stakeholder (subject)
operational link provides varustaa is provided by varustajana
operational link about aiheesta a task is about a topic. This overlaps with has topic; merge them?
property link logical link looginen linkki Relations based on logic
property link set theory link joukko-oppilinkki Relations based on set theory
set theory link part of osana has part sisältää osan is a part of a bigger entity, e.g. Venus is part of Solar System. Wikidata property P361 (part of) & P527 (has part). Previously we had relations about a decision: substance of, decision process of, stakeholder of, method of, task of, irrelevant to. But these are depreciated and replaced by has part, because the class of the object makes specific relations redundant.
set theory link context for kontekstina has context omistaa kontekstin
set theory link has subclass omistaa alajoukon subclass of alajoukkona Wikidata property P279
set theory link has instance omistaa instanssin instance of instanssina Object belongs to a set defined by the subject and inherits the properties of the set. Sysnonym for has item, which is depreciated. Wikidata property P31
logical link opposite vastakohta subject is opposite of object, e.g. black is opposite of white. Wikidata property P461; it is its own inverse
logical link inverse toisinpäin a sentence is equal to another sentence where subject and object switch places and has the inverse relation. This is typically needed in preprocessing of insight networks, and it rarely is explicitly shown of graphs. Wikidata property P1696; it is its own inverse
logical link if - then jos - niin if not - then not jos ei - niin ei If subject is true, then object is true. Also the negation is possible: if - then not. This links to logical operators and, or, not, equal, exists, for all; but it is not clear how they should be used in an insight network.
operational link prepares valmistelee prepared by valmistelijana
operational link pays kustantaa paid by kustantajana
operational link rationale for perustelee has rationale perusteltuu
operational link offers tarjoaa offered by tarjoajana
operational link executes suorittaa executed by suorittajana
operational link irrelevant to epärelevantti asiassa If there is no identified relation (or chain of relations) between a subject and an object, it implies that the subject is irrelevant to the object. However, sometimes people may (falsely) think that it is relevant, and this relation is used to explicate the irrelevance.
evaluative link finds important kokee tärkeäksi is found important tärkeäksi kokijana
evaluative link makes relevant tekee relevantiksi is made relevant relevantiksi tekijänä if the subject is valid in the given context, then the object is relevant. This typically goes between arguments, from a variable to value statement or from a value statement to a fact statement. This is a synonym of 'valid defend of type relevance'.
evaluative link makes irrelevant tekee epärelevantiksi is made irrelevant epärelevantiksi tekijänä Opposite of 'makes relevant'. Synonym of 'valid attack of type relevance'.
evaluative link makes redundant tekee turhaksi is made redundant turhaksi tekijänä Everything that is said in the object is already said in the subject. This depreciates a object because it brings no added value. However, it is kept for archival reasons and to demonstrate that the statement was heard.
evaluative link has opinion on mieltä Subject (typically a stakeholder) supports the object (typically a value of fact statement). This is preferred over 'values' and 'finds important' because it is more generic without loss of meaning.
evaluative link values arvostaa valued by arvostajana A stakeholder (subject) gives value or finds an object important. Object may be a topic or statement. Depreciated, use 'has opinion' instead.
evaluative link has truthlikeness on totuudellinen A subjective probability that subject is true. Object is a numeric value between 0 and 1. Typically this has a qualifier according to X where X is the person or archetype who has assigned the probability.
evaluative link has preference mieltymys preference of mieltymyksenä Subject is better than object in a moral sense.
evaluative link has popularity on suosiossa A measure based on likes given by users.
evaluative link has objective omaa tavoitteen objective of tavoitteena
argumentative link agrees samaa mieltä
argumentative link disagrees eri mieltä
argumentative link comments kommentoi commented by kommentoijana
argumentative link defends puolustaa defended by puolustajana
argumentative link attacks hyökkää attacked by hyökkääjänä
argumentative link relevant argument relevantti argumentti Argument is relevant in its context.
argumentative link irrelevant argument epärelevantti argumentti Argument is irrelevant in its context.
argumentative link joke about vitsi aiheesta provokes joke kirvoittaa vitsin This relation is used to describe that the subject should not be taken as information, even though it may be relevant. Jokes are allowed because they may help in creating new ideas and perspectives to an issue.
referential link topic of aiheena has topic aiheesta This is used when the object is a publication and the subject is a (broad) topic rather than a statement. In such situations, it is not meaningful to back up the subject with references. Useful in describing the contents of a publication, or identifying relevant literature for a topic.
referential link discussed in kerrotaan discusses kertoo
referential link reference for viitteenä has reference viite Subject is a reference that backs up statements presented in the object. Used in the same way as references in scientific literature are used.
referential link states väittää stated in väitetään kohteessa Describes the source of a statement; may also refer to a person.
referential link tag for täginä has tag omistaa tägin Subject is a keyword, type, or class for object. Used in classifications.
referential link category for kategoriana has category kuuluu kategoriaan
referential link associates with liittyy Subject is associated with object in some undefined way. This is a weak relation and does not affect the outcomes of inferences, but it may be useful to remind users that an association exists and it should be clarified more precisely. This is its own inverse.
referential link answers question vastaa kysymykseen has answer vastaus Used between a statement (answer) and a topic (question). In knowledge crystals, the relation is embedded in the object structure.
irrelevant argument irrelevant comment epärelevantti kommentti We don't need inverses, because the relation is always tied with an argument (the subject).
irrelevant argument irrelevant attack epärelevantti hyökkäys
irrelevant argument irrelevant defense epärelevantti puolustus
relevant argument relevant comment relevantti kommentti
relevant argument relevant attack relevantti hyökkäys
relevant argument relevant defense relevantti puolustus
property link evaluative property arviointiominaisuus characteristic of a product or work that tells whether it is fit for its purpose. Especially used for assessments and assessment work.
evaluative property property of decision support päätöstuen ominaisuus What makes an assessment or decision support process fit for its purpose?
evaluative property setting of assessment arvioinnin kattavuus What is the context and boundaries of an assessment?
setting of assessment impacts vaikutukset Which impacts are addressed in assessment?
setting of assessment causes syyt Which causes of impacts are recognised in assessment?
setting of assessment problem owner asianomistaja Who has the interest, responsibility and/or means to assess the issue?
setting of assessment target users kohderyhmä Who are the intended users of assessment results?
setting of assessment interaction vuorovaikutus How openly is an assessment produced?
interaction dimension of openness avoimuuden ulottuvuus What is the degree of openness in assessment (and management)?
dimension of openness scope of participation osallistumisen avoimuus Who are allowed to participate in the process?
dimension of openness access to information tiedon avoimuus What information about the issue is made available to participants?
dimension of openness timing of openness osallistumisen ajoitus When are participants invited or allowed to participate?
dimension of openness scope of contribution osallistumisen kattavuus To which aspects of the issue are participants invited or allowed to contribute?
dimension of openness impact of contribution osallistumisen vaikutus How much are participant contributions allowed to have influence on the outcomes? In other words, how much weight is given to participant contributions?
interaction category of interaction vuorovaikutuksen luokka How does assessment interact with the intended use of its results? Possible values: isolated (eristetty), informing (tiedottava), participatory (osallistava), joint (yhteistyöhakuinen), shared (jaettu).
property of decision support quality of content sisällön laatu
quality of content informativeness tarkkuus specificity of information, e.g. tightness of spread for a distribution. How many possible worlds does the answer rule out? How few possible interpretations are there for the answer?
quality of content calibration harhattomuus exactness or correctness of information. In practice often in comparison to some other estimate or a golden standard. How close is the answer to reality or real value?
quality of content coherence sisäinen yhdenmukaisuus correspondence between questions and answers. Also between sets of questions and answers. How completely does the answer address the assessment question? Is everything addressed? Is something unnecessary?
property of decision support applicability sovellettavuus properties in relation to the user needs in a decision process
applicability relevance merkityksellisyys correspondence between output and its intended use. How well does the information provided by the assessment serve the needs of the users? Is the assessment question good?
applicability availability saatavuus accessibility of the output to users in terms of e.g. time, location, extent of information, extent of users. Is the information provided by the assessment available when, where and to whom is needed?
applicability usability käytettävyys potential of the information in the output to trigger understanding in its users about what it describes. Can the users perceive and internalise the information provided by the assessment? Does users' understanding increase about the assessed issue?
applicability acceptability hyväksyttävyys potential of the output being accepted by its users. Fundamentally a matter of its making and delivery, not its information content. Is the assessment result (output), and the way it is obtained and delivered for use, perceived as acceptable by the users?
property of decision support efficiency tehokkuus relation of output and resources used to produce it.
efficiency intra-assessment efficiency sisäinen tehokkuus resource expenditure of producing the assessment output. How much effort is spent in the making of an assessment?
efficiency inter-assessment efficiency ulkoinen tehokkuus resource expenditure of producing assessment outputs in a series of assessments. If another (somewhat similar) assessment was made, how much (less) effort would be needed?

Appendix 3: Workspace tools: OpasnetUtils package and Opasnet Base

Ovariable

Ovariable is the object class that is used in R to operationalise knowledge crystals. In essence, impact assessment models are built using ovariables as the main tool to organise, analyse, and synthesise data and causal relations between items. The purpose of ovariables is to offer a standardised, generalised, and modular solution to modelling. Standardised means that all ovariables have the same overall structure, and this makes it possible to develop generalised functions and processes to manipulate them. Modular structure of a model makes it possible to change pieces within the model without braking the overall structure of functionality. For example, it is possible to take an existing health impact model, replace the ovariable that estimates the exposure of the target population with a new one, and produce results that are otherwise comparable to the previous results but differ based on exposure.

What is the structure of an ovariable such that

  • it complies with the requirements of variable and
  • it is able to implement probabilistic descriptions of multidimensional variables and
  • it is able to implement different scenarios?

An ovariable contains the current best answer in a machine-readable format (including uncertainties when relevant) to the question asked by the respective knowledge crystal. In addition, it contains the information needed to derive the current best answer. The respective knowledge crystal typically has an own page in Opasnet, and the code to produce the ovariable is located on that page under subheading Calculations.

It is useful to clarify terms here. Answer is the overall answer to the question asked (including an evaluated ovariable), and it is the reason for producing the knowledge crystal page in the first place. Answer is typically located near the top of the page to emphasise its importance. An answer may contain text, tables, or graphs on the web page. It typically also contains an R code for evaluating the respective ovariable. Output is the key part (technically a slot) of the answer within an ovariable and contains what the reader wants to know about the answer. All other parts of the ovariable are needed to produce the output or understand its meaning. Finally, Result is the key column of the Output table (technically a data frame) and contains the actual numerical values for the answer.

Slots

The ovariable is a class S4 object defined by OpasnetUtils in R software system. An ovariable has the following separate slots that can be accessed using X@slot (where X is the name of the ovariable):

@name
  • Name of <self> (the ovariable object) is a requirement since R doesn't support self reference.
@output
  • The current best answer to the question asked.
  • A single data frame (a 2D table type in R)
  • Not defined until <self> is evaluated.
  • Possible types of columns:
    • Result is the column that contains the actual values of the answer to the question of the respective knowledge crystal. There is always a result column, but its name may vary; it is of type <self>Result.
    • Indices are columns that define or restrict the Result in some way. For example, the Result can be given separately for males and females, and this is expressed by an index column Sex, which contains locations Male and Female. So, the Result contains (at least) one row for males and one for females. If there are several indices, the number of rows is typically the product of numbers of locations in each index. Consequently, the output may become very large with several indices.
    • Iter is a special kind of index used in Monte Carlo simulations. Iter contains the number of the iteration. In Monte Carlo, the model is typically run 1000 or 10000 times.
    • Unit contains the unit of the Result. It may be the same for all rows, but it may also vary from one row to another. Unit is not an index.
    • Other, non-index columns can exist. Typically, they are information that were used for some purpose during the evolution of the ovariable, but they may be unimportant in the current ovariable if they have been inherited from parent ovariables. Due to these other columns, the output may sometimes be rather wide.
@data
  • A single data frame that defines <self> as such.
  • data slot contains a data about direct measurements or estimates of the output? Typically, when data is used, the output can be directly derived from the information given, with possibly some manipulations such as dropping out unnecessary rows or interpreting given ranges or textual expressions as probability distributions.
  • Probability distributions are interpreted by OpasnetUtils/Interpret.
@marginal
  • A logical vector that indicates full marginal indices (and not parts of joint distributions, result columns, or units or other row-specific descriptions) of output.
@formula
  • A function that defines <self> using objects from dependencies as inputs.
  • Returns either a data frame or an ovariable, which is then used as the output of the ovariable.
  • Formula and dependencies slots are always used together. They estimate the answer indirectly in cases when there is knowledge about how this variable depends on the results of other variables (called parents). The slot dependencies is a table of parent variables and their identifiers, and formula is a function that takes the outputs of those parents, applies the defined code to them, and in this way produces the output for this variable.
@dependencies
  • A data frame that contains names and tokens or identifiers for model runs of variables required for <self> evaluation (list of causal parents). The following columns may be used:
    • Name: name of an ovariable or a constant found in the global environment (.GlobalEnv).
    • Key: the run key (typically a 16-character alphanumeric string) of a model run that is stored to Opasnet server. Key to be used in objects.get() function to fetch the dependent object.
    • Ident: Page identifier and rcode name to be used in objects.latest() function where the newest run contains the dependent object. Syntax: "Op_en6007/answer".
    • Also other columns are allowed (e.g. Description), and they may contain additional information about parents.
  • Dependencies is a way of enabling references in ovariables by using function OpasnetUtils/ComputeDependencies. It which creates variables in .GlobalEnv environment so that they are available to expressions in formula.
  • Dependent ovariables are fetched and evaluated (only once by default) upon <self> evaluation.
@ddata
  • A string containing an Opasnet identifier e.g. "Op_en1000". May also contain a subset specification e.g. "Op_en1000/dataset".
  • This identifier is used to download data from the Opasnet database for the data slot (by default, only if empty) upon <self> evaluation.
  • By default, the data defined by ddata is downloaded when an ovariable is created. However, it is also possible to create and save an ovariable in such a way that the data is downloaded only when the ovariable is evaluated.
@meta
  • A list of descriptive information of the object. Typical information include date created, username of the creator, page identifier for the Opasnet page with the ovariable code, and identifier of the model run where the object was created.
  • Other meta information can be added manually.

OpasnetUtils and operations with ovariables

OpasnetUtils is an R package found in CRAN repository (cran.r-project.org). It contains tools for open assessment and modelling in Opasnet, especially for utilising ovariables as modelled representations of knowledge crystals. Typically, ovariables are defined on Opasnet pages, and their data and evaluated output are stored to Opasnet server. There are also special user interface tools to enable user inputs before an R code is run on an Opasnet page; for further instructions, see http://en.opasnet.org/w/R-tools. However, ovariables can be used independently for building modular assessment models without any connection to Opasnet.

The example code shows some of the most important functionalities. Each operation is followed by an explanatory comment after # character.

install.packages("OpasnetUtils") # Install the package OpasnetUtils. This is done only once per computer.

library(OpasnetUtils) # Open the package. This is done once per R session.

objects.latest("Op_en4004", code_name="conc_mehg") # Fetch ovariables stored by code conc_mehg on Opasnet page Mercury concentrations in fish in Finland (with identifier 4004)

conc_mehg <- EvalOutput(conc_mehg) # Evaluate the output of ovariable conc_mehg (methyl mercury concentrations in fish) that was just fetched.

dat <- opbase.data("Op_en4004", subset="Kerty database") # Download data from Kerty database on the same page and put that to data.frame dat

a <- Ovariable("a", data=data.frame(Fish=c("Herring","Salmon"), Result=c(1,3))) # Define ovariable for scaling salmon results with factor 3.

mehg_scaled <- conc_mehg * a # Multiply methyl mercury concentrations by the scaling factor.

An ovariable is well defined when there is enough data, code or links to evaluate the output. Ovariables often have upstream dependencies whose output affect the output of the ovariable at hand. Therefore, ovariables are usually stored in a well defined but unevaluated format (i.e. without output). This makes it possible to use the same ovariable in different contexts, and the output varies depending on the upstream dependencies. On the other hand, it is possible to store all evaluated ovariables of a whole assessment model. This makes it possible to archive all details of a certain model version for future scrutiny.

Ovariables have an efficient index handling, which makes it possible to do arithmetic operations such as sums and products in a very simple way with ovariables. The basic idea is that if the outputs of two ovariables have two columns by the same name, they are automatically merged (or joined, in the SQL vocabulary) so that rows are merged iff they have the same location values in those two columns. The same principle applies to all pairs of columns by the same name. After the merge, the arithmetic operation is performed, row by row, to the Result columns of each ovariable. This results in an intuitive handling of outputs using a short and straightforward code.

Recursion is another important property of ovariables. When an ovariable is evaluated, a code checks whether it has upstream dependencies. If it does, those ovariables are fetched and evaluated first, and recursively the dependencies of those ovariables are fetched also, until all dependencies have been evaluated. Case-specific adjustments can be done to this recursion by fetching some upstream ovariables before the first ovariable is evaluated; if an upstream ovariable exists already in the global environment, the existing object is used and the respective stored object is not fetched (dependencies are only fetched if they do not already exist; this is to avoid unnecessary computation).

Decisions and other upstream commands

The general idea of ovariables is such that they should not be modified to match a specific model but rather define the knowledge crystal in question as extensively as possible under it's scope. In other words, it should answer its question in a reusable way so that the question and answer would be useful in many different situations. (Of course, this should be kept in mind already when the question is defined.) To match the scope of specific models, ovariables can be modified without changing the ovariable code by supplying commands upstream. A typical decision command is to make a new decision index with two scenarios, "business as usual" and "policy" and use the original ovariable result for business as usual and adjust the result for the policy e.g. by adding or multiplying it by a constant reflecting the impact of the policy on the ovariable. Such adjustments can be done on the assessment level without a need to change the ovariable definition in any way.

Evaluating a latent ovariable triggers first the evaluation of its unevaluated parent ovariables (listed in dependencies) since their results are needed to evaluate the child. This chain of evaluation calls forms a recursion tree in which each upstream variable is evaluated exactly once (cyclical dependencies are not allowed). Decision commands about upstream variables are checked and applied upon their evaluation and then propagated downstream to the first variable being evaluated. For example, decisions in decision analysis can be supplied this way:

  1. pick an endpoint ovariable
  2. make decision variables for any upstream ovariables (this means that you create new scenarios with particular deviations from the actual or business-as-usual answer of that ovariable)
  3. evaluate endpoint ovariable
  4. optimize between options defined in decisions.

Other commands include: collapse of marginal columns by sums, means or sampling to reduce data size; and passing input from model level without redefining a whole ovariable.

Opasnet Base

Opasnet Base is a storage database for all kinds of data needed in open assessments. It may contain parameter values for models, which are typically shown as small tables on knowledge crystal pages, from which they are automatically stored to the database. It may also contain large dataset such as research datasets or population datasets of thousands or even millions of rows, and they are uploaded to the database using an importer interface. Each table has its own structure and may or may not share column names with other tables; however, if a table is directly used as data slot for an ovariable, it must have a Result column.

Technically, Opasnet Base is a noSQL database using MongoDB software. Metadata of the tables is stored in a MySQL database. This structure offers the speed, searchability, and structural flexibility that a large amount of non-standard data requires. The database also offers version control, as old versions of a data table are kept in the database when new data is uploaded.

The database also contains data about model runs that have been performed on Opasnet, if objects were stored during that model run. This makes it possible to fetch objects produced by a particular code on a particular knowledge crystal page. Typically the newest version is fetched, but information about the old versions are kept as well. The objects stored are not located in MongoDB but on server files that can be accessed with a key. It is also possible to save objects in a non-public way so that the key is not stored in the database and is only given to the person who ran the code. Due to disc storage reasons, Opasnet does not guarantee that stored objects will be kept permanently; therefore, it is a good practice to store final assessment runs with all objects to another location for permanent archival.

There are several ways to access database content.

For further instructions, see http://en.opasnet.org/w/Opasnet_Base_UI for user interface and http://en.opasnet.org/w/Table2Base for the wiki interface of small tables.

Appendix 4: Tools to help in shared understanding

There are lots of software and platforms to support decision making. Some of them have been listed here. The focus is on open source software solutions when available. Many examples come from Finland, as we have practical experience about them. The list aims to cover different functionalities and show examples rather than give an exhaustive list of all possibilities; such lists may be found from Wikipedia, e.g. https://en.wikipedia.org/wiki/Comparison_of_project_management_software. All links were accessed 4 Jan, 2018.

Table A4-1. Useful functionalities and software in open policy practice.
Item Functionality or process phase Tool or software
Decision process Information-based decision support There is no single tool covering the whole decision process. Development work is needed. An interesting pilot software is being developed by the city of Helsinki for comprehensively managing and evaluating their ambitious Climate Neutral Helsinki 2035 action plan and its impacts.
Initiative Several websites for launching, editing, and signing citizen initiatives at municipality or national level: Kansalaisaloite (Citizen Initiative), Nuortenideat (Ideas of the Young), Kuntalaisaloite (Municipality Initiatives). Similar tools could be used also for initiatives launched by Members of Parliament or the Government.
Substance Content management Diary systems, file and content management systems. Lots of individual solutions, mostly proprietary. VAHVA project by the Finnish Government will provide knowledge and tools for content management.
Research data and analyses AVAA, IDA, Etsin and other data management tools help in managing research data from an original study to archival. Avoin data (open data in Finland), platform for publishing open data. Findicator: indicators from all sectors of the society. Quilt for open data sharing. Tools for separate analysis tasks are numerous, e.g. QGIS for geographical data. Several research fields have their own research and article databases, such as ArXiv.org (articles about physics, mathematics and other fields). Several biological databases.
Public discussion, argumentation, statements Otakantaa, Facebook, Twitter, blogs, and other social media forums for discussion. Websites for fact checking: Factbar, Fullfact, Need to know project for fact checking. Agoravoting is an open voting system. Lausuntopalvelu collects statements from the public and organisations related to planned legislation and Government programs in Finland. Swarm AI for collective intelligence
News News feeds (open source) CommaFeed, Tiny Tiny RSS. Semantic, automated information searches, e.g. Leiki.
Description and assessment of decision situations and relevant causal connections Opasnet for performing Open assessments and impact assessments. Knowledge crystals as integral parts of models and assessments. Simantics System Dynamics in semantic models. Jupyter notebooks for collaborative model development. Wikidata, Wikipedia as storages of structured data and information.
Laws and regulations Semantic Finlex contains the whole Finnish legislation and e.g. the decisions by the Supreme Court in a semantic structure.
Methods Preparation of documents, co-creation, real-time co-editing Several co-editing tools, e.g. Hackpad, MS Office365, Google Docs, Etherpad, Dropbox Paper, MediaWiki and Git. These tools enable the opening of the planning and writing phase of a decision. E.g. the Climate Neutral Helsinki 2035 action plan was co-created online with Google Docs and Sheets in 2018.
Development and spreading good practices InnoVillage helps to develop practices faster, when everyone's guidance is available online and can be commented.
Organising systems for information and discussions Decentralised social networking protocol Activitypub

Tools: Full Fact automated fact checking Compendium. Vocabularies and semantic tools: Resourse Description Framework (RDF), Finto (Finnish Thesaurus and Ontology Service), AIF-RDF Ontology using Conceptual Graphics User Interface COGUI. These act as a basis for organising, condensing and spreading knowledge.

Information design, visualisations Interactive and static visualisations from complex data. Shiny, Diagrammer, Gapminder, Lucify Plotly Cytoscape
Work Work processes in decision making, research etc: follow-up, documentation Ahjo decision repository and Openahjo interface document and retrieve decisions that have been done in the city of Helsinki. Git enables reporting of both research and decision processes. There are several new platforms for improving science, such as Open Science Framework for facilitating open collaboration in research. Omidyar Network is a philantropic investment firm supporting e.g. governance and citizen engagement. Tsampo is a platform for matching questions, funding, and experts.
Co-creation, experiments, crowdsourcing Kokeilun paikka promotes experiments when applicable information is needed but not available. Sociocracy 3.0 provides learning material and principles for open collaboration in organisations of any size.
Project management There are lots of project management software, mainly targeted for enterprise use but somewhat applicable in decision making or research. Some examples: OpenProject, Project Management Body of Knowledge, Comparison of project management software, Fingertip.
Stakeholders Expert services Network of innovators, ResearchGate Solved and other expert networks.

See also