Open policy ontology

This page is a knowledge crystal of subtype method. The page identifier is Op_en7783
Moderator:Jouni (see all)

Upload data {{#opasnet_base_link:Op_en7783}}

The structure of shared understanding describes the information structures that are needed to document shared understanding of a complex decision situation.

Question

What information structures and information tools are needed to document shared understanding in such a way that

it can be operationalised and managed and used for automatic inferences by a computer,
it can systematically organise information objects used in open assessment, such as variables and statements,
it can represent each participant's views systematically as a part of the whole even if people disagree,
it is intuitive enough to be used by non-experts?

Answer

Shared understanding aims at producing a description of different views, opinions, and facts related to a specific topic such as a decision process. The structure of shared understanding describes the information structures that are needed to document shared understanding of a complex decision situation. The purpose of the structure is to help people identify hidden premises, beliefs, and values and explicate possible discrepancies. This is expected to produce better understanding among participants.

The basic structure of a shared understanding is a network of items and relations between them. This network uses Resource description framework, which is an ontology standard used to describe many Internet contents. Items and relations (aka properties) are collectively called things. Each item is typically of one of the types mentioned below. This information is documented using property instance of (e.g. Goherr assessment is instance of assessment).

Different item types have different levels of standardisation and internal structure. For example, variables are web pages that always have headings question, answer and rationale, and the information is organised under those headings. Some other items describe e.g. statements that are free-text descriptions about how a particular thing is or should be (according to a participant), and yet some others are publications. A common feature is that all items contain information that is relevant for a decision.

In the structure of shared understanding, each item may have lengthy texts, graphs, analyses or even models inside them. However, the focus here is on how the items are related to each other. The actual content is often referred to as one key sentence only (description). Each item also has a unique identifier URI that is used for automatic handling of data. Voting age is an example discussion and shows a structured description in table format.

**Ontological structure of shared understanding**
Items (information objects)	Properties (relations)
Assessment. Assessment has parts Scope, Answer and Rationale. Scope has part Decisions and scenarios. Decision may be used independently. Rationale has part Stakeholder. It may be used independently. Variable has parts Question, Answer and Rationale. The Question of a variable may be called Topic and used independently. Topic may also be objective. Method has parts Question, Answer and Rationale. Discussion. Discussion has parts Statement, Resolution and Argumentation. Statement has subclasses Value statement, Fact statement. Statement may be used independently. Action. Action has parts Who, When, What. Action has subclass Decision. It may be used without applying internal structure. Publication	set theoretical (instance of, subclass of, has context) logical (if - then, and, or, not,... ) causal (affects, increases, decreases, prevents) referential (makes relevant; associates to; has reference, tag, category) evaluative (has truthlikeness, value, popularity, finds important) argumentative (attack, defend, comment) topical where the object is always a decision (substance of, decision process of, task of, method of, stakeholder of, irrelevant to)

Rationale

Item types

Free-structured items

These items contain information in a free structure, i.e. the item type does not impose any particular internal structure for the item. Of course, many items such as publications may have a very detailed structure.

Statement is a factual or value statement about how something is or should be, respectively. A statement may be developed into a discussion by adding and organising related argumentation (according to pragma-dialectics), or into a claim by organising premises and inference rules (according to Perelman).
- Fact statement is any text that claims something about the world.
- Value statement is any text that claims that something ought to be or that it is better than something else.
Publication is any published report, book, web page or similar permanent piece of information that can be unambiguously referenced.
Stakeholder describes an individual or organisation that participates in a decision process by providing information, making decisions, implementing them, or being affected by them.
Topic is a description of an area of interest. It defines boundaries of a content rather than defines the content itself, which is done by statements. When the information structure is improved, a topic often develops into a question of a variable, while a statement develops into an answer of a variable.
- Decision describes a particular event where a decision maker chooses among defined alternatives. This may also be a part of an assessment under heading Decisions and scenarios.
- Objective is a desired outcome of a decision. In shared understanding description, it is a topic (or variable) that has value statements attached to it.
Action is work targeted to produce information for or make a decision, or implement it. It may have parts Who, What and When, but its internal information structure has not been standardised.

Structured items

These items have a standardised information structure. The first three types are knowledge crystals with the structure question-answer-rationale. They are used in open assessments.

Assessment describes a decision situation and typically provides relevant information to decision makers before the decision is made (or sometimes after the decision about its implementation or success). It is mostly about the knowledge work, i.e. tasks for decision support.
Variable describes a real-world topic that is relevant for the decision situation. It is about the substance of the topic.
Method describes how information should be managed or analysed so that it will answer the policy-relevant questions asked. It is about methods.
Discussion, or structured argumentation, describes arguments about a particular statement and a synthesis about an acceptable statement. There are two kinds of statements, factual and value-based. Note that also the answer of a variable typically is a statement, even if it is often produced by the means of a statistical analysis rather than an argumentation.

Properties

Relations are sentences that connect one item (subject) with a property (predicate or verb) to another item (object), possibly with qualifiers about scoping, references etc. In the structure of shared understanding, one key idea is that the relations are straightforward and simple enough so that a computer can make inferences about the items and their relations. Therefore, the number of relations is kept small. Some relations are the same as used in Wikidata. This approach is compatible with resource description framework (RDF), the same system that is used in Wikidata database. These are the key relations classified based on discipline.

Set theory:
- instance of (<> inverse of has instance): subject belongs to a set defined by the object and inherits the properties of the set.
- subclass of (<> has subclass): subject is a subset of object.
- part of (<> has part): is a part of a bigger entity, e.g. Venus is part of Solar System.
- has context: subject given that object is true. ----#: . Can this be expanded? I can't quite understand the description. --Tuukka 09:02, 2 July 2018 (GMt+2) (type: truth; paradigms: science: comment)
Topical (in relation to a particular decision):
- substance of (<> has substance): is or describes substantive issues of a particular decision.
- decision process of (<> has decision process): is or describes parts of decision process of a particular decision.
- task of (<> has task): is or describes a task to promote a particular decision.
- method of (<> has method): is or describes methods used in a particular decision.
- stakeholder of (<> has stakeholder): is or describes a stakeholder of a particular decision process (including decision makers and experts) or someone who is affected by that decision.
- irrelevant to (<> has no use for): is or describes information that does not bring any added value to a particular decision.
Logical:
- if - then: If subject is true, then object is true. Also the negation is possible: if - then not.
- opposite of: subject is opposite of object.
- <> inverse of: a sentence is equal to another sentence where subject and object switch places and has the inverse relation. ----#: . When is this used and why is it used instead of just using the inverse of the relation? --Tuukka 09:02, 2 July 2018 (GMt+2) (type: truth; paradigms: science: comment)
- and, or, equal, exists, for all: logical operators.
Causal:
- affects: (more specifically cause of, immediate cause of, contributing factor of). May have specifying qualifiers such as increases, decreases etc. if the outcome is a quantitative rather than qualitative.
- has causality {function, conditional probability}
Referential:
- makes relevant (<> is relevant given): if the subject is relevant in the given context, then also the object is. This typically goes from a variable to value statement or from a value statement to a fact statement.
- makes irrelevant (<> is irrelevant given): opposite to makes relevant. If the subject is relevant (or true?) in the given context, then the object is irrelevant. ⇤--#: . Too similar to irrelevant to. These two should be merged. --Jouni (talk) 09:36, 23 November 2017 (UTC) (type: truth; paradigms: science: attack)
- makes redundant (<> is redundant given): Everything that is said in the object is already said in the subject. This depreciates a subject because it brings no added value. However, it is kept for archival reasons and to demonstrate that the statement was heard.
- has reference (<> is reference for): object is a reference that backs up statements presented in the subject. Used in the same context as references in scientific literature are used.
- stated in: the source of a statement; may also refer to a person.
- has tag (<> is tag for): object is a keyword, type, or class for subject. Used in classifications.
- has category' (<> is category for)
- associates to (<> associates to, i.e. is its own inverse): subject is associated to object in some way. This is a weak relation and does not affect the outcomes of inferences, but it may be useful to remind users that an association exists and it should be clarified more precisely.
- has topic (<> is discussed in): this is used when the subject is a publication and the object is a (broad) topic rather than a statement. In such situations, it is not meaningful to back up the object with references. Useful in describing the contents of a publication, or identifying relevant literature for a topic.
- answers question (<>is question for): this relation is used between a statement and a topic. In knowledge crystals, the relation is embedded in the object structure.
Evaluative:
- has truthlikeness: A subjective probability that subject is true. Object is a numeric value between 0 and 1. Typically this has a qualifier "according to X" where X is the person or archetype who has assigned the probability.
- has preference: subject is better than object in a moral sense.
- has popularity: measured based on likes given by users.
- values (<> is valued by): a stakeholder gives value or finds an object important. Object may be a topic or statement.
- finds important (<> is important to): synonym to "values" ----#: . Do we need both? --Jouni (talk) 14:53, 9 November 2017 (UTC) (type: truth; paradigms: science: comment)
- has opinion: a relation between a stakeholder and a value statement ----#: . Are there situations where "values" cannot be used? --Jouni (talk) 14:53, 9 November 2017 (UTC) (type: truth; paradigms: science: comment) ⇤--#: . Yes, if it points to a fact statement. --Jouni (talk) 14:53, 9 November 2017 (UTC) (type: truth; paradigms: science: attack)
Argumentative:
- attacks, defends, comments: these argumentative relations are typically used within discussions to reach a statement. Therefore they rarely show up in the structure of shared understanding.
Operational:
- performs (<> is performed by): who does a task?
- about: a task is about a topic
- produces: an output of a task of by a stakeholder
- is used by (<> uses): an input for a task
Participatory:
- participates in (comments, negotiates, provides, uses, asks, answers, decides)
- has interest in / finds important ----#: . Do we need both? Is the former factual and the latter value-based? --Jouni (talk) 10:47, 4 November 2017 (UTC) (type: truth; paradigms: science: comment)
- claims
- is affected by (a decision)
- has owner (<> is owned by)
Other:
- There may also be mathematical and functional relations, but they are typically used in models within variables. Therefore they rarely show up here.
- exists within (the subject exists within the boundaries set by the object; existence does NOT mean that subject is true, only that there is a reason to consider its contents. ----#: . Example needed. --Jouni (talk) 10:47, 4 November 2017 (UTC) (type: truth; paradigms: science: comment)
- There are important relations within decisions. Actions may be exclusive (if you do one, you cannot do another). Some sets of actions may be mutually exhaustive (all possible actions in a given situation are listed). It should be possible to describe these issues with relations as well, but such properties are not well developed yet.

In the case of a complex information object such as a book, it is impossible to classify all its contents using a single topical property or statement. Therefore, several topics, statements, or variables may be developed from its content. In many cases, the better usefulness of the new items justifies the extra work of organising information. When the new information structures are used, the original document is used as a reference to back up the new items.

Topical information areas

Hierarchies(-)
Obs	Class	Lan	Item	Description
1	substance	en	substance, risk factor, risk indicator, data, data work, objective, priority, value judgement, expense	yleinen sanasto
2	substance	fi	substanssi, riskitekijä, riski-indikaattori, data, tietoaineisto, tiedon käyttö, tavoite, prioriteetti, arvoarvostelma, ovariable, tietokide, kustannus	generic vocabulary
3	substance	en	health impact	special words
4	substance	fi	terveysvaikutus	erikoissanastoa
5	stakeholder	en	stakeholder, social and health organisation, ministry, expert organisation
6	stakeholder	fi	toimija, sidosryhmä, sote-organisaatio, ministeriö, asiantuntijalaitos, hallinto-organisaatio
7	health organisation		health organisation, sote-organisaatio
8	data		tietoaineisto, data
9	process, task	en	process, task, action, decision, work, prevention, treatment, support
10	process, task	fi	prosessi, tehtävä, päätös, toimenpide, toiminta, ehkäisy, hoito, tuki, HNH2035-toimenpide, tehtäväkokonaisuus, osiotyyppi, JHS-luokka
11	method	en	method, analysis, reporting, measurement, study, objective, publication
12	method	fi	metodi, menetelmä, analyysi, raportointi, measurement, study, objective, publication
13	objective		tavoite, objective, kustannustavoite, expense objective
14	publication		julkaisu, publication
15	statement	en	fact statement, value statement
16	value judgement		arvostus, arvoarvostelma, ulottuvuus, arviointikriteeri, value judgement, value resolution
17	fact judgement		arvio, estimate, fact resolution
18	causal link	en	reduces, improves, sustains, increases, affects, affected by, prevents, decreases, cause of, immediate cause of, contributing factor of
19	causal link	fi	edistää, estää, vähentää, parantaa, ylläpitää, lisää, vaikuttaa, vaikuttuu, syy, välitön syy, vaikuttava tekijä
20	positive causal link		edistää, parantaa, ylläpitää, lisää, increases, improves, sustains
21	negative causal link		estää, vähentää, reduces, prevents, decreases
22	participatory link	en	has performer, performs, decides, asks, participates, has participant, accepts, develops, proposes, answers, has responsible person, is responsible for, negotiates, recommends, controls, claims, has owner, owned by
23	participatory link	fi	toteuttajana, toteuttaa, tehtävänä, tekee, ylläpitäjänä, valvojana, päättää, kysyy, osallistuu, osallistujana, hyväksyy, kehittää, ehdottaa, vastaa asiasta, vastuullisena, vastaa kysymykseen, neuvottelee, suosittelee, kontrolloi, väittää, omistaa, omistuu
24	operational link	en	has option, is option for, is index for, has index, tells, describes, is described by, contains data, data contained in, is data for, uses, used by, produces, provides, has part, part of, about, has context, subclass of, has subclass, instance of, has instance, opposite of, inverse of, prepares, pays, has rationale
25	operational link	fi	kertoo, kuvaa, kuvautuu, sisältää dataa, dataa sisältyy, käyttää, käyttyy, tuottaa, tarjoaa, omistaa osan, on osana, suoritaa, aiheesta, omistaa kontekstin, omistaa alajoukon, on alajoukko, on alkio, omistaa alkion, vastakohta, toisinpäin, omistaa indeksin, on indeksinä, sisältää vaihtoehdon, on vaihtoehtona, tehtäväkokonaisuus, ulottuvuus, JHS-luokka, osiotyyppi, kustantaa, valmistelee, omistaa perustelun
26	evaluative link	en	finds important, makes relevant, makes irrelevant, has opinion, values, has truthlikeness, has preference, has popularity
27	evaluative link	fi	pitää tärkeänä, tekee relevantiksi, tekee turhaksi, on mieltä, arvostaa, on totuudellinen, on arvostus, on suosio
28	argumentative link	en	disagrees with, attacks, defends, comments
29	argumentative link	fi	samaa mieltä, eri mieltä, hyökkää, puolustaa, kommentoi
30	referential link	en	has topic, dicussed in, has reference, reference for, stated in, has tag, tag for, has category, category for, associates to, answers question
31	risk factor		riskitekijä, risk factor
32	indicator	en	risk indicator, indicator, effectiveness indicator, operational indicator
33	indicator	fi	riski-indikaattori, indikaattori, vaikuttavuusindikaattori, toimintaindikaattori, Hyte-indikaattori, HYTE-indikaattori, Sote-indikaattori, SOTE-indikaattori, Kuva-mittari
34	irrelevant argument	en	irrelevant attack, irrelevant defense, irrelevant comment
35	argument	en	value judgement, true argument, false argument, statement, fact statement, value statement, resolution, fact resolution, value resolution, fact discussion, value discussion
36	statement	en	fact statement, value statement
37	value discussion	en	value discussion, value resolution
38	fact discussion	en	fact discussion, fact resolution

Inverse relations(-)
Obs	Preferred	Inverse
1	kuvautuu	kuvaa
2	described by	describes
3	vastuullisena	vastaa asiasta
4	has responsible person	responsible for
5	osallistujana	osallistuu
6	has participant	participates
7	toteuttajana	toteuttaa
8	has performer	performs
9	sisältää alkion	on alkio
10	has instance	instance of
11	sisältää osan	on osa
12	has part	part of
13	dataa sisältyy	sisältää dataa
14	data contained in	contains data
15	affects	is affected by
16	is index for	has index
17	is option for	has option
18	valmistelee	valmistelijana
19	prepares	prepared by
20	kustantaa	kustantajana
21	pays	paid by
22	omistaa perustelun	perustelee
23	has rationale	is rationale for

This ontology is specifically about decision making, and therefore actions (and decisions to act) are handled explicitly. However, any natural, social, ethical or other phenomena may relate to a decision and therefore the vocabulary has to be very generic. When we descbribe actions under planning, those descriptions are called "Decision processes", while all other descriptions are called "Substance".

The properties described below are used in the following kind of sentences: "Information X describes Y of decision Z" where X is a particular piece of information that describes a phenomenon, Y is the topical information area that is described, and Z is a particular decision.

Substance (information about a substantive topic or phenomenon itself): What issues relate to a decision Z? What causal connections exist between issues? What scientific knowledge exist about the issues? What actions can be chosen? What are the impacts of these actions? What are the objectives and how can they be reached? What values and preferences exist?
Decision process (information about how a decision Z will be made): What will be decided? When will it be decided? Who has the authority to decide? Who are involved? How is the decision prepared? What political realities and restrictions exist?
Tasks (information about organising the information work to support decision making): What tasks are needed to collect and organise the information? When do these tasks need to be done? Who is responsible of what? How is information work organised? Tasks are also important afterwards to distribute merit and evaluate the process: Who did what? How did information evolve? Where did data come from?
Methods (information about methods used in the information work): How to perform information work? What methods are available for a task? How to participate in the work? How to use statistical and other methods and tools?
Stakeholders (information about stakeholders and decision makers in the information work): Who participates? Who should participate? Who has necessary skills for contributing? How to motivate participation? How to measure merit of contributions?
Irrelevant issues: Information that do not fall into any of the previous categories is thus irrelevant for a particular decision Z. If there is no identified relation between an information object and a decision, it implies that the information object is irrelevant. However, because not all relevant relations have been considered and documented, it is often useful to explicate the irrelevance, especially if people may (falsely) think that it is relevant.

How to manage shared understanding

Shared understanding is a structured description of a decision situation. A key idea is that it is much faster to produce than a quantitative assessment, is more usefully organised than a free-format document (not to mention unmoderated discussions), and can distill information from both into a coherent information structure.

What steps does the work process contain?

Take a new piece of information.
Identify a decision situation to which it is relevant (or it is commonly considered relevant even if it is not).
Choose one of the topical relations for the piece.
Link the piece to something else within the decision situation with a relation. If it links to nothing, it is irrelevant.
Add descriptive tags, references etc. as appropriate.
Link the relations to the decision situation the relation part of (this should happen automatically).

Questions for further developing shared understanding:

What are the main questions within each topical area?
What are necessary structures and relations?
What software tools can be used?

Potential tools for managing the information

Google Drive graphical tool [1]
Protégé software for ontologies
Wikidata for RDF database using Wikibase software [2]
Git for version control [3]
Shiny R package for user interface [4] [5]
- Shiny packages for user table output dataTableOutput [6] ShinySky
R and D3 [7][8]

Example of using the structure

Columns that may have several values per risk are marked with *

Riskilomake (a variable with the question: What is a risk that is relevant for the success of THL's mission? There are several variables with an identical question, but each variable describes exactly one risk as an answer.)

Each column is described within the variable answer unless otherwise noted a property that is used to link the column contents to the variable.

id# Tarkastelukohde (yksikkö) * (has tag) Riskialue (aihepiiri karkea) * (has tag) Sisäalue / sisältösivu (aihepiiri tarkka) * (has tag) Riski Tarkennus Todennäköisyys Vakavuus Riskiluku (todennäköisyys*vakavuus) Hallintatoimet (kpl) Muistiinpanot Omistaja Tila Hallintatoimien valmius-% Omat liitteet * Luotu (automatic from version control) Päivitetty (automatic from version control)

Hallintatoimilomake (many-to-many relationship with risks) id# Hallintatoimi Määräpäivä Vastuuhenkilö Tila Omat liitteet * Luotu (automatic from version control) Päivitetty (automatic from version control)

Example about climate neutrality in Helsinki

Example about Arvoprofiili

Example about fisheries management in the Baltic Sea

Susan Star: This is not a boundary object. SSHV 2010

Technical prerequisites

Structure of shared understanding can be implemented using an RDF database, e.g. Wikibase. These are some links to resources and guidance about that.

Related concepts

Deliberative democracy

James Fishkin, a key proponent of deliberative democracy, describes two approaches to public opinion, raw vs. refined: what people actually think vs. what their opinion would be after it has been tested by the consideration of competing arguments and information coscientiously offered by others who hold contrasting views. Political process can be seen as whether a filter or a mirror. The filter creates counterfactual but deliberative representations of public opinion. The mirror offers a picture of public opinion just as it is, even if it is debilitated or inattentive. The conflicting images suggest a hard choice between the reflective opinion of the filter and the reflected opinion of the mirror.^[1]

It is only through the deliberations of a small face-to-face representative body that one can arrive at the "cool and deliberate sense of the community" (James Madison, Federalist No 63). ... A key desideratum in the Founders' project of constitutional design was the creation of conditions where the formulation and expression of deliberative public opinion would be possible.^[1] A smallish group of randomly selected people are likely to act as a filter, while e.g. a referendum would act as a mirror. During the early days of the United States, James Madison actively designed governance structures that would enable the formation of refined public opinion in the national US policy. The electorate was such a construct, designed to enable informed argumentation about president candidates before the final vote. However, this role has completely disappeared, as nowadays the outcome of the electoral vote is known as soon as the composition of the electorate is known.

Shared understanding follows these lines of reasoning and aims to produce a deliberative outcome of informed argumentation. However, the major difference is that the deliberative process does not aim to produce a decision by the participants, but a comprehensive description of shared understanding with all relevant points and disagreements. This written description enables other people to learn and form their own opinions of the matter, and thus help in other similar decision situations. Although producing such a description may be time-consuming and labourious, re-usability of the information makes it worth the effort.

Cognitive democracy

Henry Farrell (George Washington University), Cosma Shalizi (Carnegie Mellon University and The Santa Fe Institute). 2012?. An Outline of Cognitive Democracy [9],

Farrell and Shalizi analyze three main approaches to socially achieve results: hierarchies in different forms (with problems that those who are in power are not receiving information from the others); markets (with problems that they converge to individual benefit, which is sometimes in conflict with social benfit), and democracy (with problem how to actually implement the main principle of equal power among individuals). They suggests approaches to improve democracy.

Pol.is

Pol.is is a website for organised democratic discussion. It helps large organizations and communities understand themselves by visualizing what people think.

An example discussion about sote indicators [10] (in progress)
A case study of temperature check [11]
A case study from Taiwan [12]: vTaiwan: Public Participation Methods on the Cyberpunk Frontier of Democracy. In the midst of the signal failure known as the US electoral season, here’s something to be inspired about: a true story about rational deliberation on a national scale.

Professionalism

Jonathan Rauch and Benjamin Wittes. (May 2017) More professionalism, less populism: How voting makes us stupid, and what to do about it. Center for Effective Public Management at Brookings. [13]

Rauhankone

Artificial intelligence may solve some of the structural problems related to development of shared understanding. How this would actually happen is largely unclear. However, professor Timo Honkela is working toward this aim. For more details, see op_fi:Rauhankone.

Inforglobe

A similar but simpler approach is by Mikaeli Langinvainio and Juha Törmänen, who used to work for Crisis Management Initiative. They use statistics to understand views and opinions of different stakeholder groups. (HS 25.6.2017 Voiko rauhanneuvotteluja edistää matematiikalla?) Their company inforglobe produces consulting services based on these ideas. [14]

Their web tool contains these information structures and functionalities (for more details, see Inforglobe link above):

Likelihood of affecting the project vs impact for the project vs knowledge level on the risk or threats vs opportunities

Attributes

Categories (e.g. project planning, logistics and safety, or joker risks)
Participants (e.g. project team, planning organisation, partner organisation, or customer representative)

Issues e.g.

Contractor network (joker)
Cost stucture (planning)
Logistics
Machinery placement (logistics and safety)
Staff competence (planning)

Values e.g.:

Large or small 1-5
Likely 1-5
Knowledge good 1-5

Additional properties

Each value can be enhanced with a suggestion how to mitigate the impact or decrease the likelihood.
Individual answers can be shown with participant attributes and suggestions.
Each issue has a more detailed description.
Based on individual answers, you can make shared conclusions about each issue and how to manage the risks.
Assessment can be done several times.
Participants can be categorised based on position or sector (and maybe other attributes as well)

System dynamic maps

Issues as nodes
Complex system maps: Causal edges between them with strength
Significance of edges (links) is measured in some way and used to select nodes and edges for display
Co-operation: Links can also describe how well items (in this case organisations) communicate with each other.

References

↑ ^1.0 ^1.1 James Fishkin. (2011) When the people speak. Democratic deliberation and public consultancy. Publisher: Oxford University Press. ISBN 978-0199604432

[fishkin2011-1] 1.0 ^1.1 James Fishkin. (2011) When the people speak. Democratic deliberation and public consultancy. Publisher: Oxford University Press. ISBN 978-0199604432

[1]