View source for Openness in participation, assessment, and policy making upon issues of environment and environmental health: a review of literature and recent project results

{{nugget
|authors= [[User:Mikko Pohjola|Mikko V. Pohjola]], [[User:Jouni|Jouni T. Tuomisto]] 
|reference=  
|urn=  
|ethics=  
|journal=
|moderator= Essi Vuorinen
|stub=}}

This page (including the files available for download at the bottom of this page) contains a draft version of a manuscript, whose '''final version is published and is available in the [http://www.ehjournal.net/content/10/1/58 Environmental Health 2011]'''. If referring to this text in scientific or other official papers, please refer to the published final version as: '''Mikko V. Pohjola and Jouni T. Tuomisto: Openness in participation, assessment, and policy making upon issues of environment and environmental health: a review of literature and recent project results. Environmental Health 2011, 10:58 http://www.ehjournal.net/content/10/1/58'''.

The revision history and earlier version can be found from the [http://heande.opasnet.org/heande/index.php?title=Open_participation_in_environmental_health_assessment&oldid=29817 corresponding Heande page] (password protected).

==Title==

Openness in participation, assessment, and policy
making upon issues of environment and
environmental health: a review of literature and
recent project results

===Authors and contact information===

:Mikko V. Pohjola, corresponding author
:(National Institute for Health and Welfare, Kuopio Finland, firstname.lastname@thl.fi)

:Jouni T. Tuomisto 
:(National Institute for Health and Welfare, Kuopio, Finland, firstname.lastname@thl.fi)

==Abstract==

Issues of environment and environmental health involve multiple interests regarding e.g. political, societal,
economical, and public concerns represented by different kinds of organizations and individuals. Not
surprisingly, stakeholder and public participation has become a major issue in environmental and environmental
health policy and assessment. The need for participation has been discussed and reasoned by many, including
environmental legislators around the world. In principle, participation is generally considered as desirable and
the focus of most scholars and practitioners is on carrying out participation, and making participation more
effective. In practice also doubts regarding the effectiveness and importance of participation exist among policy
makers, assessors, and public, leading even to undermining participatory practices in policy making and
assessment.

There are many possible purposes for participation, and different possible models of interaction between
assessment and policy. A solid conceptual understanding of the interrelations between participation, assessment,
and policy making is necessary in order to design and implement effective participatory practices. In this paper we
ask, do current common conceptions of assessment, policy making and participation provide a sufficient
framework for achieving effective participation? This question is addresses by reviewing the range of approaches to
participation in assessment and policy making upon issues of environment and environmental health and some
related insights from recent research projects, INTARESE and BENERIS.

Openness, considered e.g. in terms of a) scope of participation, b) access to information, c) scope of
contribution, d) timing of openness, and e) impact of contribution, provides a new perspective to the
relationships between participation, assessment and policy making. Participation, assessment, and policy
making form an inherently intertwined complex with interrelated objectives and outcomes. Based on
experiences from implementing openness, we suggest complete openness as the new default, deviation from
which should be explicitly argued, in assessment and policy making upon issues of environment and
environmental health. Openness does not undermine the existing participatory models and techniques, but
provides conceptual means for their more effective application, and opens up avenues for developing new
kinds of effective participatory practices that aim for societal development through collaborative creation of
knowledge.

===Abbreviations===

THL: National Institute for Health and Welfare (Finland); INTARESE: Integrated
Assessment of Health Risks of Environmental Stressors in Europe (EU-funded
research project); BENERIS: Benefit-Risk Assessment of Food: an Iterative
Value-of-Information Approach (EU-funded research project); EU: European
Union; EC: European Community; EIA: Environmental impact assessment;
USA: United States of America; IEHIA: Integrated environmental health
impact assessment; EEC: European Economic Community; BEPRARIBEAN: Best
Practices for Risk-Benefit Analysis of Foods; REACH: Registration, Evaluation,
Authorisation and Restriction of Chemical Substances (EU); NRC: National
Research Council (USA); NGO: Non-governmental organization; YVA:
Environmental impact assessment (acronym in Finnish); UNEP: United Nations Environment Programme; UNECE: United Nations Economic
Commission for Europe; ECLAC: Economic Commission for Latin America &
the Caribbean; OECD/NEA: Organisation for Economic Co-operation and
Development/Nuclear Energy Agency; IIED: International Institute for
Environment and Development; MNP: Netherlands Environmental
Assessment Agency (current acronym for the agency is PBL); NIBR:
Norwegian Institute for Urban and Regional Research; ECHA: European
Chemical Agency; ITRC: The Interstate Technology & Regulatory Council.

==Introduction==

Stakeholder and public participation is undoubtedly one
of the most central topics in contemporary discourse
regarding environmental and environmental health policy
and assessment. Environmental issues typically
involve multiple interests regarding e.g. political, societal,
economical, and public concerns and particularly in
cases where they are known or perceived to relate either
directly or indirectly to human health and well-being,
the concerns often also become very personal. In such a
setting of physical, chemical, biological, and societal
complexity, it is widely accepted as important to include
plural perspectives, particularly from the “affected parties”,
in the processes of policy making as well as the
processes of producing information to guide and support
policy making. As the idea of participation mainly
builds on the theories and practices of democracy [1,2],
this is particularly the case in the so called Western
democracies, but increasingly also in countries not generally
considered as democratic by their constitution,
such as the People’s Republic of China [3,4].

In addition to being founded on the principles of
democracy, public participation is addressed in several
intergovernmental agreements, e.g. the Principle 10 of
the Rio Declaration [5], and the Aarhus Convention on
Access to Information, Public Participation in Decision-
Making and Access to Justice in Environmental Matters
[6]. Also several laws on different levels of governance
around the world, e.g. the EU Strategic Environmental
Assessment Directive (2001/42/EC), the EU Public Participation
Directive (2003/35/EC), The Law of the People’s
Republic of China on Environmental Impact Assessment
[4], and the Finnish Environmental Impact Assessment
(EIA) Act (468/94) and corresponding EIA Decree (713/
2006), explicitly consider public participation and
describe legal frameworks for its application. The legal
requirements provide, however, only one perspective to
participation. Importance of participation is also argued
for example based on ethical, political, pragmatic, and
epistemological [7] as well as substantive, normative, and
instrumental reasons [1], and participation is seen to
have the potential to deliver e.g. substantive, procedural,
and contextual effects [8,9]. Participation in assessment
and policy making upon issues of environment and environmental
health has become commonplace.

This paper explores the following question: do current
common conceptions of assessment, policy making and
participation provide a sufficient framework for achieving
effective participation? By effectiveness we mean
influence on the outcomes, i.e. changes in values, attitudes,
and behavior in the society (cf. [10]), of the processes
that the participation relates to, e.g. participatory
assessments or policy making.

Policy making is here understood as decision making
upon issues of societal importance and assessments are
considered as systematic science-based endeavors of
producing information to support policy making. Public
participation and stakeholder involvement are here seen
as instances of the same issue which is mostly referred
to as participation, meaning contributions from the parties,
organizations or individuals that do not have formal
roles as decision makers or experts in the assessment or
policy processes in question.

These broad definitions allow inclusion of various
types of participation, assessment, and policy making,
practiced in and designed for several societal, institutional
and geographical contexts by many different
actors. For example, risk assessment, environmental
impact assessment, and health impact assessment,
whether practiced by consultants, federal agencies, or
academic researchers, in Europe, USA, or China, are
considered as just different manifestations of the fundamentally
same process of science-based policy support.
They are, however, clearly distinguishable from curiosity-
driven research, ad hoc assessments, or assessments
made to justify predetermined decisions. Here we focus
on issues relevant to environment and environmental
health, but the implications can be extended also to
many other substantive contexts.

Answers to the question are sought for by discussing
recent literature relevant to the question. That knowledge
is complemented with some insights from recent
research projects, INTARESE (Integrated Assessment of
Health Risks of Environmental Stressors in Europe) [11]
and BENERIS (Benefit-Risk Assessment of Food: an
Iterative Value-of-Information Approach) [12].

INTARESE was an EU-funded research project running
from 2005 to 2011, developing methodology and
tools for integrated environmental health impact assessment
(IEHIA), and testing them in case studies. BENERIS
was also an EU-funded research project running
from 2006 to 2009, developing a framework and tools
for complicated benefit-risk situations, and applying
them for analyzing benefits and risk of certain foods.

The review starts from purposes of participation and
ends in consideration of openness. Overall it presents a
new perspective to the relationships between participation,
assessment, and policy making.

==Review==

===Purpose of participation===

The discourse on participation, involving both scholars
and practitioners has primarily focused on implementation
of participation while the multiple objectives and
purposes of participation, particularly in relation to the
objectives and purposes of the processes they relate to, have received much less attention [13]. This discourse
has resulted for example in various guidance documents
for stakeholder involvement [8,14,15], detailed presentation
and discussion of various models for public participation,
[16-21], and analysis of the applicability of
participation techniques [22-24]. Although they are all
important contributions to developing understanding
about participation and its implementation, it is not
always easy to identify how they link to the “outcome
effectiveness of participatory processes in their societal
context”, as Newig [25] put it in developing his analytical
framework for evaluating the impact of participation
to improved environmental quality. Many means for
public participation exist, but the ends they serve may
not always be explicitly identified (for more on the theory
of means-ends relationships, see e.g. [26]).

Despite the theoretical stance that participation is generally
viewed as highly desirable and its benefits are often
assumed to be obvious and substantial [13], the practices
of policy making and assessment do not always represent
this view. For instance, in a Finnish environmental permit
case on a waste treatment activity the decision-maker,
the permit applicant, as well as the stakeholders all questioned
the meaningfulness of participation in the process,
although in principle participation was seen as important
by all [27]. The inconsistent utilization of public’s contributions
has also been seen as a general weakness in the
Finnish environmental impact assessment system due to
being strongly dependent on the developer’s attitudes
towards participation as well as the weak links between
the assessment and related decision making processes
[28]. This can be assumed representative of many other
environmental impact assessment systems conducted
under the EU Environmental Impact Assessment Directive
(85/337/EEC) as well. Also in Canada the record of
project-based environmental assessment in delivering on
the promise of meaningful public participation has been
identified as less than promising [29].

A major source of the problem with participation is
that it has been more focused on process and access,
rather than on outcomes [29]. It appears that the issue
of effective participation needs to be considered in
terms of the different possible purposes of both participation
and assessment as well as their roles in the
related policy making processes.
O’Faircheallaigh [13] has presented a nice characterization
of ten specific purposes and activities, categorized
under three broad purposes, for public participation in
environmental impact assessment. The characterization
is made in such a generic way, i.e. not bound to any
specifics of contemporary environmental impact assessment
practices, that we here assume it generalizable to
all policy making and assessment regarding environmental
and environmental health issues. According to

O’Faircheallaigh [13] the three broad purposes for participation
are:


* Obtain public input into decisions taken elsewhere
* Share decision making with public
* Alter distribution of power and structures of decision making


These categories roughly correspond to 1) participation
influencing assessments and (potentially) their
outputs, 2) participation influencing policy making
and (potentially) policy decisions, and 3) participation
as a means for influencing policy making from outside
the existing institutional policy making structures.
Within the broad purposes there can be several more
specific sub-purposes, many of which are identified
and discussed by O’Faircheallaigh [13], e.g. according
to the kinds of expected, desired or allowed participant
contributions. It is important to note that the
purposes for participation are not exclusive, but can,
and in fact often do, co-exist and interact. Advancing
of different purposes of participation is strongly
dependent on the attitudes towards participation
among those who control the policy making and
assessment processes, but also the types of interaction
between assessment and policy making. Particularly
this becomes apparent when attempting to advance
several specific purposes of participation across categories,
for example simultaneously filling information
gaps with local knowledge, inviting public to decision
making, and especially empowering marginalized
groups (cf. [13]).

The relationships between participation, assessment,
policy making, and their outcomes are outlined in Figure
1 and discussed in following sections.

===Role of participation in assessment===

There are multiple kinds of assessment that serve different
purposes and address different kinds of questions, and thereby provide different contexts for participation.
For example, Pope et al. [30] have differentiated between
a) ex-post, project-based assessments (a typical kind for
environmental impact assessment), b) ex-ante, objectives-
led assessment (a typical kind for strategic environmental
assessment), and c) (a more theoretical)
assessment for sustainability. Briggs [31], on the other
hand has differentiated between i) diagnostic assessment
(does a problem exist, is policy action needed?), ii) prognostic
assessment (implications of potential policy
options, which option to choose?), and iii) summative
assessments (effectiveness of existing policies). Assessment
approaches may also be characterized according to
their contexts of development and application as more
regulatory or academic in their nature [32].

Many other classifications exist and new ones could
be made, but what is important from the point of view
of participation in assessments is the possible influence
that is allowed for participation in different assessment
settings. For example, does the assessment structure
allow for rethinking a project at the time the public is
engaged in a project-based environmental assessment
(cf. [29])?, can the stakeholders influence the choice of
policy options to be considered in a prognostic integrated
environmental health impact assessment (cf.
[31])?, or does a down-stream user of a chemical product
have any other role besides providing assessors
with information on specific chemical use contexts in a
REACH (Registration, Evaluation, Authorisation and
Restriction of Chemical Substances (EU)) chemical
safety assessment [33]? The framing of an assessment
approach can be a significant constraining factor for
potential effectivness of participation.

Quite often stakeholder involvement and public participation
are seen as specific steps or stages in the assessment
process (e.g. [31,34,35], Finnish EIA Act (468/94)),
which may limit the possible influence of participation
to only certain questions topical at that particular stage.
Also according to the study on the state-of-the-art in
benefit-risk analysis conducted in the BEPRARIBEAN
(Best Practices for Risk-Benefit Analysis of Foods) project
[36], the commonly applied, contemporarily well
established approaches to environmental health assessment
treat stakeholder involvement and public participation
rather as an add-on, often brought about by legal
requirements, than as an essential aspect of assessment
or decision making processes [32].

[[Image:OIP-Fig1..JPG|thumb|center|400px|'''Figure 1: Relationships between participation, assessment, policy making, and their outcomes.''' Arrows depict alternative routes for potential influence from participation to outcomes.]]

===Role of participation in policy making===

On the other hand, in many aspects the level of influence
that participation in assessments can have is not
directly in control of the assessors. For example, in the
aforementioned Finnish environmental impact
assessment system, where participation is legally
enforced as a part of the assessment process, the decision
making structures outside the assessment may
induce that certain aspects of assessment results, e.g. in
particular public concerns regarding social impacts, cannot
be given weight in the decision making [28]. Also
the Finnish land use planning system, in which there are
legal requirements for impact assessment including public
participation, treats planning (zoning) and development
as separate processes, which means that the
details of planned development, issues of great public
interest, are outside the scope of assessment and stakeholder
involvement [37]. Both of these examples
describe national implementations of EU directives, and
are thereby somewhat representative of the whole target
area of the corresponding EU legislation.

The influence of assessment, participation, as well as all
other potential inputs to policy making is much determined
by the setting in which policy decisions are being
made. As an example from another kind of societal context,
the Chinese authorities may welcome public participation
if it improves the quality of information available
to government decision makers, but may not at all be
willing to give public the power to contribute to and
influence decision making by participating in the formulation
of a proposal, the whole assessment process, the
implementation, and the evaluation of a proposal [4,13].

===Indirect participatory influence===

If no satisfactory roles are provided for public, or even
expert, input directly in decision making or indirectly
through assessment, alternative options for influencing
policy need to be looked for outside the institutionalized
decision making structures, as pointed out also by
O’Faircheallaigh [13]. In fact, quite many, particularly
the more academic, assessment approaches, although
explicitly aiming to support policy making, do not explicitly
describe their linkages to any particular specific
policy uses [32]. This may be interpreted indicative of
their, more or less implicit, intentions to activate also
other channels than only direct influence to policy making.
An alternative way to advance the societal purposes
of assessments is e.g. to promote social learning among
public officials, market players, and citizens. Also in the
recent evaluation of the existing EIA legislation in Finland
the indirect influence of the information and
knowledge obtained in the participatory assessment,
which does not directly serve the formal sectoral permit
decision processes related to the assessed project, was
interpreted as an important aspect of the Finnish EIA
system by contributing to the general awareness among
the society upon the environmental and health impacts
of on-going developments [38].

===Assessment-policy interaction===

As has been pointed out above, the interaction between
assessment and policy making can be crucial for effective
participation. Another question then is what influences
the assessment results, potentially influenced by
participation, have in the related decision making processes.
Although often quite credulously assumed by
assessors and assessment scholars that assessments have
significant impacts to the decision making processes
they aim to serve (cf. [13]), few approaches to assessment
actually even explicitly consider assessment performance
in terms of the outcomes of using the assessment
results in their intended contexts of use [32,39]. Concerns
have also been expressed that the emphasis in
environmental impact assessment has been more on
process and procedure, rather than on purpose and
effects [40,41].

Assessment-policy, as well as related science-policy
and research-practice, relationships have recently been
subjects of intense discussion and several characterizations
of the interfaces or boundaries in between them
have been presented from different viewpoints. For
example Sterk et al. [42] have characterized five boundary
arrangements of varying levels of engagement
between science and policy, van Kerkhoff and Lebel [43]
have presented six categories of relationships between
research-based knowledge and action, a continuum of
increasing engagement and power sharing, and Cashmore
[40] has described a spectrum of five models
representing varying conceptions of the role of science,
and participation, in environmental impact assessment.
In addition, the relationships have been considered in
multiple other discourses, e.g. on trans- and interdisciplinary
research [44-47], regulatory science [48], Integration
and implementation sciences [49], post-normal
science [50], integrated research [51], informing science
[52], knowledge brokerage [53,54], science integrators
[55], boundary organizations, objects and systems
[56-58], science-policy interfaces [59], participatory integrated
assessment [60,61], environmental health assessment
[32], making use of science in policy [62-64],
adaptive governance [65-67], policy integration [68], policy
practice [69], and policy analysis [70].

Although the viewpoints, bases and contexts in the
above mentioned discourses vary, in aggregate they
seem to be pointing to the direction of increased openness.
According to our interpretations in the context of
this paper, of the main lessons from these discourses
regarding assessment-policy interaction and participation
are as follows:


* The traditional model of disengaged scientific assessment and policy making is increasingly considered both by policy makers and researchers as inadequate to address existing policy needs sufficiently
* There is a need for more pragmatic needs-oriented question setting in assessments
* Deeper engagement between assessment and policy making is essential for policy effectiveness
* Stakeholder and public participation is essential for relevance both in assessment and policy making
* Values are an important aspect of the needed knowledge input for both assessment and policy making.


This broad gradual movement could be characterized
as a shifting of both assessment and participation from
the lower degrees of involvement, e.g. informing or
information collection, towards the higher degrees of
involvement, e.g. co-deciding, delegated power, joint
planning, or partnering, in relation to policy making (cf.
[8,15,20,40]). The shift can also be identified e.g. by
observing the development in the perspectives to the
relationship between risk assessment and risk management
adopted in the publications of the NRC (National
Research Council (USA)): from strict disengagement in
the so called Red Book [71] to binding through deliberative
characterization in the so called Orange Book [72],
and on to an intertwined process of risk-based decision
making in the recent so called Silver Book [73]. Also the
role of stakeholder involvement has grown alongside the
development of assessment-policy interaction.

Participation, assessment, and policy making are
becoming to be perceived as an intertwined complex
that needs to be considered as a whole, not as separate
independent entities. The question of effective participation
is thus meaningful only in the broader context that
also concerns the purposes and effects of policy making
and the processes of producing the knowledge that it is
based on. However, as has been pointed out above, the
common current practices of participation, assessment,
and policy making are not necessarily always in line
with the latest discourses in the literature.

===Dimensions of openness===

One obstacle for effectively addressing the issue of effective
participation may be the concept of participation
itself. As long as the discourse focuses on participation,
one is easily misled to considering it as an independent
entity with purposes, goals and values in itself, without
explicitly relating it to the broader context of the processes
whose purposes it is intended to serve. The conceptual
framework we call the dimensions of openness
attempts to overcome this obstacle by considering the
issue of effective participation in terms of openness in the processes of assessment and decision making. The
framework was developed as a part of the assessment
methodology development in the INTARESE project,
and it is intended as guidance for designing and managing
participatory assessment and decision making practices.
In the project, the development work was
originally motivated by a notion of a simultaneous need
to improve effectiveness of assessments in environmental
health policy making as well as to improve effectiveness
and meaningfulness of stakeholder involvement in
environmental health assessments.

As the name implies, the framework consists of five
essential dimensions of openness in assessment and
decision making, or more generally, creation and use of
collective knowledge. The dimensions of openness does
not attempt to provide an exhaustive and mutually
exclusive list of all aspects of openness, but to explicate
and emphasize those that are seen as the most essential
ones in the context of environment and health assessment
and policy. The five dimensions of openness are:


* '''Scope of participation,''' referring to who are allowed to participate in the process.
* '''Access to information,''' referring to what information regarding the issue at hand is made available to participants.
* '''Timing of openness,''' referring to when participants are invited or allowed to participate.
* '''Scope of contribution,''' referring to which aspects of the issue at hand participants are invited or allowed to contribute to.
* '''Impact of contribution,''' referring to what extent are participant contributions allowed to have influence on the outcomes, i.e. how much weight is given to participant contributions.


The dimensions of openness compile the main issues
of participation in one solid framework. In the framework,
the more commonly addressed questions of access
(to process and to information) and timing of participation
are complemented with less commonly addressed
questions of extent and influence of participation on the
outcomes of the process. The five dimensions can be
considered as the determinants of the possibilities and
limitations provided by the context for the effectiveness
of participation. As such, the framework explicates the
aspects of openness that need to be taken account of in
order to match the processes and procedures of collective
knowledge creation and use, e.g. environmental
health assessment and related policy making, with their
aims and purposes. Thereby it also provides a means for
identifying the relationships between participation,
assessment, and decision making.

The framework bears resemblance e.g. to the criteria
for evaluating implementation of the Aarhus Convention
principles by Hartley and Wood [23], the categories to
distinguish a discrete set of public and stakeholder
engagement options by Burgess and Clark [74], and particularly
the seven categories of principles of public participation
by Webler and Tuler [75]. However, whereas
they were constructed for the use of evaluating or
describing existing participatory practices or designs, the
dimensions of openness framework is explicitly and particularly
intended to be used as a checklist type guidance
to support design and management of
participatory assessment and decision making processes.

The perspective adopted in the framework can be
characterized as contentual because it primarily focuses
on the issue in consideration and describing the prerequisites
to influencing it, instead of being confined to
only considering techniques and manoeuvres to execute
participation events. Thereby it helps in participatory
assessment and decision making processes to achieve
their objectives, and on the other hand in providing possibilities
for meaningful and effective participation. The
framework does not, however, tell how participation
should be arranged, but rests on the existing and continually
developing knowledge base on participatory models
and techniques. Although a contentual perspective to
participation, dimensions of openness does not contradict
with the procedural perspectives to participation,
but rather provides a backdrop for their effective
application.

The contentual perspective makes the framework
applicable in design and management of both assessment
and policy making processes. Although assessment
and decision making may appear as very different kinds
of processes, the choice of point of view is actually only
a question of adjusting the scope of application of the
framework; whether decision makers are included in an
assessment or not? After all, assessment and related
decision making should ideally only be alternative perspectives
to the same issue, the former emphasizing the
development of knowledge, the latter emphasizing the
use of knowledge. Within the contentual perspective,
everyone, including also e.g. authorities, project managers,
and experts, not only public, stakeholders, NGO’s
(non-governmental organizations) etc., are considered as
participants to development and implementation of
knowledge. They are all considered as, at least potentially,
relevant contributors to either creating knowledge
or deciding about an issue of interest. The different
kinds of participants naturally take different roles
according to their interests, capabilities, professions, as
well as formal and legal positions in relation to the
issue.

The degree of openness can be managed in terms of
the dimensions of openness according to specific purposes
and goals. The situational, contextual, and practical
issues, for example legal requirements, public
perceptions, available resources, time constraints, complexity
of the case, confidentiality etc. also need to be
taken account of in deciding upon suitable degree of
openness. The degree of openness can be adjusted separately
for different groups of participants, or even on an
individual basis, and varying from a case to another, as
needed. The overall openness of the process can be considered
as a function of all five dimensions across all
roles, although it should be noted that the dimensions
are not independent, but rather interrelated.

For example, the first dimension, scope of contribution,
determines the participant groups among which
questions regarding e.g. access to information or scope
of contribution are only even relevant. In addition, while
all dimensions contribute to the overall openness, it is
the fifth dimension, the impact of contribution, which
ultimately determines the effect on the outcome.
Accordingly, it is recommended that aspects of openness
in assessment and decision making processes are considered
step-by-step, following the order as presented
above.

The greatest power of the framework is that it puts
the issue at hand in focus and does not build on any
preconceptions about possible or acceptable inputs to
its development. It allows to first ask what are the
inputs needed to develop the issue to achieve its purpose,
and then consider the arrangement for its realization,
without being preconfined to existing conventions
and institutions of participation, assessment and decision
making, which, as argued above, are in many cases
known to be inadequate. The framework i) provides a
context for evaluation and constructive criticism of
existing conventions and institutions, ii) facilitates innovative
application of existing means for participatory
processes within and alongside the existing conventions
and institutions, and iii) promotes development of new
means, conventions and institutions for participatory
practice. Thinking in terms of openness provides a new
perspective to participation in assessment and policy
making.


===Implementation of openness===


[[Image:OIP-Fig2..JPG|thumb|400px|'''Figure 2: Open assessment as a collaborative social knowledge process.''' The strawmen depict the members of a society. The paper sheet depicts an assessment in Opasnet (Q = question, R = rationale, A = answer). Yellow arrows depict observation (here of an undesired event, a toxic liquid spill). Green arrows depict information flow (from members of society to an assessment in Opasnet, from the assessment to members of society, or directly between members of society). Blue arrow depicts knowledge-based action (correctly taking care of the spill).]]

The first version of the dimensions of openness framework
was developed already in the early phases of the
INTARESE project. At the same time also an alternative,
procedural, assessment approach was developed within
the project, eventually leading to the formulation of the
IEHIA method [31]. Although in retrospect it can be
seen that the dimensions of openness framework and
the IEHIA method are complementary rather than
fundamentally contradictory, some practical difficulties
in merging these views led into their development side
by side rather than together within the project. Consequently,
the dimensions of openness framework was
eventually taken for further development and application
also outside the project. In practice, this meant that the
authors, also the initiators of the framework development,
began to develop and apply the dimensions of
openness framework in their work for developing a new,
more holistic approach for environmental health risk
analysis within one of the partnering institutions, the
National Institute for Health and Welfare (THL) from
Finland. The approach was first known as Pyrkilö (originating
from the Finnish word pyrkiä, to strive for), and
later as open risk assessment [76]. Eventually the
method became named open assessment and the webworkspace
for conducting open assessments became
named Opasnet [32,77-79]. A major part of the early
development of the Opasnet web-workspace was particularly
carried out in the BENERIS project.

The method development work took account of several
of the aspects of policy making, assessment, and
participation discussed above, and was influenced by the
research results and experiences e.g. on collective
knowledge creation in fields of education, psychology,
and philosophy [80], computer-supported collaborative
learning [81,82], mass collaboration [83] as well as
crowd-sourcing [84,85]. Application of the framework in
this setting led into a somewhat extreme interpretation
of participatory practice in the context of environmental
health: the assessments should be made completely
open by default and limitations in degree of any dimension
of openness should be done only based on cogent
and explicitly argued reasons! The framework also illuminated
that assessments, the knowledge creation processes,
need to be deeply intertwined with the decision
making processes, the knowledge use, if they seriously
attempt to achieve their purposes of influencing policy.
This makes decision makers a particularly essential kind
of active assessment participants. These ideas, quite contrary
to the currently common conceptions of assessment
and participation, became two of the fundamental
principles guiding open assessment method and Opasnet
web-workspace development. The idea of open assessment
is illustrated in Figure 2.

The unconventionality of the principle of complete
openness, and how it relates to the interrelations
between participation, assessment, and decision making,
is illustrated by the comparison, in terms of the dimensions
of openness, of five assessment approaches relevant
in the field of environment and environmental
health in table 1. The comparison also demonstrates the
application of the framework in evaluating existing practices.
For the sake of clarity the focus of comparison is here on “external participation” in the assessment processes,
e.g. by decision makers, stakeholders or public.
The possible differences in the roles of the expert assessors
nominated for the assessment task are thus not
explicitly considered. The five assessment approaches
included in the comparison are 1) Open assessment
[32,76-79] (State of the art in benefit-risk analysis:
Environmental health. Unpublished manuscript developed
in the BEPRARIBEAN project), IEHIA [31], YVA -
the Finnish environmental impact assessment system
(Finnish EIA Act 468/94) [28], the so called “Red Book”
risk assessment [71], and its recent update, the “Silver
Book” risk-based decision making framework [73].

In addition to describing varying degrees of openness,
the comparison illustrates striking differences in how
the approaches see the interrelations between participation,
assessment, and decision making. Open assessment
and the Red Book risk assessment represent the two
extremes. Whereas the former sees participation, assessment,
and policy making as aspects of the same collaborative
process, the latter does not even consider
participation and explicitly recognizes interaction
between assessment and the external world only in distribution
of assessment results to decision making. The
three other examples, IEHIA, YVA, and Silver Book fall
in between these extremes by allowing some degrees of
openness, although in somewhat different ways, and
identifying linkages between participation, assessment,
and decision making. However, as was mentioned earlier,
the linkage from YVA assessments, and participation
organized within them, is known to be weak. Also
for IEHIA, the description of the relationship between
assessment and decision making remains quite implicit.
The Silver Book perspective makes a radical update to
the Red Book perspective, yet it still retains the risk
assessment as a fundamentally independent and exclusive
expert process.

The scrutiny of openness according to the dimensions
of openness framework can, as exemplified above, reveal
interesting aspects of participation, assessment, and policy
making practices, and their potential to deliver what
they intend to. Although in many discourses participation
seems to be assumed to have “value in itself”, or
plainly seen to “belong to democracy”, from a contentual
point of view this kind of reasoning misses the
main point of openness. Openness is not an end in
itself, but rather a means for advancing societal development
through creation and use of broadly distributed
collective knowledge. Openness calls into question the
assumptions behind the institutional practices that we
have accustomed ourselves to take for granted.

For example, complete openness, as adopted in open
assessment, actually applies an inverse interpretation of
the dimensions of openness, i.e. who should not be
allowed to participate, what information should not be
made available to participants etc. From this perspective
it often becomes difficult to argue e.g. for exclusion
of any specific groups or individuals from
assessments, or withholding important information,
especially if also the arguments are exposed to open
critique. Particularly this is the case in the context of
environment and environmental health, where the
issues addressed are often relevant to virtually everyone
and every organization or individual is a potential
source of relevant contributions. As an example, issues
regarding global climate change involve a nearly infinite
amount of actors in different roles e.g. as contributors
to climate change, its mitigation and
adaptation, or parties affected by impacts of climate
change or its mitigation and adaptation actions.

Openness necessarily also requires a more inclusive
view to assessment than what the conventional conceptions
of assessment provide. Assessment should not
only be confined to mean the expert-driven, natural
science influenced, fact-based, sometimes strictly quantitative,
so called scientific assessments aiming to find
objective answers. It should also extend to explicit
inclusion of values and all kinds of knowledge from all
sources, qualitative treatment of information, and creation
of contextual, situational, and pragmatic knowledge
among assessment participants. Such a
conception of assessment actually ideally also includes
decision making, and other possible uses of knowledge
created by assessment. Although confronted with
“scientific assessment” above, the open conception of
assessment is actually not any less scientific. After all,
the heart of science is in development of shared belief systems based on open critique, evidence, interpretation,
and argumentation, which necessitates openness.
Creation of new fora for scientific discourse, and invitation
of new participants and new topics to enter
these fora does not in itself guarantee scientifically
valid outputs, but it provides possibilities for overcoming
the identified limitations to effectiveness in policy
making, assessment, and participation in the conventional
approaches that build on exclusivity and disengagement
rather than openness.



{|{{prettytable}}
|+'''Table 1. Perspectives to openness in “external participation” for five example assessment approaches considered according to the dimensions of openness framework.'''
|-
!Dimension/Approach
!Scope of participation
!Access to information
!Timing of openness
!Scope of contribution
!Impact of contribution
|-----
|Open assessment
|Everyone, e.g. decision makers, NGO’s, citizens, external experts, allowed to participate. User participation particularly important.
|All information should be made available to all participants.
|Continuous.
|All aspects of the issue can be addressed by everyone.
|Based on relevance and reasoning, not source. All relevant contributions must be taken into account. Conclusions from collaboration intended to turn into action through collective knowledge creation among participants in a shared web-workspace.
|-----
|IEHIA
|Specified users (e.g. policy makers), and stakeholders (preferably by proxy) invited to participate.
|?<ref>? = could not be determined based on the information source</ref>
|User and stakeholder participation during issue framing, design and appraisal phases (not during execution phase).
|Users and stakeholders can participate in scoping and design of assessment and interpretation of results.
|Participant views influence the construction of the assessment framework. Appraisal phase discourse regarding the assessment results, their implications for action, and their linkage to the goals defined in issue framing assumed to ensure that those involved accept the outcomes
|-----
|YVA
|Public, liaison authority (e.g. regional environmental center), other authorities.
|Assessment plan and assessment report provided to the public by the project developer. Liaison authority also has access to information regarding e.g. other plans, projects and operations elevant to the project.
|Participation in two phases. Public hearing periods, possible authority statements regarding both assessment plan and assessment report. Liaison authority gives its statements after the public and the other authority statements.
|Any public representative can give any statements, and the liaison authority may ask specific statements from other authorities in both phases. Liaison authority gives an overall statement on both the assessment plan and the assessment report.
|Public statements filed along with the liaison authority statements. Ultimately up to theproject developers and the decision makers to decide if and how public statements are taken account of in project design or decision making. The liaison authority, also taking account of public and other authority statements, can also demand e.g. certain issues to be considered in the assessment or other additional information to be provided by the project developer.
|-----
|Red Book
|N/A <ref>N/A = not applicable.</ref> (Assessment for nominated scientific experts only).
|N/A
|N/A
|N/A
|Assessment results provided for decision makers and intended to be taken into account, alongside options evaluation, in decision
making and action by federal agencies.
|-----
|Silver Book
|Decision-makers, technical specialists, and other stakeholders.
|Formal provisions for internal and external stakeholders at all stages.
|At all stages: problem formulation and scoping, planning and conduct of risk assessment, and risk
management.
|Problem formulation and scoping, confirmation of utility of risk assessment, and risk management.
|Stakeholders as active participants. However, participation should in no way compromise the technical assessment of risk, which is carried out under its own standards and guidelines.
|}

<references/>

===Challenges of openness===

Openness definitely also brings about significant challenges
in terms of e.g. manageability of broad participation,
information quality control, prevention from
intentional bias or promotion of vested interests, protection
from vandalism, cost and time expenditure, etc.
(cf. [14]), but we claim that these problems are rather
practical than fundamental in their nature. Nevertheless,
they are real challenges to practical implementation
of openness. Methods and tools to help overcome
many of these challenges already exist or are being
developed (see e.g. the references regarding influences
to open assessment development above) and some of
the problems may even be solved by increased openness
itself.

For instance, within the conventional frameworks
openness is often assimilated with time-consuming
appeals, endless disputes, and costly stakeholder involvement
actions. However, well designed and executed
open assessment or policy processes can also mean high
acceptability of outcomes, rapid solutions to welldefined
problems through web-based collaboration, and
improved societal impacts.

Professional assessors and policy makers may, and
often do, fear losing their power in open processes, and
it is true that openness does affect their roles. The
experts in assessment or decision making should see
themselves not as obtaining inputs to their own private
assessment or policy processes, but rather as feeding the
open collaborative processes of assessment and policy
making (as in Figure 2) with their expertise.

However, the law seldom requires very high degrees of
openness, and legislation regarding participation is
usually built on the conventional frameworks. Thereby
the conventions will probably have to change first,
before more support for the shift towards openness can
be expected from legislation regarding environmental
and environmental health assessment and policy making.
Perhaps in the end the greatest challenge lies in the
scientists’, assessors’ and decision makers’ attitudes
towards openness, and the internal resistance to change
contemporary research, assessment and decision making
practices more open.

==Conclusions==

In conclusion, based on the review of literature and
insights from recent research projects, we state that:

# Inclusion of stakeholders and public to participate in assessments and policy making upon issues of environment and environmental health is an issue of both great interest and importance.
# The discourses on both assessments and participation in the contexts of environment and environmental health have been too much focused on processes and procedures, and too little attention has been given to their purposes and outcome effectiveness in policy making.
# Consideration of effective participation is meaningful only in the context of purposes and effects of the assessment and policy making processes that participation relates to.
# The dimensions of openness framework provides a conceptual means for identifying and managing the interrelations between the purposes and outcomes of participation, assessment, and policy making, and thereby also for effective application of existing participatory models and techniques.
# The dimensions of openness framework also provides a context for evaluation and constructive criticism of contemporary conventions and institutions of participation, assessment, and policy making, and a basis for developing new conventions and institutions.
# From a contentual point of view, it can be argued that participation, assessment, and policy making upon environmental and environmental health issues should be considered as completely open rather than exclusive processes by default.
# Openness should not, however, be considered as an end in itself, but rather a means for advancing societal development through creation and use of broadly distributed collective knowledge upon issues of great societal relevance.
# Openness brings about challenges, but they are mostly practical, rather than fundamental in their nature.

==Acknowledgements==

The authors would like to thank the funders of the research considered in
this article; Academy of Finland (grants 53307 & 111775), and EU (BENERIS;
http://en.opasnet.org/w/Beneris, and INTARESE; http://www.intarese.org/
projects sponsored under the 6th EU Framework Programme for Research
and Development).

Special thanks to Madeleine Kangsen Scammell, Glenn Rice, and Seth Tuler
for their constructive criticism and helpful comments on different versions of
this article. Also thanks to Dr. Marko Tainio and Prof. Juha Pekkanen from
THL for their contributions on the precursory draft version of this article.

==Authors’ contributions==

The review of literature and preparation of the paper have mainly been
responsibilities of MVP. JTT has also contributed to the analysis of the
literature in the light of the presented framework and drawing conclusions
based on the analysis. The revisions, as requested by reviewers, were mainly
made by MVP. MVP and JTT are the two main figures behind the
development of the dimensions of openness framework, open assessment
method and the Opasnet web-workspace. All authors read and approved
the final manuscript.

==Competing interests==

The authors declare that they have no competing interests.

Received: 9 December 2010 Accepted: 16 June 2011
Published: 16 June 2011

==References==

1. Fiorino DJ: Citizen participation and environmental risk: A survey of
institutional mechanisms. Science, Technology and Human Values 1990,
15:226-243.

2. Laird FN: Participatory analysis, democracy, and technological decision
making. Science, Technology and Human Values 1993, 18:341-361.

3. Tang B, Wong S, Chi-hong Lau M: Social impact assessment and public
participation in China: A case study of land requisition in Guangzhou.
Environmental Impact Assessment Review 2008, 28:57-72.

4. Wang Y, Morgan RK, Cashmore M: Environmental impact assessment of
projects in the People’s Republic of China: new law, old problems.
Environmental Impact Assessment Review 2003, 23:543-579.

5. UNEP: Rio Declaration on Environment and Development 14 June 1992.
[http://www.unep.org/Documents.Multilingual/Default.asp?documentid=78&articleid=1163].

6. UNECE: Convention on Access to Information, Public Participation in
Decision-making and Access to Justice in Environmental Matters 25
June 1998.[http://www.unece.org/env/pp/documents/cep43e.pdf].

7. ECLAC: Science and Technology for Sustainable Development. A Latin
American and Caribbean Perspective. Report on the Latin American and
Caribbean Regional workshop on Science and Technology for Sustainable
Development, 5-8 March 2002, Santiago, Chile. Series Seminars and
Conferences No. 24 Santiago, Chile: ECLAC; 2002.

8. OECD/NEA: Stakeholder involvement techniques - Short guide and annotated
bibliography. NEA No. 5418 OECD; 2004 [http://www.nea.fr/html/rwm/reports/2004/nea5418-stakeholder.pdf].

9. van den Hove S: Participatory approaches for environmental governance:
Theoretical justifications and practical effects. OECD/NEA: Stakeholder
involvement tools: Criteria for choice and evaluation. Proceedings of a Topical
Session at the 4th meeting of the NEA Forum on Stakeholder Confidence Paris:
OECD; 2003 [http://www.oecd-nea.org/rwm/docs/2003/rwm-fsc2003-10.pdf].

10. Matthews KB, Rivington M, Blackstock K, McCrum G, Buchan K, Miller DG:
Raising the bar? - The challenges of evaluating the outcomes of
environmental modelling and software. Environmental Modelling &
Software 2011, 20:247-257.

11. INTARESE. [http://www.intarese.org].

12. BENERIS. [http://en.opasnet.org/w/Beneris].

13. O’Faircheallaigh C: Public participation and environmental impact
assessment: Purposes, implications, and lessons for public policy
making. Environmental Impact Assessment Review 2010, 30:19-27.

14. IIED: Environmental planning. IIED; 1998, 11[http://www.iied.org/pubs/pdfs/7789IIED.pdf].

15. MNP: Stakeholder Participation Guide for the Netherlands Environmental
Assessment Agency: Main Document, Practice Guide & Checklist Netherlands
Environmental Assessment Agency and Radboud University Nijmegen;
2008.

16. Becker DR, Harris CC, McLaughlin WJ, Nielsen EA: A participatory approach
to social impact assessment: the interactive community forum.
Environmental Impact Assessment Review 2003, 23:367-382.

17. Hamlett PW, Cobb MD: Potential Solutions to Public Deliberation
Problems: Structures Deliberations and Polarization Cascades. The Policy
Studies Journal 2006, 34:629-648.

18. Renn O, Blättel-Mink B, Kastenholz H: Discursive methods in
environmental decision making. Business Strategy and the Environment
1997, 6:218-231.

19. Richardson T, Dusik J, Jundrova P: Parallel public participation: An answer
to inertia in decision-making. Environmental Impact Assessment Review
1998, 18:201-216.

20. Sinclair J, Diduck A: Public education: An undervalued component of the
environmental assessment public involvement process. Environmental
Impact Assessment Review 1995, 15:219-240.

21. Webler T, Kastenholz H, Renn O: Public Participation in Impact
Assessment: A social learning Perspective. Environmental Impact
Assessment Review 1995, 15:443-463.

22. Daniels SE, Walker GB: Collaborative learning: Improving public
deliberation on ecosystem-based management. Environmental Impact
Assessment Review 1996, 16:71-102.

23. Hartley N, Wood C: Public participation in environmental impact
assessment - implementing the Aarhus Convention. Environmental Impact
Assessment Review 2005, 25:319-340.

24. Väntänen A, Marttunen M: Public involvement in multi-objective water
level regulation development projects - evaluating the applicability of
public involvement methods. Environmental Impact Assessment Review
2005, 25:281-304.

25. Newig J: Does public participation in environmental decisions lead to
improved environmental quality? Communication, Cooperation,
Participation (International Journal of Sustainability Communication) 2007,
1:51-71.

26. von Wright GH: Practical inference. The Philosophical Review 1963,
72:159-179.

27. Inkinen A: Whose Knowledge? The role of participation in environmental
decision-making. In Papers from the 8. Nordic Environmental Social Science
Research Conference June 18-20 2007 Workshop 5. Edited by: Balberg I,
Hofstad H. Oslo, Norway: NIBR; 2007:117-132[http://www.nibr.no/uploads/publications/d26bb6876285a7b741ca2f4b00a455d8.pdf].

28. Pölönen I, Hokkanen P, Jalava K: The effectiveness of the Finnish EIA
system - What works, what doesn’t, and what could be improved?
Environmental Impact Assessment Review .

29. Doelle M, Sinclair AJ: Time for a new approach to public participation in
EA: Promoting cooperation and consensus for sustainability.
Environmental Impact Assessment Review 2006, 26:185-205.

30. Pope J, Annandale D, Morrison-Saunders A: Conceptualising sustainability
assessment. Environmental Impact Assessment Review 2004, 24:595-616.

31. Briggs DJ: A framework for integrated environmental health impact
assessment of systemic risks. Environmental Health 2008, 7:61.

32. Pohjola MV, Leino O, Kollanus V, Tuomisto JT, Gunnlaugsdóttir H, Holm F,
Luteijn JM, Magnússon SH, Odekerken G, Tijhuis MJ, Ueland Ø, White BC,
Verhagen H: State of the Art in Benefit-Risk Analysis: Environmental
Health. Environmental Health .

33. ECHA: Guidance on Information Requirements and Chemical Safety
Assessment. Guidance for the implementation of REACH European Chemicals
Agency; 2008.

34. Liu Y, Gupta H, Springer E, Wagener T: Linking science with environmental
decision making: Experiences from an integrated modeling approach to
supporting sustainable water resources management. Environmental
Modelling & Software 2008, 23:846-858.

35. ITRC: Use of Risk Assessment in Management of Contaminated Sites
Washington D.C.: Interstate Technology & Regulatory Council, Risk
Assessment Resources Team; 2008.

36. BEPRARIBEAN. [http://en.opasnet.org/w/Bepraribean].

37. Peltonen L, Sairila R: Integrating impact assessment and conflict
management in urban planning: Experiences from Finland. Environmental
Impact Assessment Review 2010, 30:328-337.

38. Jantunen J, Hokkanen P: YVA-lainsäädännön toimivuusarviointi
-Ympäristövaikutusten arviointimenettelyn toimivuus ja kehittämistarpeet
(Evaluation of the existing EIA legislation - how EIA procedures function in
practice and areas needing improvement) Helsinki: Ministry of the
Environment; 2010.

39. Pohjola MV, Tuomisto JT, Tainio M: Properties of good assessment: The
factors that contribute to assessment performance [abstract]. Proceedings
of the Society for Risk Analysis Europe annual meeting, King’s College, London
21-23 June 2010 SRAE; 2010, 65[http://www.sraeurope.org/filedown.aspx?file=3778].

40. Cashmore M: The role of science in environmental impact assessment:
process and procedure versus purpose in the development of theory.
Environmental Impact Assessment Review 2004, 24:403-426.

41. Jay S, Jones C, Slinn P, Wood C: Environmental impact assessment:
Retrospect and prospect. Environmental Impact Assessment Review 2007,
27:287-300.

42. Sterk B, Carberry P, Leeuwis C, van Ittersum MK, Howden M, Meinke H, van
Keulen H, Rossing WAH: The Interface between land use systems
research and policy: Multiple arrangements and leverages. Land Use
Policy 2009, 26:434-442.

43. van Kerkhoff L, Lebel L: Linking Knowledge and Action for Sustainable
Development. Annual Review of Environment and Resources 2006,
31:445-477.

44. Castán Broto V, Gislason M, Ehlers MH: Practising interdisciplinarity in the
interplay between disciplines: experiences of established researchers.
Environmental Science and Policy 2009, 12:922-933.

45. Pohl C: Transdisciplinary collaboration in environmental research. Futures
2005, 37:1159-1178.

46. Pohl C: From science to policy through transdisciplinary research.
Environmental Science and Policy 2008, 11:46-53.

47. Wickson F, Carew AL, Russell AW: Transdisciplinary research:
characteristics, quandaries and quality. Futures 2006, 38:1046-1059.

48. Jasanoff S: Procedural choices in regulatory science. Technology in Society
1995, 17:279-293.

49. Bammer G: Integration and implementation sciences: Building a new
specialization. Ecology and Society 2005, 10:6[http://www.ecologyandsociety.org/vol10/iss2/art6/].

50. Funtowicz S, Ravetz JR: Science for the Post-Normal Age. Futures 1993,
25:739-755.

51. van Kerkhoff L: Integrated research: concepts of connection in
environmental science and policy. Environmental Science and Policy 2005,
8:452-463.

52. Cohen EB: A philosophy of informing science. Informing Science: the
International Journal of an Emerging Transdiscipline 2009, 12:1-15[http://
www.inform.nu/Articles/Vol12/ISJv12p001-015Cohen399.pdf].

53. Michaels S: Matching knowledge brokering strategies to environmental
policy problems and settings. Environmental Science and Policy 2009,
12:994-1011.

54. Sheate WR, Partidário MR: Strategic approaches and assessment
techniques - Potential for knowledge brokerage towards sustainability.
Environmental Impact Assessment Review 2010, 30:278-288.

55. Jacobs K, Garfin G, Lenart M: More than just talk: Connecting science and
decisionmaking. Environment 2005, 47:6-21.

56. Guston DH: Boundary organizations in environmental policy and science:
An introduction. Science, Technology & Human Values 2001, 26:399-408.

57. Bowker GC, Star SL: Sorting Things Out: Classification and Its Consequences
Cambridge, MA: MIT Press; 1999.

58. Cash DW, Clark WC, Alcock F, Dickson NM, Eckley N, Guston DH, Jäger J,
Mitchell RB: Knowledge systems for sustainable development. Proceedings
of the National Academies of Science of the United States of America 2003,
100:8086-8091[http://www.pnas.org/content/100/14/8086.full].

59. van den Hove S: A rationale for science-policy interfaces. Futures 2007,
39:807-826.

60. Kloprogge P, van der Sluijs J: The inclusion of stakeholder knowledge and
perspectives in integrated assessment of climate change. Climatic
Change 2006, 75:359-389.

61. Lemos MC, Morehouse BJ: The co-production of science and policy in
integrated climate assessments. Global Environmental Change 2005,
15:57-68.

62. Holmes J, Clark R: Enhancing the use of science in environmental policymaking
and regulation. Environmental Science and Policy 2008, 11:702-711.

63. Juntti M, Russell D, Turnpenny J: Evidence, politics and power in public
policy for the environment. Environmental Science and Policy 2009,
12:207-215.

64. Sanderson I: Evaluation, policy learning and evidence-based policy
making. Public Administration 2002, 80:1-22.

65. Gunderson L, Light SS: Adaptive management and adaptive governance
in the everglades ecosystem. Policy Sciences 2006, 39:323-334.

66. Lynch AH: Adaptive governance: how and why does government policy
change? ECOS Magazine 2009, 146:31.

67. Nelson R, Howden M, Stafford Smith M: Using adaptive governance to
rethink the way science supports Australian drought policy.
Environmental Science and Policy 2008, 11:588-601.

68. Mickwitz P, Aix F, Beck S, Carss D, Ferrand N, Görg C, Jensen A, Kivimaa P,
Kuhlicke C, Kuindersma W, Máñez M, Melanen M, Monni S, Branth
Pedersen A, Reinert H, van Bommel S: Climate policy integration,
coherence and governance. PEER Report No 2. Helsinki: Partnership for
European Environmental Research; 2009 [http://peer-initiative.org/media/m235_PEER_Report2.pdf].

69. Brunner RD: A paradigm for practice. Policy Sciences 2006, 39:135-167.

70. Hajer M: Policy without polity? Policy analysis and the institutional void.
Policy Sciences 2003, 36:175-195.

71. NRC: Risk Assessment in the Federal Government: Managing the Progress
Washington D.C: National Academy Press; 1983.

72. NRC: Understanding Risk: Improving Decisions in a Democratic Society
Washington D.C.: National Academy Press; 1996.

73. NRC: Science and Decisions: Advancing Risk Assessment Washington D.C.:
National Academy Press; 2009.

74. Burgess J, Clark J: Evaluating public and stakeholder engagement
strategies in environmental governance. In Interfaces between science and
society. Edited by: Guimarães Pereira A, Guedes Vaz S, Tognetti S. Sheffield:
Greenleaf Publishing; 2006:225-252.

75. Webler T, Tuler S: Fairness and Competence in Citizen Participation -
Theoretical Reflections From a Case Study. Administration & Society 2000,
32:566-595.

76. Tuomisto JT, Pohjola MV: Open Risk Assessment - A new way of providing
information for decision-making. Publications of the National Public Health
Institute B18/2007 Kuopio: KTL - National Public Health Institute; 2007.

77. Opasnet. [http://en.opasnet.org].

78. Pohjola MV, Pohjola P, Paavola S, Tuomisto JT: Knowledge services in
support of converging knowledge, innovation, and practice. In
Proceedings of I-KNOW 2010: 1-3 September 2010; Graz, Austria. Edited by:
Tochtermann K, Maurer H. J. UCS; 2010:255-266[http://i-know.tugraz.at/proceedings].

79. Pohjola MV, Pohjola P, Paavola S, Bauters M, Tuomisto JT: Pragmatic
Knowledge Services. Journal of Universal Computer Science 2011,
17:472-497.

80. Paavola S, Hakkarainen K: The Knowledge creation Metaphor - An
Emergent Epistemological Approach to Learning. Science & Education
2005, 14:535-557.

81. Hakkarainen K: A knowledge practice perspective on technologymediated
learning. Computer-Supported Collaborative Learning 2009,
4:213-231.

82. Cress U, Kimmerle J: A systemic and cognitive view on collaborative
knowledge building with wikis. Computer Supported Collaborative Learning
2008, 3:105-122.

83. Tapscott D, Williams AD: Wikinomics: How Mass Collaboration Changes
Everything Toronto: Penguin Group; 2006.

84. Noveck B: Wiki Government Brookings Institution Press; 2009.

85. The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How
Collective Wisdom Shapes Business, Economies, Societies and Nations Random
House; 2004.