Evaluating effectiveness of open assessments on alternative biofuel sources

From Opasnet
Jump to navigation Jump to search



This is a final manuscript submitted to Sustainability: Science, Practice & Policy (style). The text was originally written in heande:Evaluating effectiveness of open assessments on alternative biofuel sources. The article was published as:
Sandström, Vilma; Tuomisto, Jouni T.; Majaniemi, Sami; Rintala, Teemu; Pohjola, Mikko V.: Evaluating effectiveness of open assessments on alternative biofuel sources. Sustainability: Science, Practice & Policy (2014): 10(1) Published online May 16, 2014 (and republished in 2017?). doi:10.1080/15487733.2014.11908132 [1]

Title, authors and affiliations

Evaluating effectiveness of open assessments on alternative biofuel sources

Vilma Sandström1,2*, Jouni T. Tuomisto1*, Sami Majaniemi1, Teemu Rintala1, Mikko V. Pohjola1
1National Institute for Health and Welfare
2University of Helsinki, Faculty of Biological and Environmental Sciences, Department of Environmental Sciences
*Corresponding author
Vilma Sandström
National Institute for Health and Welfare
P.O.Box 95, FI-70701 Kuopio, Finland
vilma.sandstrom(at)helsinki.fi
Jouni T. Tuomisto
National Institute for Health and Welfare
P.O.Box 95, FI-70701 Kuopio, Finland
jouni.tuomisto(at)thl.fi
Sami Majaniemi
National Institute for Health and Welfare
P.O.Box 95, FI-70701 Kuopio, Finland
sami.majaniemi(at)thl.fi
Teemu Rintala
National Institute for Health and Welfare
P.O.Box 95, FI-70701 Kuopio, Finland
teemu.rintala(at)thl.fi
Mikko V. Pohjola
National Institute for Health and Welfare
P.O.Box 95, FI-70701 Kuopio, Finland
mikko.pohjola(at)thl.fi

Abstract

Biofuels have raised controversial debate regarding environmental, social and economical aspects and sustainability. The complexity of biofuel decisions and investments by the industry and the society necessitates inclusion of knowledge, information, and opinions from a diversity of sources.

Environmental assessments estimate environmental and other impacts of the options before a decision. Open, collaborative knowledge creation can support decisions in two ways: by building trust and credibility, and by developing a more sound knowledge base. Open assessment is a decision support method that allows open participation in assessments with transparent and freely accessible process. In this paper we evaluate two open assessment case studies about biodiesel production decisions. The evaluation compiles the views of all participants regarding the potential of the assessments to influence the decisions in terms of quality of content, applicability, and efficiency as well as openness.

According to the evaluation, openness can be feasibly implemented and it was much appreciated by the participants. More experiences on broad and active participation are still needed for further development of methods and tools. The currently common practices and attitudes for closed and disengaged processes seem to limit open decision support the most. Chosen assessment tools can also limit broad participation.

Introduction

Growing energy demand and greenhouse gas emissions from fossil fuels have increased the interest in the production of renewable energy. Within these biofuels, i.e. liquid or gaseous fuels for transport produced from biomass (EU, 2009), are of special interest. Biofuel production is a fast growing industry and it has received a lot of attention as it is considered e.g. to aid in reducing greenhouse gas (GHG) emissions from the transport sector, decrease the dependence on fossil fuels and contribute to the economic growth of the developing countries (Ryan et al. 2005; Mathews, 2008; Cassman & Liska, 2008). To help to reach the GHG emission reduction goals European Union has set a target to cover at least 10 % of the energy demand of the transport sector with renewable energy resources by 2020 (EU, 2009).

However the sustainability of the production has been criticized and whether biofuels offer carbon savings depends on how they are produced. Converting native ecosystems to biofuel production can create so called “biofuel carbon debt” and release many times more CO2 comparing with the annual GHG reductions that these biofuels would provide by replacing fossil fuels (Fargione et al. 2008). The use of nitrogen fertilizers in the crop production of commonly used biofuels such as biodiesel from rapeseed and bioethanol from corn, can contribute as much or more to global warming as the combustion of fossil fuels (Crutzen et al. 2008). Additionally the production itself can require more energy input from fossil fuels than can be created (Pimentel and Patzek, 2005). Furthermore many other ethical and environmental issues have been addressed, such as the conflict between biofuel production and the global food security, the use of limited water resources, land ownership and the conflicts between land owners and indigenous territories, competition with grazing wild and domesticated animals and the possible threats to biodiversity and soil fertility (Gomiero et al. 2010, Tilman et al. 2009, Giampetro and Mayumi 2009). Due to the many controversial issues associated with the biofuel production the sustainability of the production can vary significantly and their use may not be such an easy and trouble-free solution to climate change as at first might have appeared.

When deciding about new investments in biofuel production and supply, industrial decision makers have to consider a wide range of scientific and non-scientific information that has to do with financial, environmental and social aspects of the production chain. For example, the GHG-emissions as well as other impacts and costs of the production can vary considerably between different raw materials and production sites. Also the policies and the views of stakeholders and the general public on local, regional and global level pose significant constraints on the biofuel investment decisions.

Different kinds of assessments (see e.g. Pohjola et al. 2012), which may apply models (see e.g. Jakeman et al. (Eds.) 1998) as well as decision analytical methods (e.g. multicriteria analysis, Zopounidis and Pardalos 2010), can be applied as means to obtain knowledge to support such decisions. They can also be accompanied with various participatory techniques (see e.g. Pohjola and Tuomisto 2011). However, there are many possible limitations to the effectiveness of the support they provide (Matthews et al. 2011, Pohjola and Tuomisto 2011, Pohjola et al. 2012, Pohjola et al. manuscript). In situations where implications of decisions are complex and difficult to anticipate, multiple needs, interests and sources of knowledge must be taken account of in decision making (Figure 1).

Figure 1. Complex decisions need to take account of multiple needs, interests and sources of knowledge (reproduced from Tijhuis et al. 2012 with permission).

Public participation and stakeholder involvement in assessment and policy making is built on the ideas of democracy (Fiorino, 1990), and it is seen to enhance acceptance, integrate local knowledge to the scientific information and produce more flexible and transparent decisions (Reed, 2008; van den Hove, 1999). However, the emphasis of environmental assessment is claimed to often be more on process and procedure, rather than its purpose and effects (Cashmore, 2004; Jay et al. 2007) and only few approaches to assessment actually even explicitly consider assessment performance in terms of the outcomes of using the assessment results in their intended contexts of use (Pohjola et al. 2012, Pohjola et al. manuscript). Correspondingly, although participation is increasingly appreciated as a part of environmental assessment and decision making, its implementations have often concentrated in process and access rather than outcomes (Doelle & Sinclair, 2006).

Open assessment method was created to provide a means for more purpose-driven and effective support to decision making in open collaboration (Pohjola & Tuomisto 2011, Pohjola et al. 2012). It aims to support decision making by means of systematic analysis of different decision options and providing a forum for all involved parties to collect and integrate knowledge and views and influence the decisions. As its name implies, open assessment differs from most common assessment approaches in terms of openness. In principle, everyone is allowed to participate in and contribute to open assessments (Pohjola and Tuomisto 2011). In addition, tight linkage between assessments and the use of their results is seen as a necessity for effectiveness (Pohjola et al. 2012, Pohjola et al. manuscript). Ideally, everything in the process should be transparent and all content subject to scientific criticism. The method emphasises substantive content over participatory procedures. This means that the assessors seek all relevant views rather than a set of views expressed by a balanced representative group, and that arguments are evaluated based on how they hold against criticism rather than how many participants support them. In this thinking, there is no need to discriminate participants with vested interests as long as there is large enough a pool of participants and information sources. As long as the above mentioned fundamental principles (for more detailed list of principles, see Pohjola et al. 2011) are not violated, open assessments can address almost any topics and apply many kinds of methods for assessing impacts, risks, benefits etc. and analyzing decision options.

Open assessments are usually conducted in the Opasnet web-workspace (http://www.opasnet.org, Pohjola et al. 2011), which is particularly designed to facilitate open assessments and hosts functionalities that are useful for open assessment. Open assessment and Opasnet have been developed and applied as one of the central assessment tools in Department of Environmental Health of the National Institute for Health and Welfare in Finland (THL) since 2006. The Opasnet web-workspace is discussed more in the next section. Currently the Opasnet in English contains in total 2600 pages and more than 30 open assessments with a wide variety of topics including fine particle air pollution, biofuels, swine flu vaccination, farmed salmon, metal mines, city-level climate policies, environmental impact assessment law, and the government plan of Finland (http://en.opasnet.org/w/Category:Assessments). Most, but not all, of the assessments have health outcomes in focus. Although open assessment and Opasnet have until now been mostly applied by its developers, collaborators in assessments and users of assessment results have come from many context, both from within Finland and internationally. Opasnet is in the open internet and is available for use by anyone.

In this paper we evaluate two open assessment case studies that assessed the feasibility of two alternative biodiesel feedstock, Jatropha curcas oil plant (later referred to as Jatropha) and the oil extracted from fish waste by-products from fish farming. The cases were limited to only assessing biodiesel production, not bioalcohol or other forms of bioenergy. The assessments were requested and financed by Neste Oil Corporation and they were performed by THL in the open Opasnet workspace (http://en.opasnet.org). The information produced in the assessment was primarily collected and analyzed in order to support the decision making processes of the primary user (Neste Oil), but at the same time it was also intended to be applicable in societal decision making processes in other situations.

This paper aims to contribute to the development of improved assessment and policy making methods, tools and practices by evaluating the effectiveness of two open assessment cases in terms of their quality of content, applicability, efficiency and openness (Pohjola & Tuomisto 2011, Pohjola et al. manuscript). By effectiveness we fundamentally mean influence of assessment on the outcomes, i.e. changes in values, attitudes, and behavior in the society (Matthews et al. 2011), but in practice what is possible to evaluate rather reflects the likelihood of an assessment to achieve the desired results and goals set for it (Hokkanen & Kojo, 2003). Open assessment has been developed and argued as a means for overcoming the limitations of effectiveness in science-based decision support (cf. Matthews et al. 2011, Pohjola and Tuomisto 2011, Pohjola et al. manuscript). Therefore, evaluation of its application is important both to the development of open assessment and Opasnet, but also to the development of science-based decision support in general. Correspondingly, the research questions we sought answers for were:

  1. Was open assessment method and the Opasnet workspace feasible means for the assessments?
    • Did they provide means for all participants to influence the assessments?
    • Did the assessments influence the knowledge of the primary users and other participants?
  2. Was the applied evaluation approach feasible for evaluating assessment effectiveness?

First we present the two open assessment case studies, discuss the theory behind our evaluation approach, and describe the implementation of the evaluation. Then we present and discuss the results of evaluation, and ultimately draw conclusions upon their implications to assessment theory and practice.

Material and methods

Open assessment case studies

Figure 2. In open assessment, members of a society adopt different roles in relation to identifying needs, making assessments, making decisions, and taking actions. An assessment page in Opasnet has a central role in collecting observations (yellow arrows; here of an undesired event, a toxic liquid spill) and spreading information (green arrows) to and from members. Knowledge-based actions (blue arrow) are taken to clear up the spill. (Note: the picture version used is the version 24 Nov 2012.
Figure 2. Collaborative knowledge creation in open assessment. Reproduced from Pohjola et al. 2012 with permission. ⇤--#: . This is an alternative Figure 2 as suggested by Reviewer 2. --Jouni 10:27, 24 November 2012 (EET) (type: truth; paradigms: science: attack)

Environmental health assessments are typically rather limited in openness (Pohjola et al. 2012), and therefore there is a need for a proof of concept for more open approaches. many of the previous open assessments by THL had been motivated by scientific rather than practical interests. Therefore, THL was interested in performing this kind of assessments in order to test open assessment in a real-life situation with a clear customer need.

The basic idea behind open assessments is to collect information and create knowledge that is needed for decision making in broad collaboration among participants (Figure 2). The information is organised as an assessment that predicts the impacts of different decision options on some outcomes of interest. Decisions, outcomes, and other issues are modelled as distinct parts that are, in practice, web pages in Opasnet, an open web-workspace dedicated for making these assessments.

In Opasnet, users can collect, synthesise, describe, discuss, and distribute information using a wiki; upload data to a database; upload files; and build and run computational models. The main interface between Opasnet and its users is the wiki. It is built on the Mediawiki platform, and many of its basic functions resemble those e.g. in Wikipedia (http://en.wikipedia.org), but many additional functionalities e.g. for modeling and oragnization of information have been developed. The wiki provides a forum for the participants to collaborate upon finding well-founded solutions to practical problems (see Figure 2). Opasnet is located in the open internet. Anyone can read its contents and post comments on its pages. Editing of the pages, however, requires logging into the system. Anyone can create a user account.

Both assessment cases were performed in Opasnet and each part of the assessments (e.g. cultivation of Jatropha, amounts of oil produced from the harvest, or social impacts) were described, discussed and estimated on a separate page in the workspace. Both assessments were done and published in Opasnet in Finnish, but a summary is also provided in English (http://en.opasnet.org/w/Biofuel_assessments).

The two biofuel assessments in question were requested and financed by Neste Oil Corporation (a Finnish refining and marketing company focusing on advanced, cleaner traffic fuels, http://www.nesteoil.com) and they were performed by THL between June 2011 and February 2012. The primary aim of the assessments was to investigate the feasibility of the two potentially interesting alternative raw materials in biodiesel production, Jatropha and waste oil from fish industry. The focus was on the environmental, climate, and social impacts and the acceptance of the production by Finnish stakeholders. The production of biodiesel from waste oil from fish industry was considered to take place in Southeast Asia using local or regional raw material sources. The location for biodiesel production from Jatropha and origin of raw material was not specified.

Previously, Neste Oil had assessed that palm oil is an ecological and economical alternative to fossil fuels. To their surprise, there was a major outrage and anti-Neste Oil campaign after Neste Oil started fuel production based on palm oil in Singapore in 2010 (http://www.greenpeace.org/finland/en/media/Press-releases/Protest-against-Neste-Oils-palm-oil-diesel/). Therefore, Neste Oil had an interest in understanding how open, participatory assessments would work in practice and seeing whether open assessments could explore the attitudes about certain topics in the society. Jatropha and waste oil from fish industry had already been identified as potential economically and technically feasible raw materials for biodiesel production. As a Finland-based company, Neste Oil was primarily interested in the views of the Finnish stakeholders. however, participants were invited to discuss the issue from a global or local point of view. In addition to producing supporting information to Neste Oil and societal decision makers, the assessments aimed to increase the awareness and knowledge of stakeholders and public on the two new potential alternatives for biofuel sources.

The work started in June 2011. The topics of interest and underlying motivation were clarified in discussions with THL and Neste Oil. Assessments started as exploratory, because no exact assessment questions were given in the beginning. A key interest was to identify potential reasons for not using Jatropha or waste oil from fish industry in biofuel production. In the beginning, more attention was paid to Jatropha, because the first information sources found in the beginning of the work were optimistic about Jatropha cultivation on poor lands. The scoping of the fish oil case was not clarified in the beginning, and the assessment team started by looking at ocean fishing; the current megatrend of depletion of ocean fishery stocks (e.g. Myers & Worm, 2003) was seen as a major obstacle of using waste oil from fish industry in large quantities at least for long periods of time. Only later, the scope was redirected to South East fish farming by Neste Oil.

The assessments were performed by a group of assessment coordinators (a group of four of environmental scientists, open assessment experts, and modellers). In addition seven university or high school students worked as summer trainees doing most of the practical work in the assessments. The coordinators were not experts in Jatropha, fish industry, or fuel distillation, but the focus was on environmental issues and attitudes. Neste Oil participated in the assessments as the primary user of their results and contributed mostly in assessment scoping. In addition, a group of stakeholders was invited to contribute to any parts of the the assessments. As the assessments were conducted as open assessments, it was possible for also anyone else to participate.

Most of the assessment work was done between June and August 2011. The information sources included both scientific and non-scientific journal articles and web pages. The results were summarised on wiki pages of the Opasnet assessment workspace (numerically when possible) and used as parts of the computational models built for the assessments. Uncertainties were described as probabilities handled by Monte Carlo simulation algorithms. The models, coded in an open source R language, were built in the same web workspace as all the text content, and they can be read and run by anyone directly from the web pages.

In September 2012, THL and Neste Oil organised a meeting where the draft results were presented and discussed. It had been found out that Jatropha is actually not a very productive plant unless cultivated on rich land, possibly leading to land use competition with food crops. In contrast, waste oil from fish industry turned out to be more interesting, especially when Neste Oil wanted to focus on large fish farming industry in Southeast Asia. This stimulated a period of new data collection. Key findings are presented in http://en.opasnet.org/w/Biofuel_assessments.

Assessment coordinators contacted stakeholder groups in October and November 2011 and invited them to participate. Participation intended i) to inform stakeholders about the assessments and their results, ii) to help in collecting further information and iii) to get a comprehensive picture of different stakeholder’s views. Altogether 18 stakeholders including 3 energy companies, a human rights organization, 5 environmental organizations, 10 researchers, research centers or expert organisations (detailed list at http://en.opasnet.org/w/Biofuel_assessments) were invited to collaborate by commenting on the existing assessment and introducing possible lacking information. Feedback was received from six groups or individuals. All the process took place in a wiki-based internet page where anyone could comment.

The stakeholders were asked to comment on the assessments and to argue whether from their point of view it was feasible to invest in Jatropha or waste oil from fish industry as feedstock for biodiesel production. All feedback was included in the assessments in the form of formal argumentation and relevant page contents. Conclusions were updated when warranted. A few new aspects were raised (e.g. about the role of EU climate policies), but the main conclusions of the study did not change due to the feedback.

The main conclusion about Jatropha was that it might be useful in small-scale fuel oil production, especially if the plant has other simultaneous uses such as prevention of erosion, but not in large-scale industry. The main conclusion about waste oil from fish industry was that it is a promising source and seems to be available in large enough quantities at least in Southeast Asia, but ecologically it has dual impacts and the balance is uncertain: it reduces the side stream of fish industry waste, but it may stimulate the primary process of fish farming and its potentially harmful impacts.

In February 2012, the assessments had reached a sufficient degree of maturity. The website and a small seminar were the final products for Neste Oil. Finally, all participants (Neste Oil, stakeholders, summer trainees, and coordinators) were asked to evaluate the final output and the making of the assessments. The evaluation is described in more detail below.

Assessment effectiveness

While the assessments were conducted according to an approach allowing open participation via a web-based assessment workspace throughout the assessments, also a novel approach to evaluating effectiveness of assessment was adopted. This approach is based on the frameworks Properties of good assessment and Dimensions of openness recently developed in the EU-funded INTARESE (Integrated Assessment of Health Risks of Environmental Stressors in Europe) and BENERIS (Benefit–Risk Asessment of Food: An iterative Value- of-Information approach) projects, and it considers assessment effectiveness in terms of quality of content, applicability, efficiency and openness (Pohjola & Tuomisto 2011, Pohjola et al. manuscript, Tuomisto & Pohjola 2007). The actual objects of interest in evaluating effectiveness of assessments are the changes they provoke (Matthews et al. 2008), but as it would require follow-up and post-hoc analysis, such evaluation would provide little guidance on the assessment in question. Therefore the evaluation approach adopted here focuses on identifying the potential of the assessment to serve its explicated purposes (cf. Hokkanen & Kojo 2003), and thereby providing guidance already to the design and execution phases of the assessment. Simultaneously it also creates a basis for possible post-hoc analyses effectiveness that address the realization of that potential. It should be noted, however, that in the case study assessments discussed in this paper the evaluation of assessment effectiveness was done only after the delivery of results, not so much as an intrinsic part of conducting the assessment.

In this study, two separate frameworks were utilized for the evaluation of the effectiveness and performance of the assessment. The first framework, called dimensions of openness (Pohjola & Tuomisto, 2011) was designed as a tool for characterizing the approaches and settings of supporting decision making by means of science-based analysis and participation. It considers the possibilities and constraints for assessors and participants to influence the decisions and consequent actions in terms of:

  • Scope of participation, Who are allowed to participate in the process?
  • Access to information, What information about the issue is made available to participants?
  • Timing of openness, When are participants invited or allowed to participate?
  • Scope of contribution, To which aspects of the issue are participants invited or allowed to contribute?
  • Impact of contribution, How much are participant contributions allowed to have influence on the outcomes? In other words, how much weight is given to participant contributions?

In this study, the framework "dimensions of openness" was primarily applied to evaluate the effectiveness of participation by characterizing the possibilities of the invited stakeholders to influence the assessment as well as the decisions and actions of the primary user (Neste Oil) through the assessment. It should be noted, however, that the framework itself is not limited to considering only external participation, but all activities in assessment-policy interaction.

The second framework (the properties of good assessment -framework) was designed as a tool for evaluating and managing performance of models and assessments, particularly in the context of environment and health (Pohjola et al. manuscript, Tuomisto & Pohjola 2007, http://en.opasnet.org/w/Properties_of_good_assessment). It considers the potential of the processes and outputs of assessments to meet their explicated purposes and influence the decision processes and consequential actions that they address in terms of their i) quality of content, ii) applicability, and iii) efficiency (Table 1). The first version of the framework was published in 2007 (Tuomisto & Pohjola 2007) and it has recently been updated (http://en.opasnet.org/w/Properties_of_good_assessment). In this study we applied a slightly simplified version (Table 1.), particularly emphasizing the properties characterizing applicability, as the basis for the participant evaluation questionnaires.

Table 1. A simplified version of the properties of good assessment as applied as a basis for the evaluation questionnaire for the participants of the biofuel assessment.
Category Description Questions
Quality of content Specificity, exactness and correctness of information. Correspondence between questions and answers. How many possible worlds does the answer rule out? How few possible interpretations are there for the answer? How close is the answer to reality or real value? How completely does the answer address the assessment question? Is everything addressed? Is something unnecessary?
Applicability Relevance: Correspondence between output and its intended use. How well does the information provided by the assessment serve the needs of the users? Is the assessment question good in relation to the purpose of the assessment?
Availability: Accessibility of the output to users in terms of e.g. time, location, extent of information, extent of users. Is the information provided by the assessment available when, where and to whom is needed?
Usability: Potential of the information in the output to generate understanding among its user(s) about the topic of assessment. Can the users perceive and comprehend the information provided by the assessment? Does users' understanding increase about the assessed issue?
Acceptability: Potential of the output being accepted by its users. Fundamentally a matter of its making and delivery, not its information content. Is the assessment result (output), and the way it is obtained and delivered for use, perceived as acceptable by the users?
Efficiency Resource expenditure of producing the assessment output either in one assessment or in a series of assessments. How much effort is spent in the making of an assessment? If another (somewhat similar) assessment was made, how much (less) effort would be needed?


In Table 1, the description column provides a general explanation of the meaning of each category or property. The question column then attempts to explicate what is intended by the description by providing sample questions that could be asked in evaluating a model or assessment in terms of that category or property. The category quality of content characterizes the information content in the assessment output. The properties under applicability characterize both the output and the process according to their capability of delivering the information content to the intended use. Attributes under efficiency characterize how much output is delivered with the spent effort.

The two above described frameworks, dimensions of openness and properties of good assessment, overlap most apparently in relation to the properties under the applicability category. For example, have the needs of different participants been taken account of in scoping and question setting of the assessment (relevance), to what extent is it possible for different participants to contribute to assessment (availability), is assessment content comprehensible to all participants (usability), and are there possibilities to participate to an acceptable degree (acceptability). The focus in this study are, however, in evaluating the potential of participatory influence to assessment and the potential of the assessment to influence its primary users as well as other participants.

Evaluation of assessment effectiveness

After completion of the assessments and delivery of results to the primary user, all participants were contacted again and asked to evaluate the performance of the assessments by completing evaluation questionnaires based on the properties of good assessment and dimensions of openness -frameworks described above (see http://en.opasnet.org/w/Talk:Biofuel_assessments for the questionnaires). To get a comprehensive feedback of the assessments, the questionnaires were sent to the primary users, invited stakeholders and summer trainees as well as the assessment coordinators at THL.

Due to different roles and perspectives adopted, the questionnaires were slightly modified for each participating group (see also Table 2). The question 1 on inclusion of stakeholder contributions was targeted only to stakeholder representatives. Questions 2-6, addressing quality of content and applicability of assessment, were targeted to all groups. Question 7 on assessment efficiency was targeted only to the primary users (see http://en.opasnet.org/w/Talk:Biofuel_assessments for questionnaire). Respondents were free to choose which questions they provided their answers to, and some respondents left some questions unanswered e.g. due to not feeling capable or willing to evaluate those aspects of the assessments. Both assessments (Jatropha and the fish oil cases) were evaluated together, but so that the respondents were allowed to provide differing numerical values or comments on each assessment if needed. For each question, the respondents were asked to provide a numerical integer value between the range 1-5 (1 meaning bad and 5 meaning good). In addition, textual comments were asked to accompany the numerical values.

As the number of respondents remained low, in total 12, it was only possible to calculate the characteristics of the numerical data. In addition we tested the medians of each question, as well as averages for questions 3-6 (applicability) and questions 1-7 (effectiveness), for deviation from value 3 with Wilcoxon signed-rank test. The respondent averages were calculated using all existing values and omitting missing values, and the averages were interpreted as continuous and normally distributed variables The significance level was set as p < 0.05. The analysis was made for the numerical questionnaire results using R 2.14.2 software. The data and the code (including also an ordered logit model and one-way ANOVA with Tukey post-foc test for testing differences between respondent groups, which is not considered in this paper) is available at http://en.opasnet.org/w/Talk:Biofuel_assessments#Analyses.

The results based on numerical data from the questionnaires can, however, only be considered as providing some indication of possible variation between different properties addressed in different questions of the questionnaire. Instead, the main focus of evaluation is on the textual comments provided with the numerical responses. These are scrutinized along with informal communications during the assessments, subjective experiences among the assessment coordinators, as well as the indications based on numerical results in the next section as an overall evaluation of the assessment effectiveness in these two cases.

Results

The numerical results of the evaluations asked from all participants provide an indication of possible variation between questions and question sets. Altogether evaluations were received from 12 respondents. Three of these were from primary users, 4 from stakeholders (1 with only textual comments), 2 from summer trainees, and 3 from assessment coordinators. The range of possible values in numerical evaluation was 1-5. The characteristics and results of the Wilcoxon signed rank test are given in Table 2.

Table 2. Characteristics of the numerical data from the evaluation questionnaires and results of Wilcoxon signed rank test. * = significant (p < 0.05) deviation from value 3.
Question type Question N Min Max Mean Variance Median
Influence of participation (stakeholders only) 1. Are stakeholder comments included well in assessment? 2 3 3 3 0,000 3
Quality of content 2. Is assessment question answered accurately, truthlikely and comprehensively? 10 3 4 3.70 0.233 4*
Applicability 3. Relevance: Does assessment serve well your (organization's) knowledge needs? 9 1 4 3.44 1.028 4
4. Availability: Has assessment content reached you (your organization)? 7 1 5 3.29 1.571 3
5. Usability: Has assessment influenced your understanding on the assessed issues? 11 1 5 3.55 2.673 4
6. Acceptability: Was assessment made in a good and acceptable way? 9 3 5 4.22 0.444 4*
Questions 3-6 average 11 1.00 5.00 3.59 1.072 3.67*
Efficiency (Primary users only) 7. How good is assessment output given the resources used? 3 4 5 4.67 0.33 5
Effectiveness Questions 1-7 average 11 1.80 4.70 3.72 0.609 4.0*

As the Table 2 shows the means and medians of the numerical evaluations for all questions are at least 3. Of these, the question 2 on quality of content and question 6 on acceptance alone as well as the average for questions 3-6 on applicability as whole and average for all questions 1-7 (effectiveness) are statistically significantly higher than value 3 according to the Wilcoxon signed-rank test. In addition, the median for the question on efficiency shows to be almost statistically significantly (p = 0.102) higher than value 3.

On a general level, the numerical results can be interpreted to indicate that the participants evaluated the assessments at least moderately effective across all questions. Looking at the data it seems that the external participants, i.e. users and stakeholders, would be slightly more critical in their evaluation than the internal participants, i.e. summer trainees and assessment coordinators, regarding some questions. The data is, however too small for reliably testing this with statistical analysis (for more, see http://en.opasnet.org/w/Talk:Biofuel_assessments#Analyses).

These findings are applied as complementary to the analysis of textual comments by respondents discussed below in terms of openness, quality of content, applicability and efficiency.

Openness

In theory everyone was allowed to participate in the biofuel assessments. In practice the active participation was mainly reduced to the three representatives of the user (from Neste Oil) initiating and following the assessment, the seven summer trainees doing most of the assessment work, the four coordinators guiding the assessment work, and the eighteen stakeholders that were invited to participate and out of which six actually contributed to the assessment. The webpage statistics, however, show that there was also some passive participation in the assessment, meaning that outside of the group of the active participants some people were reading the wiki pages of the assessment, but not contributing to the content of the pages.

The web pages of the two assessments were downloaded 9403 times in total between June 2011 and June 2012. This made the biofuel assessments one of the most popular assessments in Opasnet. A download was counted every time someone loaded a page for reading, and also when someone saved contributions on a page. Most of the downloads were made by the coordinators and the summer trainees, as implied by the 5673 downloads already during the active work period in June - August 2011. There are no detailed statistics about who was actually downloading the pages, but time-wise statistics imply that there was also a lot of activity from outside the contributor group during the commenting period in November - February (2422 downloads). The number was clearly lower earlier, during the updating period (730 downloads) and after the commenting period (578 downloads).

The number of downloads suggest that there was clear activity related to the assessments, and the information produced was indeed absorbed by a fair group of interested people. On the other hand, biofuels are a hot topic that is of interest to a much larger group than what was reached. In a sense, the potential of informing people was larger than what actually was realised, probably because marketing of the results was not set as a high priority and the related activities were modest.

The assessments were performed in the Opasnet website and all the information in the assessment pages was available not only to all active participants, but also to anyone interested throughout the assessment. All contributions were added on the assessment pages even if they had been provided via other means of communication than the Opasnet wiki. However, along the course of assessments it became apparent that the users did not openly reveal all the information they had regarding the assessed topics nor their interests, which had effects both on the process and the output of the assessments. It is also difficult to estimate how much relevant knowledge possessed by the invited persons may have been left out of the assessments due to unwillingness to participate, technical difficulties of contribution and other reasons.

In principle it was possible to follow and contribute to the progress of the assessments continuously via the assessment pages in the Opasnet workspace. The assessments developed rapidly between June and August 2011 as a result of the work by the summer trainees, and at a slower rate until February 2012 mainly as a result of the work by the assessment coordinators. The stakeholders were contacted only after some months of assessment work, when the assessments were already in quite comprehensive form, and most of the stakeholder contributions were given in reaction to this request. The contributions of the users were mostly given via meetings or discussions between them and the assessment coordinators regarding initiation, intermediate checkpoint and delivery of assessment results. Only a few spontaneous contributions were made by the stakeholders and users outside these somewhat formal contribution periods.

All participants, and in fact anyone, were allowed to comment and contribute to all parts of the assessment, except for the assessment questions that were discussed between the users and coordinators. All contributions were compiled in Opasnet and integrated in the assessment. Stakeholder comments were listed on corresponding assessment pages in the form of formal argumentation (see http://en.opasnet.org/w/Talk:Biofuel_assessments for examples) and relevant page contents and conclusions were updated accordingly when needed. Because most stakeholder participants were unfamiliar with working with wikis, many contributions were provided by more conventional means, e.g. by e-mail or through discussion, and added to corresponding Opasnet pages by the assessment coordinators. This probably did not, however, limit the contents of the contributions or which aspects they addressed.

Altogether, all types of contributions were allowed to influence the assessment content. However, one stakeholder representative expressed her slight dissatisfaction in the evaluation about the fact that her point regarding use of fish waste in biogas production did not result in updating of the assessment scope, although the point was agreed to be relevant. The discussion was, however, included in the assessment.

At this point it is not easy to estimate what impact that the assessment, and all contributions to it, has on the decisions and actions of the user. However, the assessments did not link directly to any on-going decision process. Also the evaluation comments indicate that the assessments confirmed rather than changed the understanding about the assessed topics among the users. The influence on practical decisions and actions most likely remained at best moderate.

Quality of content

The high score of evaluation of quality of content indicates that assessment outputs were considered relatively good by the participants. Particularly, the inclusion of various aspects of the production chain in the assessment was appreciated. In addition, the comments provided by stakeholders had only little impact on the output of the assessment as most of the provided information was already found in the assessments, which can be seen as an indication of their comprehensiveness. However, the assessment questions were originally formulated as too vague and thereby difficult to be answered accurately. The questions were clarified during the assessment, but eventually also many issues that were out of the scope of the final questions were addressed. Also the reliability of the information produced was criticized by some of the participants due to the information sources used, as not all the data of accurate, up-to-date information was available.

Applicability

The applicability in general was evaluated as relatively high by the participants, but some differences between questions can be seen.

As regards relevance, the phrasing of the original research questions was too vague to provide users with the information they were looking for. That was partly because the used approach was new for the users and they did not succeed in expressing their needs very clearly from the beginning. Eventually, after the research questions were specified, both of the assessments succeeded relatively well in fulfilling the users' needs in terms of describing the feasibility of Jatropha and waste oil from fish farming as biofuel feedstock. Also learning about the open assessment methodology was appreciated both by the users and the stakeholders.

Although the assessments were performed in the Opasnet environment and were freely accessible to anyone interested throughout the assessments, the participants did not evaluate them as having reached their target participants very well. This is probably mainly due to the limited active participation in the assessments. However, as mentioned above, the web page statistics indicate that the assessments did reach some people (readers) also outside the group of active participants.

Aspects of usability seemed to split the opinions among participants most clearly. While the users and the stakeholders mostly saw that the assessments had little or no influence on their understanding of the topics, those mostly involved in making the assessment, i.e. summer trainees and coordinators, saw them as very enlightening. This could be interpreted so that the users and the active stakeholders were already very familiar with the topics that were assessed, while the summer trainees and coordinators were not. In addition, some of the stakeholders found the Opasnet workspace difficult to use which limited or even prevented their active participation.

Acceptability of the assessments and the way they were performed was considered high by the participants. The openness of the process was especially appreciated, as was also mentioned above. The numerical evaluation responses indicate that within applicability, particularly acceptability of the assessments was considered good.

Efficiency

The efficiency was considered very high according to the evaluation by the users, but the number of respondents was too low for reliably showing significance in statistical analysis. However, as pointed out by one of the user representatives, efficiency depends on how much active participants can be attracted in assessments. Due to the similarity of the two assessments, they both benefited from each other because there were many variables that were possible to develop simultaenously for both assessments. Even though the assessments are not active anymore, their information remains in the Opasnet workspace, and it can be easily accessed and used by anyone, e.g. in future assessments on related topics.

Effectiveness

Both the numerical questionnaire results and textual feedback indicate moderate success of the assessments in the eyes of the participants. Indeed, when considering the assessments only as processes of answer seeking and information producing, this is probably a correct interpretation. Also it probably gives a fair characterization of the potential of the assessments and the methods they applied in delivering that information to use. However, as was discussed already above in relation to openness, in practice the assessment effectiveness in terms of practical decision support did not achieve much of its potential.

If we consider the effects as changes in the knowledge in the members of the society and the decisions and actions influenced by this knowledge, it is hard to see that these assessments have changed or will change the world much, even if their contents were of good quality and their making efficient and credible. Based on the evaluation, two specific aspects behind the gap between potential and realization can be pointed out. First, the mere possibility for unlimited continuous participation did not result in broad and active collaboration. Second, the active and open involvement of the users is crucial for obtaining meaningful assessment outputs and creating a true linkage between the assessment and use of its results. Both of these aspects are relevant for the development of assessment methods and tools, but particularly for the development of collaborative practices for creation and use of knowledge. It is not enough that the methods and tools allow for openness and deep engagement, if the assessment participants choose to act according to the traditional closed and disengaged models of participation, assessment and decision making (cf. Pohjola & Tuomisto 2011, Pohjola et al. 2011).

The more or less positive evaluation in these example cases of limited participation is not a sufficient proof for the goodness of open assessment method and the usefulness of the Opasnet workspace. Experiences with broader collaboration and deeper engagement are needed. However, neither does the evaluation indicate that the method and the workspace would not be functional. Despite some difficulties, the approach enabled the assessments to fulfill the objectives set for them, and the openness of the process was particularly appreciated. Although some participants faced some difficulties in using the Opasnet workspace, inclusion of all contributions worked well because of the technical assistance given by the coordinators. Also the transparency and the possibility to criticize all the steps of the assessment process make open assessment a method worth consideration. After all, the effectiveness of typical assessments and models is usually not any better (Pohjola et al. 2011, Matthews et al. 2008, Pohjola et al. manuscript). By means of open assessment and Opasnet, all the assessment information at least remains openly accessible for anyone in the internet for free possible further use. This increases the potential for effectiveness.

The open assessment method and Opasnet workspace provided functional means and tools to work openly and transparently and to involve various stakeholders in the assessment process. However, the realization of broad, active and continuous collaboration among plural participants remained far from its potential in the two biofuel assessment cases. This was mainly due to the active involvement of both users and stakeholders remaining relatively low. The number of respondents in evaluation of the influence of participation is too small to provide any credible insight, but the evaluation of acceptability indicates that the potential for openness as well as the explicit and transparent inclusion of all contributions were considered valuable among the participants.

Evaluation approach

Also the applied evaluation approach, based on the dimensions of openness and properties of good assessment frameworks, was shown to be usable for evaluating assessment effectiveness. Despite the recognized limitation that it emphasizes the potential for effectiveness rather than actual changes influenced by assessments, it still provides a more comprehensive and meaningful characterization of the aspects that contribute to assessment effectiveness (cf. Pohjola et al. 2011, Matthews et al. 2008, Pohjola et al. manuscript). It also helps to illuminate the strengths, weaknesses and points of improvement, and provides support to conveying the assessment information to its intended use. However, this evaluation exercise was done only after the assessments were completed, and the capabilities of this approach to aid design and execution of assessments was not sufficiently tested here.

Conclusions

The possibility to participate and to influence the decision-making processes regarding environmental and health issues is an important matter for stakeholders and the public in general. In this paper we presented two environmental open assessment cases, where openness was implemented as a principal characteristic of the procedure. Based on the evaluation it can be concluded that:

  1. Open assessment method and the Opasnet workspace are feasible means for performing an open assessment and providing decision support, at least regarding topics such as those considered in the two case studies. Experiences on broader and more active participation are needed to guide their further development.
  2. Participants in the two case studies considered that they had good possibilities to contribute to the assessments. In fact, open assessment and the Opasnet workspace provided all assessment participants more means for contributing to assessments than they were either willing, capable, or ready to make use of. Contradictingly this may have prevented some participants from contributing.
  3. The assessment cases did not have a great influence on the users and other participants, except for the learning about the assessed topics among the summer trainees and coordinators, who were the people mostly involved in making the assessment. This was mostly due to a weak link between the assessment and its use, in other words weak involvement of the users. More active engagement of the users would have been needed to realize the potential of the assessments in terms of effectively supporting their practical decision making.
  4. The evaluation approach applied in evaluating the two assessment cases was shown to be feasible for evaluating effectiveness of assessments and decision support in general. However, the small number of responses to the evaluation questionnaires did not allow for proper application of statistical analysis within the evaluation framework.

On a general level it can be said that functional methods and tools allowing for openness and effectiveness in assessment already exist. It seems that an open approach is also much appreciated. The currently common practices and attitudes adapted for closed and disengaged processes, however, still seem to limit their effective application in practical science-based decision support. In addition, room still exists for improving the user-friendliness of Opasnet and other tools for collaborative assessment.

Acknowledgements

The authors would like to thank Neste Oil Corporation for initiating and participating in the biofuel assessments discussed here. In addition, big thanks to the summer trainees at THL (Minttu Hämäläinen, Pauli Ordén, Tiia Sorjonen, Jaakko Örmälä, Matleena Tuomisto, Johannes Kröger, and Elina Hirvonen) who did most of the work in performing the assessments. Thank you also to the anonymous reviewers for their constructive comments.

References

Cashmore, M. 2004. The role of science in environmental impact assessment: process and procedure versus purpose in the development of theory. Environmental Impact Assessment Review 24:403-426.

Cassman, K.G. & Liska, A.J. 2007. Food and fuel for all: realistic or foolish? Biofuels, Bioproducts ad Biorefining1:18-23.

Crutzen, P.J., Mosier, A.R., Smith, K.A., Winiwarter, W. 2007. N2O release from agro-biofuel production negates global warming reduction by replacing fossil fuels. Atmos. Chem. Phys. Discuss. 7:11191-11205.

Doelle, M. & Sinclair A.J. 2006. Time for a new approach to public participation in EA: Promoting cooperation and consensus for sustainability. Environmental Impact Assessment Review 26:185-205.

European Union. 2009. Directive of the European Parleament and of the Council on the promotion of the use of energy from renewable sources amending and subsequently repealing Directives 2001/77/EC and 2003/30/EC. European Union, Brussels, Belgium, http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2009:140:0016:0062:EN:PDF

Fargione, J., Hill, J., Tilman, D., Polasky, S., Hawthorne, P. 2008. Land clearing and the biofuel carbon debt. Science 319:1235-1238.

Fiorino, D.J. 1990. Citizen participation and environmental risk: A survey of institutional mechanisms. Science, Technology and Human Values 15:226-243.

Giampietro, M., and Mayumi, K. 2009. The biofuel delusion: The fallacy of large scale agro-biofuels production. Earthscan, London.

Gomiero, T., Paoletti, M.G., Pimentel, D., 2010. Biofuels: Ethics and concern for the limits of human appropriation of ecosystem services. Journal of Agriculture and Environmental Ethics 23(5): 403-434 MacKay, D.J.C. 2009.

Hokkanen, P. & Kojo, M. 2003. How environmental impact assessment influences decision-making [in Finnish]. Ympäristövaikutusten arviointimenettelyn vaikutus päätöksentekoon. Suomen ympäristö 612. Ympäristöministeriö. Helsinki: Edita Prima Oy.

Jakeman, A.J., Voinov, A.A., Rizzoli, A.E., Chen, S.H. (Eds.) 1998. Environmental Modelling, Software and Decision Support. Elsevier, Amsterdam.

Jay, S., Jones, C., Slinn, P. & Wood, C. 2007. Environmental impact assessment: Retrospect and prospect. Environmental Impact Assessment Review27:287-300.

Mathews, J. 2008. Biofuels, climate change and industrial development: can the tropical South build 2000 biorefineries in the next decade? Biofuels, Bioproduts and Biorefining 2(2):103-125.

Matthews, K.B., Rivington, M., Blackstock, K.L., McCrum, G., Buchan, K. & Miller, D.G. 2011. Raising the bar? The challenges of evaluating the outcomes of environmental modelling and software. Environmental Modelling & Software 26:247-257.

Myers, R.A. & Worm, B. 2003. Rapid worldwide depletion of predatory fish communities. Nature 15;423(6937):280-3.

Pimentel, D., and Patzek, T. 2005. Ethanol production using corn, switchgrass, and wood: biodiesel production using soybean and sunflower. Natural Resources Research 14(1):65-76.

Pohjola, M.V., Pohjola, P., Paavola, S., Bauters, M. & Tuomisto, J.T.2011. Pragmatic Knowledge Services. Journal of Universal Computer Science 17:472-497. http://dx.doi.org/10.3217/jucs-017-03-0472

Pohjola, M.V., Tainio, M., Pohjola P. & Tuomisto, J.T. Process, output or outcomes? Perspectives to model and assessment performance. Manuscript.

Pohjola, M.V. & Tuomisto J.T. 2011. Openness in participation, assessment, and policy making upon issues of environment and environmental health: a review of literature and recent project results. Environmental Health 10:58. http://dx.doi.org/10.1186/1476-069X-10-58

Pohjola, M.V., Leino, O., Kollanus, V., Tuomisto, J.T., Gunnlaugsdόttir, H., Holm, F., Kalogeras, N., Luteijn, J.M., Magnusson, S.H., Odekerken, G., Tijhuis, M.J., Ueland, Ø., White, B.C. & Verhagen, H. 2012. State of the art in benefit–risk analysis: environmental health. Food Chem. Toxicol. 50:40-55. http://dx.doi.org/10.1016/j.fct.2011.06.004

Reed, M.S. 2008. Stakeholder participation for environmental management: A literature review. Biological Conservation 141:2417-2431.

Ryan, L., Convery, F. & Ferreira S. 2005. Stimulating the use of biofuels in the European Union: Implications for climate change policy. Energy Policy 34(17):3184-3194.

Tijhuis, M.J., Pohjola, M.V., Gunnlaugsdóttir, H., Kalogeras, N., Leino, O., Luteijn, J.M., Magnússon, S.H., Odekerken, G., Poto, M., Tuomisto, J.T., Ueland, Ø., White, B.C., Holm, F., Verhagen, H., 2012. Looking beyond Borders: Integrating best practices in benefit-risk analysis into the field of food and nutrition. Food Chem. Toxicol. 2012, 50:77-93. http://dx.doi.org/10.1016/j.fct.2011.11.044

Tilman et al., 2009. Beneficial Biofuels -The Food, Energy, and Environment Trilemma. Science 325: 270-271. ⇤--#: . Add complete list of authors. --Mikko Pohjola 16:01, 4 December 2012 (EET) (type: truth; paradigms: science: attack)

Tuomisto, J.T. & Pohjola, M.V. (Eds.) 2007. Open Risk Assessment – a new way of providing information for decision-making. Publications of the National Public Health Institute B18/2007, KTL – National Public Health Institute, Kuopio.

van den Hove, S. 2000. Participatory approaches to environmental policy-making: the European Commission Climate Policy Process as a case study. Ecological Economics 33:457–472.

Zopounidis , C., Pardalos, P.M. (Eds): Handbook of Multicriteria Analysis (Applied Optimization). Springer 2010, ISBN 978-3540928270.

See also

Related files

Vilma Sandstrom, Jouni T. Tuomisto, Sami Majaniemi, Teemu Rintala, & Mikko V. Pohjola (2014): Evaluating effectiveness of open assessments on alternative biofuel sources. Sustainability: Science, Practice, & Policy 10 (2): 1-12. heande:File:1207-032.sandstrom.pdf

Retrieved from "https://en.opasnet.org/index.php?title=Evaluating_effectiveness_of_open_assessments_on_alternative_biofuel_sources&oldid=42714"