Talk:Open assessment workshop 2009

From Opasnet
Jump to navigation Jump to search

Workshop report and feedback summary

The report is a freely formatted compilation of formal and informal feedback from the workshop participants as well as organizers own perceptions.

Preparation and marketing

Plenty of space for improvement in this aspect. We were well on time in fixing the dates for the workshop week, and deciding on the case theme (climate change). Otherwise the performance of the organizers was poor or at best satisfactory. This resulted in having less than expected participants in the workshop, not reaching as advanced-level application of methods and tools in the exercises as anticipated, and aggregately not quite achieving the expected overall impact in propagating the tidings of joy of open assessment to the global research and assessment community. Things that have been identified as important to be addressed better in the future include at least the following:

  1. Early planning of workshop content and dissemination of plans and marketing material to potential interested participants.
  2. Identification of target groups to whom the workshop is primarily intended and to whom the marketing is focused. Early planning and marketing efforts also allow for adjustments of workshop plans and marketing efforts.
  3. Early preparation and dissemination of workshop materials and advance study materials to potential participants. Advance readings and exercises would help in concentrating on most essential issues during the workshop week and fitting more efficient efficient work time within the already tight schedule.
  4. Distribution of organization and support responsibilities to a broader organizer group. Ensuring enough people more familiar with the methods and tools in the exercises in order to avoid getting stuck with and losing efficient working time to dealing with simple practicalities.

Workshop arrangements: lectures and exercises

Given the setting that we actually ended up having in the workshop, it appears that we succeeded quite well in organizing the lectures and exercises. The lectures were mainly considered good and well organized in the eyes of the participants, although the lecturers were most often very late in preparing the lecture contents. Mostly positive feedback was also received regarding the exercises, despite that we had to change the exercise plans a bit during the week, and we did not really get as deep enough to the actual substance as was anticipated and planned. The points of improvement listed above would also have positive impacts to these shortcomings.

Some things that were mentioned in the feedback or were otherwise noticed:

  1. Some participants thought there was a bit too much repetition of the same things over and over again in both lectures and exercises.
  2. On the other hand it was mostly very much liked that the lectures allowed for discussions on the issues that were triggered by the questions and comments rising from the participants interests.
  3. The workshop days were considered to be long and intensive, and in particular the afternoon exercise sessions turned out quite tiring towards the end. Perhaps a more loose workshop schedule with more scheduled breaks, and also time slots for project meetings and other non-workshop, but work related get-togethers, would help in making the workshop exercise sessions more concentrated and effective.
  4. It was considered as a bit of a problem that many people were coming and going in and out of exercises, causing additional and unnecessary work of explaining basic things to new group members having missed some crucial parts of the particular exercise or previous exercises.
  5. The organizers also planned the exercises too much based on the assumption that there would be active participation of those already relatively familiar with the tools and methods in the exercises, and therefore having enough expertise regarding the basic issues in using the tools and methods in each group. Unfortunately most of these people were too busy with non-workshop activities during the exercise sessions, causing too much of the exercise effort spent on going over the "simple basic things" and not getting into as much detail as intended.
  6. The order of lectures could have been such that the performance lecture, which now was the last lecture, were in the very beginning of the workshop week as it somehow demonstrated the philosophy behind open assessment, and it could have been illuminating to consider the contents of the other lectures in that light.

This year we succeeded better in providing remote participation. Several lectures were followed through Skype, some lectures had even several remote participants. Video from all the lectures was also recorded. It is however still unclear what will be done with the material. Possibly some edited short clips might be used for teaching material later on.

Workshop arrangements: facilities, social activities etc.

All the general arrangements were fine, and the social program was liked very much. Some suggestions were given by the workshop veterans about renewing the social activity provision as some stuff has been done already for a couple of times. Having the workshop during summer time for a change would naturally provide more opportunities for some new kinds of social activities as a part of the workshop program.

Ideas for future workshops

A whole bunch of good ideas and recommendations for future workshops were given and identified (some already mentioned in the text above):

  • performance criteria to be presented and practiced in the beginning of the workshop
  • more brief and concentrated exercise tasks with more time allocated for detailed discussions rising from the exercise work
  • part of the discussions could be organized in association with some social activity in a more informal and relaxed way
  • more practical work of applying the methods and tools
  • stronger involvement of participants' own work into the workshop program (possibly involving some advance preparations)
  • some more time allocated for project and other meetings during the week
  • disaggregating workshop into basic level and advanced level workshops as there are already quite a few people with varying levels of expertise on the methods and tools
    • a tentative plan of having a advanced-level workshop in Lapland next autumn (known as the ruska-workshop) was developed
  • arrangement of lectures so that a very brief introduction to all important topics is given, either in the beginning of the week or beginnings of the days, and then we go into detail on the topics that are of most interest to the participants (demand-driven combination of breadth and detail)

Summary

Overall it can be said that the workshop itself was successful. Tying the workshop and its content into a broader context, and thereby creating the impact and importance, was not so successful. It appears that the method and tools are beginning to be ready enough to be applicable, but a lot more work is still needed to make it happen. However, as was mentioned by many participants, the workshop was (again) a clear improvement to the previous ones. If we just learn the lessons of this workshop, the next ones will be even better. We are looking forward to the great moments with all the enthusiastic people participating in the workshops. Big thanks to all the participants of this years workshop! Again this workshop provided a big input to developing open assessment further.


If you have any additional comments regarding past workshop(s) or ideas regarding future workshops, please feel free to add them here.

Ideas and plans for 2009 workshop

Participant case studies

How to read discussions

Fact discussion: .
Opening statement: Participant case studies should be used.

Closing statement: Accepted.

(A closing statement, when resolved, should be updated to the main page.)

Argumentation:

←--1: . The lectures of the workshop in 2008 explained on the prinicipal idea, structure, methods (and tools) for open assessment (see listing below). The workshop offered the ultimate setting for vivid discussion on the rationale and applicability of open assessment and the suggested methods. In my opinion, the challenge for the workshop in 2009 is to see how current assessment methods&tools (e.g. health impact modelling) can be used for open assessments and what methods&tools are additionally required for successful open assessments. Each of the participants could explain on his or her experience with one or several assessment methods & tools and discuss its suitability for open assessment purposes. I could for example reflect on the issue-framing method and in this context present a tool for policy-relevant indicator-selection. The discussion could focus on how to generate an issue-framing tool that allows for integrating diverse stakes and perspectives. It would also be interesting to hear about experiences with open assessment performances: its successes, experiences difficulties, lessons and challenges. --Eva Kunseler, dd 29Oct08 (type: truth; paradigms: science: defence)

←--2: . This is a great idea. We should go for real-life cases, variables, or methods developed or used by the participants. These would be described beforehand in Opasnet by the participant, and evaluated and criticized by us. Then, there would be practical examples of interesting issues in open assessment already when the workshop starts. These topics could be presented in the workshop, and some of them could be worked on in smaller groups. --Jouni 12:35, 12 November 2008 (EET) (type: truth; paradigms: science: defence)


Possible additional variables for exercises:

  • Future development paths according to IPCC scenarios
  • Global greenhouse gas emissions (UNFCC)

Topics to be covered in workshop

  • Assessment process
    • argumentation - roles and relations free discussion, argumentation and corroborated information content
    • collaboration - assessments as mediated processes of creating shared knowledge
    • scientific method - assessments as testing of networked hypotheses
  • Structure
    • causal diagram - application of causal diagrams in different phases of an assessment
    • Bayes network - Bayes networks of variable results, creation and application
    • information objects - types and roles (similarities/differences) of different information object types
    • information structure - roles and relations of different attributes
    • combining models - assessments and variables as meta-models comprising of sub-models
  • Appraisal
    • appraisal matrices - targeting assessment information according to receivers
    • DALY - making and use of DALYs
    • monetization - conversion and communication of impacts as euros
  • Use process
    • information need - identification of need and its translation to research questions
    • policy advice - addressing real need in assessment design, execution and communication
    • publishing/dissemination - targeting assessment information to secondary (?) receivers (scientists, stakeholders and public)
    • communication - conveyance and convergence of assessment information
  • Performance
    • assessment performance - evaluating/communicating performance as quality of content, applicability and efficiency
    • quality criteria - evaluating goodness of contributions, contributions and group output
    • method development - applying scientific method in developing assessment methodology

IDEA: for upcoming workshops we could divide workshop page using "extra toolbar"

It is easy to create to a top navigation bar which could contain links to workshop sub-pages ie.: (General information, Description, Schedule, Social activities,Accommodation etc.)

This would make pages shorter and navigation easier.

--Juha Villman 14:45, 31 March 2009 (EEST)

Retrieved from "https://en.opasnet.org/index.php?title=Talk:Open_assessment_workshop_2009&oldid=11833"