ORA Workshop 2008 Comments

On this page you can place your comments and all the other things you would like to share (no nude) related to the Open Risk Assessment Workshop 2008.

Comments

Hei Kaikki, I realy enjoyed the workshop, in particular the snowboarding and the sauna. My special tanks to Mikko and Juoni for organising this interesting week. See you hopefully next year. --Sjuurd II 14:37, 22 February 2008 (EET)

Hi, this week was one of my best weeks in my life. Thanks especially to Mikko and Jouni! --Alexandra Kuhn 12:07, 24 February 2008 (EET)

Hejssan! Previous week opened my eyes with Pyrkilo -method. Thank you Mikko and Jouni!--Pauliina 13:00, 25 February 2008 (EET)

The workshop was great - just the right combination of lectures, group work, and physical activity! Really looking forward to applying what we learned in HiWATE. Thanks Mikko and Jouni for all the organising, and the great socials :) --Ninais 00:32, 26 February 2008 (EET)

Hi :-) I did enjoy very much that week spent in Kuopio … so viva ORA! Mikko and Juoni, many thanks!!! Luba

Photographic evidence

James' photos

A collection of my efforts to visually record the proceedings can be viewed at this link. Apologies to those of you whose faces I didn't manage to capture in time! If nothing else, this will be a reason for you to return next year!!

Kingas' photos
A collection of my efforts to visually record the proceedings can be viewed at this link.

Workshop report

Overview

The main idea behind organizing the workshop was to disseminate information about the developments on Open Assessment method to people who are interested in new approaches to risk, impact, integrated and whatever assessments. We wanted to increase people’s knowledge about Open Assessment and explain it to them in a way that it could use by the participants in their practical research. In addition to this educational aim we also wanted to test our method, see the reactions of people being told about Open Assessment and receive their feedback in terms of comments, questions and suggestions, and consider these in developing the method further. Finally, we also wanted to bring together a group of nice people and give them an opportunity to get to know each other and build new collegial relationships.

From the point of view of organizers and Open Assessment method developers the workshop provided again a great push forward. Preparing the lectures and group work forced us to try to compile all the different aspects of Open Assessment into one complete framework and to try to describe it all in a comprehensible way. The questions and comments during the workshop and the feedback received after the workshop clearly showed where this had been successful and where not. Also some completely new methodological findings were made during the workshop week, which was very interesting and welcome to the organizers.

Generally speaking the feeling is that we succeeded relatively well in delivering what we tried, although plenty of space for improvement still remained. The intention was to lecture the overview of Open Assessment and a collection of some important aspects of it in detail as well as to provide a possibility to try some of these out in practice in the group work. This worked out quite well as planned, but it was not necessarily exactly according to the expectations of all participants. Also the choice of which aspects of the method were chosen to be emphasized in the lectures and group work may not have been quite optimal in the eyes of all participants. Some participants also seemed to find it frustrating that the method appeared to be still developing although it was being taught and applied already.

Overall the workshop was a great experience and it is probably fair to say that it was pretty much of a success. Most of the participants perceived the workshop as interesting and useful, at least partially and to some extent, although there was some variation in the perceptions among the participants. Various kinds of points of improvement were identified by different participants and they will all be seriously considered and learnt from. The group of participants was a relatively heterogeneous group with varying levels of prior knowledge about risk assessment and in particular Open (Risk) Assessment making it a difficult task to design the workshop suitable for everyone. On the other hand, probably not enough effort was spent in trying to identify the participants’ needs and expectations and abilities to digest information about Open Assessment in advance. Instead, the design was targeted according to an implicit assumption of a relatively well informed and enthusiastic participant, which probably did apply to some, but not all, participants. Despite some (very welcome) critique, most of the participants seemed to have liked their stay overall and obviously (hopefully anyway) no-one considered his/her visit far up north a complete waste of time and money. Bringing together a group of nice and intelligent people most often produces something good, and this was the case also in this workshop.

Arrangements

We knew that the workshop will be relatively demanding, so we wanted to make the lives of participants comfortable and enjoyable so that they could fully concentrate in the lectures and group work during the workshop week. The social activities also had an important networking aspect in addition. The general arrangements were mainly carried out successfully and also the participants appeared to be happy about them. The facility, accommodation, travel etc. arrangements were conducted with no big problems and no big complaints concerning these were received from the participants. In particular the social activities were generally liked very much! However, some important points of improvement in general arrangements can be easily identified.

First of all, the background and preparation material could and should have been more comprehensive, covering also other issues relevant in assessments than only those specifically related to Open Assessment. This also relates to the above-mentioned issue of identifying and addressing participants’ needs properly. This will hopefully be improved significantly in the near future if the Intarese guidebook development proceeds as expected.

Secondly, there were a lot of problems among the participants with accessing the background material (the guidebook) before and even during the workshop. This seemed to cause quite a lot of annoyance in some of the participants and clearly was not the desired state of affairs for the organizers either. The problems could have and should have been completely avoided with proper preparation and testing well in advance of the workshop, but not enough effort was targeted to this.

Thirdly, there were an annoying amount of technical difficulties related to trying to air the lectures to remote participants and to record the lectures through Skype. This did not really affect the participants in Kuopio, but we must give our apologies to all those (two) people who bravely tried to attend the lectures via Skype, but what we could not deliver in the end. Again, it can be said that most of these problems could have been avoided with proper preparation and due attention to these issues well in advance.

Fourthly, although this goes a bit beyond the scope of workshop arrangements, the way the usernames and passwords for the KTL guest network internet connection were provided caused quite a bit of unnecessary work and annoyance to both participants and organizers. Unfortunately we can not address this issue directly ourselves, but only pass on the regards from us and all the workshop participants to the IT services of KTL.

All in all, the issues that were addressed properly worked very well. On the other hand, there were also issues that we did not have the time and resources to address enough, and that often resulted in inconvenient, but avoidable, difficulties. Perhaps a slightly bigger group of active organizers and better distribution of responsibilities among them would be good in arranging a workshop even of this size. In practice this could mean e.g. having separate people responsible for technical issues and other practical arrangements. In addition to making the arrangements more smooth this would also help to reduce the organizational burden of the ones responsible for the actual content of the workshop, and probably result in improved lectures, group work and background material etc.

Lectures

The intention regarding the lectures was to provide a relatively comprehensive overview to Open Assessment and to go into more details on some certain issues that were seen of highest priority. Another rationale behind choosing the specific lecture topics was to provide adequate support for carrying out the group work exercises. In the end, the focus of the lectures turned out to emphasize especially the information structure of assessment products and the interaction of assessment with its context (e.g. purpose, performance etc.) and leaving the processes of making an assessment to less consideration. This was an intentional choice.

The lectures were generally considered of being high quality, although some of the lecture contents were somewhat disputed. Some more practical examples and a less theoretical approach was suggested and recommended by many. Although preparing the lectures sometimes took place only just before their presentation, they did remain reasonably well organized and consistent. However, it would have probably been beneficial to have the lecture slides or at least relatively well developed drafts distributed among the participants already well in advance. in addition to providing the participants better possibilities to prepare for the lectures in advance, this would have probably also had a strong positive impact also on fine-tuning of the presentations in terms of e.g. well prepared and practical examples explaining the sometimes complicated theoretical issues.

Particular lectures were valued quite differently by different participants. The introductory lectures 1 and 2 were not commented very much except for the fact that the introduction should have also covered more of the general issues of current assessment approaches, not only the Open Assessment specific issues. Lectures 3 and 4 on information structure were most often seen as valuable and useful. Lecture 5 on argumentation was perceived both as very useful and interesting by some, but also unnecessary by some. Lecture 6, focusing on relatively new ideas on participation and quality control was commented by some as being a bit too underdeveloped and confusing to be included in the workshop content yet. Lectures 7 and 8 on evaluating assessment performance and assessing uncertainty were regarded as very interesting and important, but at the same time very difficult.

Group work

The group work exercise was chosen and designed to give the participants possibilities of trying out some of the important issues related to Open Assessment in practice. These were in particular the information structure (assessment & variables, attributes etc.), contributing to assessments and evaluation of assessment performance. The group work was conducted in wiki (Heande website), since it readily provides suitable functionalities for these purposes.

The group work was also relatively well received, although quite varying views regarding what was useful and what not were presented in the feedback. Obviously more emphasis should have been put on explaining the group work and its goals because, based on the feedback, there seemed to be quite different perceptions of what was tried to do in the group work and consequently how well was it achieved. This issue can be identified both on the level of whole group work assignment as well as individual exercises.

Again, the exercises that were most often found useful and interesting were the ones that concentrated on applying the information structure in practice, i.e. mainly the exercises 2, 3 and 4. This is probably because those participants who are already trying to apply the OA principles in their research could easily see the benefit of those exercises for their real work. On the other hand, also a completely inverse view was presented by one participant. The last exercise about evaluating assessments was seen by some participants as something completely new and therefore very interesting, but the issue itself was given too little emphasis within the workshop and it unfortunately remained unclear to most of the participants.

Summary

The workshop went well, but plenty of space for improvement still remains. The workshop was also at least somewhat useful for most of the participants and very useful the method developers. A bigger group of organizers would probably be helpful in the future to keep the workload of every individual reasonable and to guarantee that all important issues are given the emphasis they require. The interaction between the organizers and participants before the workshop could be improved in order to a) target the participants’ needs better and b) provide a more consistent prior knowledge base to the participants. Now after the workshop and having received the feedback, it seems that it would be so easy to improve from what we managed to deliver. However, this only proves that the workshop was a very good learning experience also to the organizers. Looks like there are plenty of good reasons to continue the tradition of Kuopio Workshops next year, and it seems reasonable to expect that next it year it will be a much better event again.

Very big thanks to everyone who participated in the workshop, and everyone who took part in organizing it!

Mikko Pohjola 10:52, 6 March 2008 (EET)