Talk:Assessment of the health impacts of H1N1 vaccination: Difference between revisions

From Opasnet
Jump to navigation Jump to search
(draft evaluation of the model/assessment copied here for possible further use)
Line 191: Line 191:


<references/>
<references/>
== Draft evaluation of the assessment ==
''The evaluation below was originally made for the purpose of demonstrating the application of the properties of good assessment framework in an article considering evaluation of model and assessment effectiveness, but in the end was omitted from the manuscript.
One interesting assessment example is the assessment of the health impacts of H1N1 vaccination (Figure 1, http://en.opasnet.org/en-opwiki/index.php?title=Assessment_of_the_health_impacts_of_H1N1_vaccination&oldid=20820) in Finland. The assessment and corresponding model were developed on the decision analysis and risk management (DARM) Master's level course (http://en.opasnet.org/w/darm) during spring term 2011 at the University of Eastern Finland.
[[File:H1N1 vaccination health impact.png|thumb|500px|center|Figure 1. Causal diagram of the Assessment of health impacts of AH1N1 vaccination model.]]
The assessment itself will be discussed in more detail elsewhere (Iso-markku et al., manuscript), here we evaluate the assessment in order to exemplify the application of the Properties of Good Assessment framework (Table 7). On the course the assessment primarily served the purposes of illustrating essential aspects of decision analysis and risk management, as well as an explicating the swine flu (AH1N1 influenza) pandemic and related vaccination campaign in Finland 2009 - 2010 as examples of practical contexts for decision analysis and risk management. Here we focus on its secondary purpose: to evaluate the decision to launch a nationwide vaccination campaign to alleviate the pandemic in Finland. The discourse erupted during autumn 2010 when suspicions regarding a relationship between the AH1N1 vaccine and the sudden increase in prevalence of narcolepsy in Finland were publicized in the media. Soon after that the National Institute for Health and Welfare (THL) in Finland set up a task force to determine whether such a causal relationship exists (THL, 2011a,b).
The setting for evaluating the assessment:
* Time of assessment: spring 2011.
* Assessors: organizers of the DARM course, participants of the course.
* Intended primary user: Ministry of Social Affairs and Health in Finland.
* Intended use: basis for communication about public concerns regarding swine flu vaccines and narcolepsy.
* Evaluation method: qualitative expressions on a 5-point scale from very low to very high.
* Evaluation focus: whole assessment (main message, supported by all other information).
* Basis for evaluation: information provided on the assessment page in Opasnet, complemented with additional information obtained from assessment participants where necessary.
{|{{prettytable}}
|+ '''Table 7. Evaluation of the assessment of the health impacts of H1N1 vaccination
|-----
! Category
! Property
! Characterization
! Explanation
|-----
| rowspan="3"| Quality of content
| Informativeness
| Medium
| The conclusion that vaccinating whole population was a better alternative than no vaccination is well supported by the model results. The conclusion that vaccinating whole population was a better alternative than vaccinating all except 5-19 old is more vaguely supported. The uncertainty of some variables in the model is high.
|-----
| Calibration
| Medium
| Model results and assessment conclusions are in line with the analyses by the European Medical Agency (EMA, 2011) and the National narcolepsy task force in Finland (THL, 2011b). The calibration of some variables in the model, e.g. DALY weight for narcolepsy, may well be questioned as they are based on assumptions rather than data.
|-----
| Coherence
| High
| The question is well addressed and the answer is reasoned with a model that takes account of the most important factors known to have effect on the outcome. Limitations in the comprehensiveness of the model and its parts exist, e.g. in terms of assumptions, but many of them identified and explicated. Value of information analysis indicates high coherence within the assessment.
|-----
| rowspan="4"| Applicability
| Relevance
| Very low (potentially high)
| The assessment addresses an issue that can be considered as underlying much of the discussion regarding the swine flu pandemic and the vaccination campaign. The assessment could thus be claimed to serve a real, existing need and the potential relevance could be considered high. However, in reality there was practically no interaction between the assessors and intended users and there was no demand from the users for the assessors to address the question. Consequently, the actual relevance is very low.
|-----
| Availability
| Low (potentially very high)
| The model was developed and presented on a freely accessible assessment page in Opasnet at a time when many of the issues related to the case were still unresolved and under active public discussion. Practically no technical limits to availability exist. However, awareness about the assessment among intended users remained low despite (or due to only) the minor efforts of informing them by e-mail. Consequently, the relevance is actually low, although potentially very high.
|-----
| Usability
| Medium
| The main message and its basis is presented in a structured way and is relatively easily perceivable even for non-experts. Assumptions and limitations are described and access to the data and calculations is provided and easy. However, obtaining a deeper and detailed understanding of the model requires specific knowledge, detailed scrutiny, and possibly also assistance from the developers. Explication of the intended use and guidance of use are omitted from the assessment (cf. use plans in Vermaas and Houkes, 2006). Due to lack of interaction with users the actual usability in intended use is unknown.
|-----
| Acceptability
| Medium
| The open approach can be considered to have increased acceptability in a situation where authorities were accused for non-warranted withholding of important information. On the other hand the model was developed by non-experts regarding infectious diseases and vaccines. Also the credibility of the organization, THL, that the main developers represented was strongly questioned in public at the time of developing and delivering the model. Due to lack of interaction with users the actual acceptability in intended use is unknown.
|-----
| rowspan="2"| Efficiency
| Intra-assessment efficiency
| High
| The assessment was developed as a side product of the DARM course. The development of the model consumed about 2 person months work, consisting mostly of the efforts of the course assistant, a high school graduate with good mathematical and computer skills, but no prior specific expertise on vaccines or infectious diseases.
|-----
| Inter-assessment efficiency
| High
| The assessment is mostly structured as independent variables that are applicable in other assessments. The assessment also applies some variables that were developed in previous assessments. However, the calculation in the model for the most part was not coded as independently applicable modules.
|-----
| Overall evaluation
| Effectiveness
| Low
| Despite mostly relatively good scores with regard to many properties, the overall effectiveness remains low because the intended use did not take place in reality. The potential for outcome effectiveness can be seen, but the failure of the delivery, i.e. lack of interaction between the assessment and its use, prevents it from becoming realized. The realized impacts are mostly process effects, increasing the knowledge among the participants of the assessment. As many of the participants work in roles that are relevant to the interests of the intended user, the Ministry of Social Affairs and Health, it can be assumed that some of that knowledge will eventually trickle to its intended use, but indirectly and with delay.
|}
The example assessment can be considered as somewhat typical in the sense that it fails to convey its as such good results into practice. Although the evaluation example above can be considered somewhat superficial, and is made only qualitatively, it highlights some important aspects of assessment and model performance:
# In terms of outcome effectiveness, there is a major difference between the potential of an assessment or model to deliver its intended outcomes and the actual delivery thereof.
# The properties that have been least addressed within the common contemporary approaches to performance, namely relevance and availability, are critical for transforming the potential of an assessment or model to effectiveness.
# The delivery of the assessment or model outputs to their intended use is necessary to take account of in considering assessment and model performance.
# Improving effectiveness of assessments and models is not an issue to be addressed within the communities of assessment and modelling, but requires simultaneous development of the use processes and the capacity of policy making to make use of what assessments and models can deliver.
The major limitations of the assessment indicated by the evaluation according to the properties of good assessment may seem apparent, but they would probably not show up in evaluations applying more conventional approaches. Altogether, the example shows that, despite still lacking explicit methods for its application, the Properties of Good Assessment framework can already be a useful and powerful means for evaluating and managing assessment and model performance.

Revision as of 08:05, 8 May 2012

Darm exercise 4

Tasks:

  • Make pairs and select one group A-E (D and E have Finnish material).
  • Look at page Discussion for theoretical and practical advice. Also page op_fi:Keskustelun jälkijäsentäminen may be useful.
  • Look at the two or three links on your group's page.
  • Read the texts and try to identify arguments about the use of Pandemrix. Note that arguments can be (and most are) indirect arguments.
  • Rewrite the arguments in such a way that they can be understood outside the original context.
  • Place the arguments into a hierarchical tree of attacking and defending arguments pointing toward the main statement about Pandemrix use.
  • You can also make up your own arguments, or if you have extra time, read additional material (see end of the page).
  • When each group has done their own part, there will be a general discussion about all argumentations by the groups. All argumentations will be merged onto this page based on the discussion.
  • Think: How many readers do you need to make this extra effort of collecting, organising and synthesising information and opinions a worthwhile activity of social learning?


The group specific pages can be improved until Monday (4.4.) evening.
Evaluation of the exercise will be based on the contents of the group pages as of Monday evening.
The argumentation by different groups will be compiled on this page and discussed more on the lecture on Thursday 7.4.

Pandemrix should not be used because of narcolepsy risk

How to read discussions

Fact discussion: .
Opening statement: Pandemrix should not be used any more anywhere because its narcolepsy risk is too high.

Closing statement: Not accepted. Pandemrix is still an effective and safe vaccine. However, due to precautionary reasons, other alternatives should be used when available, because the occurrence of narcolepsy is not understood.

(Resolved, i.e., a closing statement has been found and updated to the main page.)

Argumentation:

⇤--J5: . Pandemrix is a safe vaccine and narcolepsy risk is low. --Jouni 18:17, 6 April 2011 (EEST) (type: truth; paradigms: science: attack)



⇤--J5: . Pandemrix is an effective vaccine with strong benefits. --Jouni 18:17, 6 April 2011 (EEST) (type: truth; paradigms: science: attack)

←--B4: . Adjuvanted vaccines commonly provide a stronger immune response than unadjuvanted vaccines and also provide a broader immune response allowing for some potential drift of the influenza virus (Pandemrix is an adjuvanted vaccine). --Oluyemitoyinbo 11:12, 1 April 2011 (EEST) (type: truth; paradigms: science: defence) [4]
←--B5: . A positive effect of the vaccine can be felt from pregnant women, it gives immunization to the mother-to-be and the new born that is not eligible for vaccination until the age of six months. The adverse events reported so far have mainly been symptoms such as fever, nausea, headache, allergic reactions and injection site reactions, confirming the safety profile of the vaccine. The vaccine can even be given to breastfeeding mothers and this will reduce the chance that the infant will get influenza. --Oluyemitoyinbo 11:20, 1 April 2011 (EEST) (type: truth; paradigms: science: defence) [4]
←--E7: . Vaccination is a very good method of preventing swine flu infections. --Sallamari Tynkkynen 10:59, 1 April 2011 (EEST) (type: truth; paradigms: science: defence) [5]

⇤--D3b: . The vaccine may still have been used where no other option was available and upon consideration in individual cases, for instance for people travelling to areas where an epidemic was in progress. --Carmen Gil 11:25, 1 April 2011 (EEST) (type: truth; paradigms: science: attack) [6]

⇤--J1: . Despite risks, Pandemrix is an effective vaccine and has clearly net positive effects in countries where emergency treatment is poorly available for severe swine flu cases. --Jouni 23:05, 31 March 2011 (EEST) (type: truth; paradigms: science: attack)



←--J2: . The reputation of Pandemrix is globally so poor that it is impossible to use it any more. --Jouni 23:05, 31 March 2011 (EEST) (type: truth; paradigms: science: defence)



←--J6: . Pandemrix should not be used due to precautionary reasons and because there are alternatives. --Jouni 18:17, 6 April 2011 (EEST) (type: truth; paradigms: science: defence)



←--B8: . There maybe a conflict of interest in the approval of Pandemrix for swine flu vaccination in Finland --Oluyemitoyinbo 12:50, 4 April 2011 (EEST) (type: truth; paradigms: science: defence)



C12: . The interesting question about Pandemrix is perhaps not if it should not be used because of narcolepsy threat, but whether if it was actually needed at all. Russian Federations chief doctor Gennady Onishchenko stated on June 2, 2009 that swine flu was not aggressive enough to cause worldwide pandemic. He noted that the mortality rate of confirmed cases was 1,6% in Mexico and only 0,1% in United States. He also noted that there was 16,000 cases so far when during any flu season some 10,000 people become ill in Moscow alone. --Jpmannikko 19:25, 4 April 2011 (EEST) (type: truth; paradigms: science: relevant branch)

----#: . Good usage of branch functionalities! However, this is also an attack, because it changes the resolution of the statement away from "accepted". --Jouni 06:55, 6 April 2011 (EEST) (type: truth; paradigms: science: comment)

⇤--E6: . The vaccination used last year will most likely protect also against the possible swine flu epidemic of this year, although the virus has changed a bit. --Sallamari Tynkkynen 10:57, 1 April 2011 (EEST) (type: truth; paradigms: science: attack) [5]

⇤--J6: . This may be true but what is the connection to the statement? This is irrelevant. --Jouni 08:15, 2 April 2011 (EEST) (type: truth; paradigms: science: attack)
←--E8: . WHO recommended that H1N1 virus strain be included in the seasonal flu vaccines for the 2009/2010 season, because H1N1 is still in circulation, but behaving like a seasonal flu virus. --Anna Kokkonen 11:10, 1 April 2011 (EEST) (type: truth; paradigms: science: defence) [10]

⇤--E5: . Nursing staff in hospitals should be vaccinated; it is their responsibility as medical professionals. --Sallamari Tynkkynen 10:54, 1 April 2011 (EEST) (type: truth; paradigms: science: attack) [7]

⇤--J7: . This may be true but what is the connection to the statement? Does this actually relate more to E4 than the main statement? --Jouni 08:15, 2 April 2011 (EEST) (type: truth; paradigms: science: attack)

Discussion groups:

See also

Additional material:

References

  1. ↑ 1.0 1.1 1.2 1.3 Rokotusinfo: Swine flu
  2. ↑ YLE: EU agency does not find link between Pandemrix and narcolepsy
  3. ↑ 3.0 3.1 3.2 Rokotusinfo
  4. ↑ 4.0 4.1 4.2 European Centre for Disease Prevention and Control (ECDC): Questions and answers
  5. ↑ 5.0 5.1 THL press release 9 Dec 2010
  6. ↑ 6.0 6.1 6.2 6.3 THL press release 25 Aug 2010
  7. ↑ 7.0 7.1 Helsingin Sanomat: Arkkiatri moittii sikainfluenssarokotteen vastustajia (in Finnish)
  8. ↑ WHO Europe: Pandemrix® vaccine and increased risk of narcolepsy
  9. ↑ WHO Global Advisory Committee on Vaccine Safety: Statement on narcolepsy and vaccination
  10. ↑ 10.0 10.1 European Medicines Agency (EMA): Information page on Swine flu
  11. ↑ PreventDisease.com: Total of 2300 Reports of Adverse Reactions From Pandemrix Vaccine in Sweden
  12. ↑ THL recommends to stop the use of Pandemrix

Draft evaluation of the assessment

The evaluation below was originally made for the purpose of demonstrating the application of the properties of good assessment framework in an article considering evaluation of model and assessment effectiveness, but in the end was omitted from the manuscript.

One interesting assessment example is the assessment of the health impacts of H1N1 vaccination (Figure 1, http://en.opasnet.org/en-opwiki/index.php?title=Assessment_of_the_health_impacts_of_H1N1_vaccination&oldid=20820) in Finland. The assessment and corresponding model were developed on the decision analysis and risk management (DARM) Master's level course (http://en.opasnet.org/w/darm) during spring term 2011 at the University of Eastern Finland.

Figure 1. Causal diagram of the Assessment of health impacts of AH1N1 vaccination model.

The assessment itself will be discussed in more detail elsewhere (Iso-markku et al., manuscript), here we evaluate the assessment in order to exemplify the application of the Properties of Good Assessment framework (Table 7). On the course the assessment primarily served the purposes of illustrating essential aspects of decision analysis and risk management, as well as an explicating the swine flu (AH1N1 influenza) pandemic and related vaccination campaign in Finland 2009 - 2010 as examples of practical contexts for decision analysis and risk management. Here we focus on its secondary purpose: to evaluate the decision to launch a nationwide vaccination campaign to alleviate the pandemic in Finland. The discourse erupted during autumn 2010 when suspicions regarding a relationship between the AH1N1 vaccine and the sudden increase in prevalence of narcolepsy in Finland were publicized in the media. Soon after that the National Institute for Health and Welfare (THL) in Finland set up a task force to determine whether such a causal relationship exists (THL, 2011a,b).

The setting for evaluating the assessment:

  • Time of assessment: spring 2011.
  • Assessors: organizers of the DARM course, participants of the course.
  • Intended primary user: Ministry of Social Affairs and Health in Finland.
  • Intended use: basis for communication about public concerns regarding swine flu vaccines and narcolepsy.
  • Evaluation method: qualitative expressions on a 5-point scale from very low to very high.
  • Evaluation focus: whole assessment (main message, supported by all other information).
  • Basis for evaluation: information provided on the assessment page in Opasnet, complemented with additional information obtained from assessment participants where necessary.
Table 7. Evaluation of the assessment of the health impacts of H1N1 vaccination
Category Property Characterization Explanation
Quality of content Informativeness Medium The conclusion that vaccinating whole population was a better alternative than no vaccination is well supported by the model results. The conclusion that vaccinating whole population was a better alternative than vaccinating all except 5-19 old is more vaguely supported. The uncertainty of some variables in the model is high.
Calibration Medium Model results and assessment conclusions are in line with the analyses by the European Medical Agency (EMA, 2011) and the National narcolepsy task force in Finland (THL, 2011b). The calibration of some variables in the model, e.g. DALY weight for narcolepsy, may well be questioned as they are based on assumptions rather than data.
Coherence High The question is well addressed and the answer is reasoned with a model that takes account of the most important factors known to have effect on the outcome. Limitations in the comprehensiveness of the model and its parts exist, e.g. in terms of assumptions, but many of them identified and explicated. Value of information analysis indicates high coherence within the assessment.
Applicability Relevance Very low (potentially high) The assessment addresses an issue that can be considered as underlying much of the discussion regarding the swine flu pandemic and the vaccination campaign. The assessment could thus be claimed to serve a real, existing need and the potential relevance could be considered high. However, in reality there was practically no interaction between the assessors and intended users and there was no demand from the users for the assessors to address the question. Consequently, the actual relevance is very low.
Availability Low (potentially very high) The model was developed and presented on a freely accessible assessment page in Opasnet at a time when many of the issues related to the case were still unresolved and under active public discussion. Practically no technical limits to availability exist. However, awareness about the assessment among intended users remained low despite (or due to only) the minor efforts of informing them by e-mail. Consequently, the relevance is actually low, although potentially very high.
Usability Medium The main message and its basis is presented in a structured way and is relatively easily perceivable even for non-experts. Assumptions and limitations are described and access to the data and calculations is provided and easy. However, obtaining a deeper and detailed understanding of the model requires specific knowledge, detailed scrutiny, and possibly also assistance from the developers. Explication of the intended use and guidance of use are omitted from the assessment (cf. use plans in Vermaas and Houkes, 2006). Due to lack of interaction with users the actual usability in intended use is unknown.
Acceptability Medium The open approach can be considered to have increased acceptability in a situation where authorities were accused for non-warranted withholding of important information. On the other hand the model was developed by non-experts regarding infectious diseases and vaccines. Also the credibility of the organization, THL, that the main developers represented was strongly questioned in public at the time of developing and delivering the model. Due to lack of interaction with users the actual acceptability in intended use is unknown.
Efficiency Intra-assessment efficiency High The assessment was developed as a side product of the DARM course. The development of the model consumed about 2 person months work, consisting mostly of the efforts of the course assistant, a high school graduate with good mathematical and computer skills, but no prior specific expertise on vaccines or infectious diseases.
Inter-assessment efficiency High The assessment is mostly structured as independent variables that are applicable in other assessments. The assessment also applies some variables that were developed in previous assessments. However, the calculation in the model for the most part was not coded as independently applicable modules.
Overall evaluation Effectiveness Low Despite mostly relatively good scores with regard to many properties, the overall effectiveness remains low because the intended use did not take place in reality. The potential for outcome effectiveness can be seen, but the failure of the delivery, i.e. lack of interaction between the assessment and its use, prevents it from becoming realized. The realized impacts are mostly process effects, increasing the knowledge among the participants of the assessment. As many of the participants work in roles that are relevant to the interests of the intended user, the Ministry of Social Affairs and Health, it can be assumed that some of that knowledge will eventually trickle to its intended use, but indirectly and with delay.

The example assessment can be considered as somewhat typical in the sense that it fails to convey its as such good results into practice. Although the evaluation example above can be considered somewhat superficial, and is made only qualitatively, it highlights some important aspects of assessment and model performance:

  1. In terms of outcome effectiveness, there is a major difference between the potential of an assessment or model to deliver its intended outcomes and the actual delivery thereof.
  2. The properties that have been least addressed within the common contemporary approaches to performance, namely relevance and availability, are critical for transforming the potential of an assessment or model to effectiveness.
  3. The delivery of the assessment or model outputs to their intended use is necessary to take account of in considering assessment and model performance.
  4. Improving effectiveness of assessments and models is not an issue to be addressed within the communities of assessment and modelling, but requires simultaneous development of the use processes and the capacity of policy making to make use of what assessments and models can deliver.

The major limitations of the assessment indicated by the evaluation according to the properties of good assessment may seem apparent, but they would probably not show up in evaluations applying more conventional approaches. Altogether, the example shows that, despite still lacking explicit methods for its application, the Properties of Good Assessment framework can already be a useful and powerful means for evaluating and managing assessment and model performance.

Retrieved from "https://en.opasnet.org/index.php?title=Talk:Assessment_of_the_health_impacts_of_H1N1_vaccination&oldid=24741"