Peer review: Difference between revisions

From Opasnet
Jump to navigation Jump to search
mNo edit summary
(full text moved to Peer review method. This was shortened to an encyclopedia article.)
Line 2: Line 2:
[[Category:Quality control]]
[[Category:Quality control]]
[[Category:Glossary term]]
[[Category:Glossary term]]
{{method}}
{{encyclopedia}}
This page is about peer review in [[open assessment]]. For other uses, see the [[:en:Peer review|Peer review]] page in Wikipedia.
This page is about peer review in [[open assessment]]. For a full description, see the [[Peer review method]]. For other uses, see the [[:en:Peer review|Peer review]] page in Wikipedia.


<section begin=glossary />
<section begin=glossary />
:'''Peer review''' in [[open assessment]] is a [[method]] for evaluating [[uncertainty|uncertainties]] that are not explicitly captured in the [[definition]] of an object (typically an [[assessment]] or a [[variable]]). Technically, it is a [[discussion]] on the Talk page of the object. In the case of a variable, it has the following [[statement]]:
:'''Peer review''' in [[open assessment]] is a [[method]] for evaluating the scientific quality of the [[definition]] of a [[variable]] or an [[assessment]], or the generalisability of the scope of a [[study]]. Technically, it is a [[discussion]] on the Talk page of the object. In the case of a variable or an assessment, the discussion has the following [[statement]]:
:: "The definition of this object is based on the state-of-the-art scientific knowledge and methods. The data used is representative and unbiased. The causalities are described in a well-founded way. The formula correctly describes how the result can be calculated based on the data and causalities. Overall, the information in the definition reflects the current scientific understanding and is unlikely to substantially change because of some existing information that is omitted."
:: "The definition of this object is based on the state-of-the-art scientific knowledge and methods. The data used is representative and unbiased. The causalities are described in a well-founded way. The formula correctly describes how the result can be calculated based on the data and causalities. Overall, the information in the definition reflects the current scientific understanding and is unlikely to substantially change because of some existing information that is omitted."
:In the case of a variable, the definition (the quality of content of the data, causalities, and formula attributes) is evaluated against the scope, which is fixed. In the case of a study, the scope is evaluated against the definition (i.e. the scientific work performed), which is fixed. Thus, the question is about how much it is possible to generalise from the results of a study.
<section end=glossary />
<section end=glossary />


==Scope==
====Peer review of the definition (for assessments and variables)====


What is such a method for gaining acceptance to an [[object]] from the scientific community that fulfils the following criteria?
:'''Peer review''' in [[open assessment]] is a [[method]] for evaluating the scientific quality of the [[definition]] of a [[variable]] or an [[assessment]], or the generalisability of the scope of a [[study]]. Technically, it is a [[discussion]] on the Talk page of the object and has the following [[statement]]:
* It is based on an evaluation of the object by peer researchers.
: "The definition of this object is based on the state-of-the-art scientific knowledge and methods. The data used is representative and unbiased. The causalities are described in a well-founded way. The formula correctly describes how the result can be calculated based on the data and causalities. Overall, the information in the definition reflects the current scientific understanding and is unlikely to substantially change because of some existing information that is omitted."
* It is not in conflict with [[open assessment]].
* It evaluates the relationship of the [[scope]] and the [[definition]]. The question is whether the relationship is well-founded according to the current scientific information.
 
==Definition==
 
===Input===
 
The input is an object to-be-evaluated.
 
===Output===
 
The output is a statement about the relationship of the scope and the definition in the light of the current scientific information.  
 
===Rationale===
 
In the case of a variable, the definition (the quality of content of the data, causalities, and formula attributes) is evaluated against the scope, which is fixed. In the case of a nugget, the scope is evaluated against the definition (i.e. the scientific work performed), which is fixed. Thus, the question is about how much it is possible to generalise from the results of a study.
 
==Result==
 
===Procedure===
 
'''Peer review of the definition


'''Peer review''' in [[open assessment]] is a [[method]] for evaluating [[uncertainty|uncertainties]] that are not explicitly captured in the [[definition]] of an object (typically an [[assessment]] or a [[variable]]). Technically, it is a [[discussion]] on the Talk page of the object and has the following [[statement]]:
====Peer review of the scope (for a study)====
: "The definition of this object is based on the state-of-the-art scientific knowledge and methods. The data used is representative and unbiased. The causalities are described in a well-founded way. The formula correctly describes how the result can be calculated based on the data and causalities. Overall, the information in the definition reflects the current scientific understanding and is unlikely to substantially change because of some existing information that is omitted."


The following classification can be used for each attribute:
[[Study]] is a special kind of object in the sense that the definition typically describes a particular study that has been performed. Therefore, it is not possible to evaluate (in the sense of attempting to improve) the definition as such, because what was done was done. Instead, the interesting question is about the generalisability of the results.  
* The attribute description is according to the state-of-the-art.
* The attribute description has minor deficiencies.
* The attribute description is unreliable because of its major deficiencies.
* Cannot be evaluated.


Thus, the peer review of a study aims to answer this question: '''"To what questions do the study actually answer reliably, based on the current scientific understanding?"


'''Who can and should do a peer review?
====Who can and should do a peer review?====


Basically, Opasnet is applying an open peer review process in its widest sense. It means that anyone can make a peer review about anything. However, a peer review is worthless unless the readers believe that the reviewer actually is a peer, which means a person who has enough relevant expertise, usually a fellow researcher. Therefore, the following guidance is advised:
Basically, Opasnet is applying an open peer review process in its widest sense. It means that anyone can make a peer review about anything. However, a peer review is worthless unless the readers believe that the reviewer actually is a peer, which means a person who has enough relevant expertise, usually a fellow researcher. Therefore, the following guidance is advised:
Line 54: Line 30:
* You should not be a major contributor of the page you review, i.e. you should not be one of those who have brought a substantive amount of scientific material to the page. Technical and linguistic edits can be done without limitation.
* You should not be a major contributor of the page you review, i.e. you should not be one of those who have brought a substantive amount of scientific material to the page. Technical and linguistic edits can be done without limitation.
** The roles of each contributor are clarified in the [[Acknowledgements]] of the page.
** The roles of each contributor are clarified in the [[Acknowledgements]] of the page.
====Alternative approaches to peer review====
Typically, a peer review means an evaluation of the [[Definition]] of a page. Whether the result is actually truthful is something that a peer cannot usually review. However, in some cases an evaluation is possible, and these are described below. Also, the [[scope]] can be reviewed in some cases. Whether these should be called peer review, is an open question.
'''Peer review of the result based on an external reference
Peer review can be performed for the result, if the peer has an alternative way of deriving the result. Then, it can be used as an external reference for evaluating the discrepancy between the result and the external reference. Of course, the validity of this review is totally dependent on the validity of the external reference. [[Informativeness]] and [[calibration]] can be evaluated against the reference.
In addition, a '''discrepancy test''' can be performed. The aim of this test is to evaluate whether it is credible to believe the result of a [[value of information]] (VOI) analysis related to the variable. The VOI analysis gives low values if the need to improve the model is low. A problem with the VOI analysis is that if the model is too bad, it might also give low VOI estimates, thus falsely implying a good model. Fortunately there are a few indirect methods to evaluate this. One way is to do a peer review of the definition. Another one is to do a discrepancy test. It measures whether the external reference is essentially included in the current result of the variable. If this is the case, it is unlikely that the VOI will be underestimated. the [[:en:Kolmogorov–Smirnov test|Kolmogorov–Smirnov test]] is a relevant discrepancy test.
In the numerical VOI analysis, the result distribution is divided into n equally probable bins. The discrepancy test asks, what the probability is that the result is in a higher (lower) bin than the external reference. If both probabilities are fairly high, it is unlikely that the result is falsely too narrow and biased. What is "fairly high", remains to be determined.
'''Peer review of the scope of a study
[[Study]] is a special kind of object in the sense that the definition typically describes a particular study that has been performed. Therefore, it is not possible to evaluate (in the sense of attempting to improve) the definition as such, because what was done was done. Instead, the interesting question is about the generalisability of the results. If they are not at all generalisable, the study is worthless. Let's take an example: there is a study that is an epidemiological case-control study with a questionnaire from all and and blood measurements of a pollutant from a subset of patients. If the blood measurements were done with an unreliable method, the results cannot be generalised even to the patient that was studied, i.e. we don't learn anything about the patient's pollutant levels even if we know the blood test result. If the blood test is good, we can believe that it reflects the patient's true pollutant level in blood. The next question is whether the patient is representative of his/her group, i.e. whether the result can be generalised to the whole group of cases or controls. In this case, blood was drawn only from a fraction of those who returned the questionnaire, and there is doubt whether the subgroup was somehow a biased sample of the bigger group. If it is biased, we cannot generalise to the group of cases or controls. There is also the question whether the group in the study actually reflects the population of interest. If the controls are drawn from a different population than the cases, it is doubtful whether they can be used as controls to compute odds ratios of the pollutant causing the disease of concern.
Thus, the peer review of a nugget aims to answer this question: "To what question(s) does the nugget actually answer reliably, based on the current scientific understanding?"
===Management===
The [[peer review]] [[discussion]] has the following form:
<big>'''Peer review'''</big>
{{discussion
|Dispute= The definition of this object is based on the state-of-the-art scientific knowledge and methods. The data used is representative and unbiased. The causalities are described in a well-founded way. The formula correctly describes how the result can be calculated based on the data and causalities. Overall, the information in the definition reflects the current scientific understanding and is unlikely to substantially change because of some existing information that is omitted.
|Outcome=
|Argumentation =
{{defend|1|The data used is representative and unbiased.|--[[User:Jouni|Jouni]] 11:37, 16 January 2009 (EET)}}
{{defend_invalid|2|The causalities are described in a well-founded way.|--[[User:Jouni|Jouni]] 23:04, 19 January 2009 (EET)}}
:{{attack|6|Attack these arguments if necessary.|--[[User:Jouni|Jouni]] 23:04, 19 January 2009 (EET)}}
{{defend|3|The formula correctly describes how the result can be calculated based on the data and causalities.|--[[User:Jouni|Jouni]] 23:04, 19 January 2009 (EET)}}
{{attack|5|The issue described in argument 4 is missing.|--[[User:Jouni|Jouni]] 11:37, 16 January 2009 (EET)}}
:{{defend|4|The issue of ...(describe the issue here)... is important and relevant for this object.|--[[User:Jouni|Jouni]] 11:37, 16 January 2009 (EET)}}
}}


==See also==
==See also==

Revision as of 06:26, 20 April 2009


This page is about peer review in open assessment. For a full description, see the Peer review method. For other uses, see the Peer review page in Wikipedia.

<section begin=glossary />

Peer review in open assessment is a method for evaluating the scientific quality of the definition of a variable or an assessment, or the generalisability of the scope of a study. Technically, it is a discussion on the Talk page of the object. In the case of a variable or an assessment, the discussion has the following statement:
"The definition of this object is based on the state-of-the-art scientific knowledge and methods. The data used is representative and unbiased. The causalities are described in a well-founded way. The formula correctly describes how the result can be calculated based on the data and causalities. Overall, the information in the definition reflects the current scientific understanding and is unlikely to substantially change because of some existing information that is omitted."
In the case of a variable, the definition (the quality of content of the data, causalities, and formula attributes) is evaluated against the scope, which is fixed. In the case of a study, the scope is evaluated against the definition (i.e. the scientific work performed), which is fixed. Thus, the question is about how much it is possible to generalise from the results of a study.

<section end=glossary />

Peer review of the definition (for assessments and variables)

Peer review in open assessment is a method for evaluating the scientific quality of the definition of a variable or an assessment, or the generalisability of the scope of a study. Technically, it is a discussion on the Talk page of the object and has the following statement:
"The definition of this object is based on the state-of-the-art scientific knowledge and methods. The data used is representative and unbiased. The causalities are described in a well-founded way. The formula correctly describes how the result can be calculated based on the data and causalities. Overall, the information in the definition reflects the current scientific understanding and is unlikely to substantially change because of some existing information that is omitted."

Peer review of the scope (for a study)

Study is a special kind of object in the sense that the definition typically describes a particular study that has been performed. Therefore, it is not possible to evaluate (in the sense of attempting to improve) the definition as such, because what was done was done. Instead, the interesting question is about the generalisability of the results.

Thus, the peer review of a study aims to answer this question: "To what questions do the study actually answer reliably, based on the current scientific understanding?"

Who can and should do a peer review?

Basically, Opasnet is applying an open peer review process in its widest sense. It means that anyone can make a peer review about anything. However, a peer review is worthless unless the readers believe that the reviewer actually is a peer, which means a person who has enough relevant expertise, usually a fellow researcher. Therefore, the following guidance is advised:

  • If you need the information of a page in your assessment or other work and the page has not been reviewed yet, you should consider reviewing the page yourself before using it. Or, if you don't feel qualified, you should put some effort in finding a person who could review the page. This way, you increase the credibility of your own work, and you also help the Open Assessors' Network to evaluate and improve the contents of Opasnet.
  • You can peer review a page in Opasnet, if you have a credible record of expertise in the area of the page. It is advised that reviewers put enough information about this on their user page (maybe a brief curriculum vitae and a list of publications).
  • You should not be a major contributor of the page you review, i.e. you should not be one of those who have brought a substantive amount of scientific material to the page. Technical and linguistic edits can be done without limitation.
    • The roles of each contributor are clarified in the Acknowledgements of the page.

See also

References

Retrieved from "https://en.opasnet.org/index.php?title=Peer_review&oldid=8845"