Peer rating
| Moderator:Jouni (see all) |
|
|
| Upload data
|
Peer rating is a method to give merit ratings or onors to a page in Opasnet. Peer rating has a scale from 0 (very poor) to 100 (outstanding). Previously this number was converted to onors using a specific conversion function, but later it was understood that it is not clear what the function should be. The rating has two parts: scientific quality, and usefulness. Both properties are opinions of the person who makes the rating. Both evaluate the actual content of the page at the time of evaluation. Note that the average rating shown on the rating bar may be largely based on the historical quality of the page. Methods to compensate for page improvements is under development.
Practical hints:
- A person needs to login to be able to make peer ratings.
- Internet Explorer contains deliberate bugs (i.e., deviations from the standard html; the bugs are installed by Microsoft) that may cause trouble using and viewing the rating bar. Improvements are under way to overcome the problem, but in the meantime, we recommend that you complain directly to Microsoft.
Question
What is such a method to give respect or merit to contributions (Opasnet pages) such that the method
- reflects the true social respect on an absolute scale,
- can be learned and used easily.
Answer
A previously used onor rating table.
A special table will be developed to rank pages based on their scientific quality and usefulness. Then the users can compare their estimates to other people's estimates of other pages.
The total onors earned by an individual i are calculated in the following way:
Oi = Σp E(Op) fi,p,
where Oi is the amount of onors earned by individual i, E(Op) is the expectation of the amount of onors given to a particular page p, and fi,p if the fraction of merit that the contributor i earns for his/her work on page p.
Onors can be exchanged to money, but in a non-linear manner. There is a threshold for exchange, and onors below the threshold are not exchanged. The actual level of threshold is determined by the members of the society. The amount of exchangeable onors is calculated in the following way:
O(E)i = Σp Op fi,p (if Op fi,p >= T then 1 else 0),
where T is the threshold. A major reason for this non-linearity is that anyone can do a large number of low-onor edits without specific skills. With a linear exchange rate, most of the money would be exchanged to such low-value contributions. It is more motivating and cost-effective to give higher prizes for hard and skilled work and none for others. This motivates people (who want to do this for living) to work especially on high-value things.
Rationale
Rating should be done with several, partially overlapping methods:
- You can get 0-30 points based on number of views and number of contributions, but not more. The marginal impact of these goes asymptotically to zero.
- Based on peer rating, you can get additional points.
- Based on VOI analysis, you can also get additional points.
- Based on peer review, you can get points.
All of these are used simultaneously, and a rating is always given based on whatever information is available.
←--#: . Functions and calculation rules should be developed to operationalise this. --Jouni 11:40, 4 August 2011 (EEST) (type: truth; paradigms: science: defence)
The rating bar functionality only accepts integers between 0 and 100. Therefore, it is practical to make the first attempt of peer rating reflect this interval.
| Fact discussion: . |
|---|
| Opening statement: Negative respect should also be possible with peer rating.
Closing statement: Not accepted. Negative respect must be given through peer review. (Resolved, i.e., a closing statement has been found and updated to the main page.) |
| Argumentation: |