Value judgement tool

Value judgement machine is a name for a web-based program that collects information about people's value rankings. It is based on a principle that an individual should be able to develop a coherent ranking of issues based on order of preference. Coherent means that A is preferred over B and B is preferred over C, then also A must be preferred over C. There are two alternative ways of performing the evaluation, namely an individual as a decision-maker, and an individual as himself. The former case is more interesting and relevant, because those valuations can be used for ranking outcomes in societal decision-making, while the usability of the latter ranking is not clear. However, both are described for completeness (the former was discovered first).

Basic idea

The idea in both cases is that there is a database of interesting issues that should be ranked for preference. The comparison is performed by individuals using internet interface for this. Issues are sampled (with weighted probabilities, see below) from the issue database two at a time, and the individual is asked to tell which one he prefers more. This is done repeatedly for different issues, until there is an unambiguous ranking for a subset of issues. If the respondent is not coherent, The incoherent answers are pointed out and the respondent is asked to clarify the ranking.

The issues have always a similar form with a few alternative attributes. If an attribute of an issue is changed, it is considered a new issue.

• An issue is about losing something that existed or getting something that did not exist. This includes a reference "nothing happens". (Can it be something that just happens without being possible to describe that as losing or getting?).
• The getting or losing happens to
  1. someone whose identity is unknown at the time of decision,
  2. everyone,
  3. the nature or similar non-personal entity, or
  4. the person who is answering
• The issue happens somewhere
  1. within the family of the respondent,
  2. in the home town of the respondent,
  3. in the home country of the respondent, or
  4. in the world.
• The issue happens
  1. immediately,
  2. this year, or
  3. after a defined number of years, usually 15 years.

The answers to the comparisons are stored in the database together with the respondents username and time of answer. Respondents are not identified, but the username (and password) are required so that a respondent's answers can be linked together, even if they have been given during more than one session. In the following, we'll describe the exact wording and form of questions that we think should be used. However, it is a slow process to go through pairs of issues, and when the respondent is familiar with the form of questions, he can start putting ranking of issues based on a list of issues where the individual issues can be dragged and dropped into a higher or lower ranking.

In principle, the two issues are randomised from the database. However, it is useful to select interesting pairs for comparison. This can be done by weighting the probabilities for being selected. This is a tentative list of criteria that make an issue or a pair interesting. In brief, the more established ranking, the less there is need for further questions.

• the issue has been ranked important before
• the previous answers have been highly varying
• the pair of this issue has been previously ranked in a similar way than this issue
• there are only a few answers so far about the issue.

Questionnaire, the respondent as a decision-maker

Suppose you are a decision-maker. Your advisors have been informed about a problem that needs to be solved. They have evaluated the problem and identified two possible solutions. They have also estimated the two outcomes of this decision, and they are convinced that either one will surely happen. However, they have not been able to identify the person(s) who will be affected. They can be anyone (in your town/country/the world; this is defined for the question), but possibly they include someone you know, and it is even possible that one of them is you. It is not known, whether the affected persons in the two outcomes are the same or not.

The decision is made so that those who are affected can find out who made the decision. (Is this necessary, because very often you can indeed find out who made a decision, but you cannot find out that the outcome was due to the decision?) You are authorised to make the decision alone. You can consult anyone if you wish, but the responsibility is only yours.

• Outcome A: A person loses an arm.
• Outcome B: A person loses a foot.

Choose between the four options:

1. I'd rather have A than B.
2. A and B are equal.
3. I'd rather have B than A.
4. It is impossible to answer this question because _______________________

Please provide some background information

1. Are you personally or is someone you love affected by the outcome defined in A? Y/N
2. Are you personally or is someone you love affected by the outcome defined in B? Y/N

---

Instead of making paired comparisons, the tool could provide a list of categories, where the user can select one or more. The contents of the selected categories appear as a list box. The user can drag and drop any issue from the list box to another list box that contains the issues to be evaluated. From here, the user can then drag and drop the issues into a third box, where the issues can be organised into the order of preference. In previous boxes, issues are in alphabetical order. Each issue in the third box can also be tagged "equally important as the previous issue". In this way, a large number of issues can be handled, and the user can rather quickly make his/her own ranking. There should be a selection of obligatory issues that are ranked always. There are then used as calibration points, as everyone has the same in their ranking. One critical issue is "nothing happens", which is the neutrality reference point.

This can be done with a database with the following tables and fields:

Issue

• issue_id
• Issue_description
• category_id

Category

• Category_id
• Category_description

User

• user_id
• password
• last_edition_date

Selection

• selection_id
• user_id
• issue_id (user_id+issue_id must be unique)

Assessment

• assessment_id
• selection_id
• preferred_over_id
• preference ['>','=']

When performing assessment, make query of Selection - Assessment for one user_id. Rules:

• preferred_over_id must be in Selection/issue_id
• preferred_over_id must be unique in this set. If it is selected another time:

→ 1) make X the selection_id; 2) find the row where preferred_over_id has its duplicate; 3) replace preferred_over_id on that row with X

Questionnaire, the respondent as himself

Consider the two issues and answer to the following questions.

• Issue A: The right arm.
• Issue B: The right foot.

From the table below, select the row that applies to you and select one of the three empty columns.

Select the row that applies to you	Outcomes	I prefer 1 to 2	I am indifferent	I prefer 2 to 1
I have both A and B	1: You lose A; 2: you lose B
I have A but not B	1: You lose A; 2: You get B
I have B but not A	1: You get A; 2: You lose B
I have neither A nor B	1: You get A; 2: You get B

Evaluating the simlarity of things

There is a critical problem in this approach that if/when the number of issues grows, the number of necessary comparisons grows in the second power. It is therefore crucial that only representative and relevant issues are taken into the database. The operator should be critical in what to put in the database. On the other hand, he should carefully listen to what the participants suggest, so that the collection of issues would be as good as possible. Below, there is one suggestion about how to organise issues so that feedback is collected from the users.

There are two outcomes that have been suggested in the outcome database. On the other hand, there are suggestions that there is no need to separate these two outcomes. Please tell what you think.

• Outcome A: A person loses an arm.
• Outcome B: A person loses a foot.

Choose between the options:

1. A is covered by B, and there is no need to describe A separately.
2. B is covered by A, and there is no need to describe B separately.
3. A and B are different things, and they cannot be described as a single outcome because _________________
4. Both A and B could be covered with a single outcome, which is ______________