Quality evaluation criteria

This page is a knowledge crystal of subtype variable. The page identifier is Op_en2685
Moderator:Jouni (see all)
Upload data {{#opasnet_base_link:Op_en2685}}

Quality evaluation criteria are a set of measures that together describe the quality of content of an information object.

Scope

What is a set of quality evaluation measures such that it fulfils the following criteria:

• The evaluation measures are applicable to all objects.
• The evaluation brings information about the properties of the goodness of the object.

Definition

• Properties of a good assessment

Previous references: P-Box: approach. Vines and copulas:

The quality of the content of a variable can be divided into two main parts based on the golden standard: goodness of the result estimate compared with the truth, and goodness of the description of existing data compared with the actually existing data.

In practice, the dependencies and result can be compared with the truth, while data and formula can be compared with the actually existing data. This is because result and dependencies typically contain few references, while data and formula contain a lot. However, this can vary a lot from one variable to another and should not be used as a strict rule.

Comparison with the truth

Of course, truth is never actually known precisely, but still people are able to give their personal estimates (subjective probabilities) about the result of a variable. In addition, they are able to estimate which variables are causally related to the variable under consideration (i.e., parent variables). To operationalise this, two concepts are defined.

Result range

Result range R is a range of plausible values within which the true value of the variable is located [r_l, r_u]. It is described as a part of the result of a variable. What "plausible" exactly means is somewhat fuzzy, as it is based on the evaluation by the group of people who have produced the current version of the variable result.

Coverage

Coverage is defined as two subjective probabilities given by a user: probability that the truth is actually below the result range; and probability that the truth is actually above the result range. If the variable is not quantitative, but e.g. a discrete probability distribution with non-ordered values, coverage means any values that are not defined in the discrete distribution; in this case, coverage is described by only one probability estimate.

Coverage can be estimated for the result, and this is what is usually meant by the word. However, coverage can also be estimated for upstream dependencies. Then, it means the probability that the rank correlation between this variable and a parent is smaller (or larger) than the current estimate (described in dependencies). It is important to notice that if a dependency is not mentioned at all, this implies a rank correlation between the variable and its parent that is exactly 0.

The individual coverage estimates can be aggregated into a probability distribution that is wide enough to capture the true value with high subjective aggregated probability estimated by the group. The distribution is clearly wider than the aggregated best estimate of the distribution. The usefulness of coverage is that with a draft assessment, coverage can be used in VOI analysis, and it is unlikely to result in false negative (the distribution being too narrow falsely implying that no further research is needed).

Subjective coverages can be aggregated to what is called group coverage.

See also

• Previous idea about "Amount of data"
• Previous idea about a technical classification of quantitative results (placeholder, guesstimate, result range, marginal distribution, joint distribution)