Value of information

This page is a knowledge crystal of subtype method. The page identifier is Op_en2480
Moderator:Jouni (see all)
Citation of this page: Jouni T. Tuomisto: Value of information. Opasnet 2010. [1]. Accessed 23 Nov 2024.
Upload data {{#opasnet_base_link:Op_en2480}}

Value of information (VOI) in decision analysis is the amount a decision maker would be willing to pay for information prior to making a decision.^[1]. Value of information is specific to a combination of a particular decision with several options, a particular objective (i.e., outcome of interest that can be quantitatively estimated), and a particular issue that is affected by the decision and is relevant for the objective. If all such issues are considered at the same time, we talk about expected value of perfect information.<section end=glossary />

Question

How can value of information be calculated in an assessment in such a way that

it helps in understanding the impacts of uncertainties on conclusions and
it helps to direct further assessment efforts to improve guidance to decision making?

Answer

+ Show code - Hide code

library(OpasnetUtils)

objects.latest("Op_en2480", code_name = "Initiate")

decisions <- data.frame(
	Decision = "D1", 
	Option = c("BAU", "D1b"), 
	Variable = "ova", 
	Cell = "C1:A", 
	Change = "Add", 
	Result = c(0, 0.1)
)

DecisionTableParser(decisions)

ova <- EvalOutput(Ovariable(name = "ova", 
	data = data.frame(C1 = c("A", "B"), Result = c("1-3", 2))
))

ova <- CheckDecisions(ova)

result(ova) <- result(ova) + runif(4000, 0, 1)

voi <- VOI(ova, decision = "D1", indices = "C1")

oprint(voi)

Rationale

Input

To calculate value of information, you need

a decision to be made with at least two different options a decision maker can choose from,
an objective (i.e., outcome of interest or indicator) that can be quantitatively estimated and optimised,
an optimising function to be used as the criterion for the best decision,
an uncertain variable of interest (optional, needed only if partial VOI is calculated for the variable; if omitted, combined value of information is estimated for all uncertain variables in the assessment model).

+ Show code - Hide code

library(OpasnetUtils)
library(ggplot2)

Optimize <- function(dat, indices = NULL, keepindex = TRUE){
	# dat is an ovariable.
	# indices is a vector of indices to save in the output. If NULL, only one value is kept.
	# keepindex = TRUE keeps the indices in the output.
	# Optimize finds the smallest value of dat for each unique group defined by indices.

	dat@output <- dropall(dat@output[!is.na(result(dat)) , ]) # Drop reduntand rows and locations

	if(is.null(indices)) {
		ind <- rep(1, length(result(dat)))
	} else {
		ind <- dat@output[indices]
	}
	
	minrow <- tapply(result(dat), ind, which.min) # Find the row for smallest value of each unique group
	rownum <- tapply(1:nrow(dat@output), ind, list) # Order row numbers according to unique groups.
   
	temp <- NULL
	for (i in 1:length(rownum)) {
		temp <- c(temp, rownum[[i]][minrow[i]]) # Pick the row with minimum of each unique group
	}
	temp <- temp[!is.na(temp)]
	
	keep <- colnames(dat@output)

	if(!keepindex) {
		keep <- keep[!keep %in% indices]
		if(is.null(indices)) {keep <- comment(result(dat))}
	}

	dat@output <- dat@output[temp , ]
	dat@output <- dat@output[keep]
	
	dat@marginal <- colnames(dat@output) %in% indices
	return(dat)
}

Aggregate <- function(dat, indices = NULL, fun = mean, name = NULL){
	# dat is an ovariable.
	# indices is a vector of indices to save in the output.
	# fun in the function to use with aggregation.
	# name is the name to be used with the new ovariable.
	# Aggregate collapses an ovariable in a similar way as the generic function aggregate.
	
	dat@output <- dropall(dat@output[!is.na(result(dat)) , ]) # Drop reduntand rows and locations

	if(is.null(indices)) {
		ind <- data.frame(Temp = rep(1, length(result(dat))))
	} else {
		ind <- dat@output[indices]
	}
	
	temp <- as.data.frame(as.table(tapply(result(dat), ind, fun)))
	temp <- temp[colnames(temp) != "Temp"]
	
	if(is.null(name)) {
		colnames(temp)[colnames(temp) == "Freq"] <- comment(result(dat))
	} else {
		colnames(temp)[colnames(temp) == "Freq"] <- paste(name, "Result", sep = "")
		dat@name <- name
	}
	
	dat@output <- temp

	dat@marginal <- colnames(dat@output) %in% indices
	return(dat)
}

VOI <- function(ova, decision, indices = colnames(ova@output)[ova@marginal]) {
	# ova is an ovariable
	# decision is an index that is used as the decision to be considered.
	# indices are the indices whose partial values of information are estimated.
	
	ncuu <- Optimize(Aggregate(ova, decision, name = "ncuu"), keepindex = FALSE)

	evpi <- Aggregate(Optimize(Aggregate(ova, c("Iter", decision), name = "evpi"), "Iter")) - ncuu

	if(length(indices) == 0) {return(evpi@output)}

	evppi <- data.frame()

	for(i in 1:length(indices)) {
		temp <- Aggregate(Optimize(Aggregate(ova, c(decision, indices[i]), name = "evppi"), c(indices[i]))) - ncuu
		evppi <- rbind(evppi, data.frame(Var = indices[i], temp@output))
	}

	out <- orbind(evpi, evppi)

	return(out)
}

objects.store(Optimize, Aggregate, VOI)

cat("Functions Optimize, Aggregate, and VOI stored.\n")

Output

Value of information, i.e. the amount of money that the decision-maker is willing, in theory, to pay to obtain a piece of information. Value of information can also be measured in other units than money, e.g. disability-adjusted life years if health impacts only are considered.

Rationale

See Decision theory, Value of information, and Expected value of perfect information in Wikipedia.
There are different kinds of indicators under value of information, depending on what level of information is compared with the current situation:
EVPI

Expected value of perfect information (everything is known perfectly)

EVPPI

Expected value of partial perfect information (one variable is known perfectly, otherwise current knowledge)

EVII

Expected value of imperfect information (things are known better but not perfectly)

EVPII

Expected value of partial imperfect information (one variable is known better but not perfectly, otherwise current knowledge)

EVIU

Expected value of including uncertainty (a decision analysis can ignore uncertainties and go with expected value of each variable, or include uncertainty and propagate that through the model. There is a difference especially if the uncertainty distributions are skewed.)

EVIO

Expected value of including an option (is there any value of including a non-optimal decision option in the final assessment?)

Standard VOI approach with counter-factual world descriptions

Counter-factual world descriptions mean that we are looking at two or more different world descriptions that are equal in all other respects except for a decision that we are assessing. In the counter-factual world descriptions, different decision options are chosen. By comparing these worlds, it is possible to learn about the impacts of the decision. With perfect information, we could make the theoretically best decision by always choosing the right option. If we think about these worlds as Monte Carlo simulations, we run our model several times to create descriptions about possible worlds. Each iteration (or row in our result table about our objective) is a possible world. For each possible world (i.e., row), we create one or more counter-factual worlds. They are additional columns which differ from the first column only by the decision option. With perfect information, we can go through our optimising table row by row, and for each row pick the decision option (i.e., the column) that is the best. The expected outcome of this procedure, subtracted by the outcome we would get by optimising the expectation (net benefit under uncertainty), is the expected value of perfect information (EVPI). ^[2] ^[3] ^[4] ^[5]

Screening approach with decisions as random variables

In this case, we do not create counter-factual world descriptions, but only a large number of possible world descriptions. The decision that we are considering is treated like any other uncertain variable in the description, with a probability distribution describing the uncertainty about what actually will be decided. In this case, we are comparing world descriptions that contain a particular decision option with other world descriptions that contain another decision option. It is important to understand that we are not comparing two counter-factual world descriptions, but we are comparing a group or possible world descriptions to another group of world descriptions.

The major benefit of the screening approach is that it is not necessary do define decision variables beforehand. Basically any variable can be taken to be a decision, as long as it is a meaningful as a decision and the model has a number of possible worlds simulated with Monte Carlo or another method such as Bayesian belief network (BBN). The idea is to conditionalise the decision variable to one decision option at a time and then compare these conditionalisations to find out which one of them gives the optimal outcome in the objective.

In this approach, it is not possible to calculate EVPI in such a straightforward way as with counter-factual world descriptions. Therefore, with this approach, we are pretty much restricted to calculating expected value of partial perfect (and imperfect) information, or EVPPI and EVPII, respectively. Some sophisticated mathematical methods may be developed to calculate this, but it is beyond my competence. One approach sounds promising to me at them moment. It is used with probabilistic inversion, i.e. using bunches of probability functions instead of point-wise estimates.^[6]

There is a major difference between the two approaches. Counter-factual world descriptions are actually utilising the Do operator described by Pearl ^[7], which looks at impacts of forced changes of a variable. In contrast, the latter case has the structure of an observational study, which looks at natural changes where several variables change at the same time. Therefore, it is subject to confounders, which are typical problems in epidemiology: a variable is associated with the effect, but not because it is its cause but because it correlates with the true cause.

Because of this confounding effect, the latter method for value-of-information analysis may result in false negatives: a decision seems to be obvious (i.e., the VOI is zero), but a more careful analysis of confounders would show that it is not. Therefore, a value-of-information analysis based on a Bayesian net should be repeated with an analysis of counter-factual world descriptions. In Uninet, counter-factual world descriptions can be created with analytical conditioning, but it does not work with functional nodes, and its applicability is therefore limited.

Result

Procedure

EVPI is calculated using the following equation:

EVPI = E(Max(U(d_i,θ))) - Max(E(U(d_i,θ))),

where E=expectation over uncertain parameters θ, Max=maximum over decision options i, U=utility of decision d (i.e., the value of outcome after a particular decision option i is chosen, measured in money, DALY, or another quantitative metric covering all relevant impacts).

The general formula for EVPII is:

EVPII = E_θ2(U(Max(E_θ2(U(d_i,θ2))),θ2)) - E_θ2(U(Max(E_θ1(U(d_i,θ1))),θ2)),

where θ1 is the prior information and θ2 is the posterior (improved) information. EVPPI can be calculated with the same formula in the case where P(θ2)=1 if and only if θ2=θ1. If θ includes all variables of the assessment, the formula gives total, not partial, value of information.

The interpretation of the formula is the following (starting from the innermost parenthesis). The utility of each decision option d_i is estimated in the world of uncertain variables θ. Expectation over θ is taken (i.e. the probability distribution is integrated over θ), and the best option i of d is selected. The point is that in the first part of the formula, θ is described with the better posterior information, while the latter part is based on the poorer prior information. Once the decision has been made, the expected utility is estimated again based on the better posterior information in both the first and second part of the formula. Finally, the difference between the utility after the better and poorer information, respectively, gives the value of information.

Management

A previous Analytica version of VOI calculation is archived. The related model file is File:VOI analysis.ANA.

Impact of a strong correlation between the decision and a variable

There is a problem with the approach using the decision as a random variable. The problem occurs with variables that are strongly correlated with the decision variable. The iterations are categorised into "VOI bins" based on the variable to be studied. In addition, iterations are categorised into "decision bins" based on the value of the decision variable. The idea is to study one VOi bin at a time and find the best decision bin within that VOI bin. If the best decision is different in different VOI bin, there is some value of knowing to which VOI bin the true value of the variable belongs. However, if the variable correlates strongly with the decision, it may happen that all iterations that are in a particular VOI bin are also in a particular decision bin. Then, it is impossible to compare different decision bins to find out which decision is the best in that VOI bin.

This problem can be overcome by assessing counter-factual worlds, because then there is always the same number of iterations in every decision bin. The conclusion of this is that the VOI analysis using decisions as random variables is a simple and quick screening method, but it cannot be reliably used for a final VOI analysis. In contrast, the counter-factual assessment is the method of choice for that. Originally developed by Jouni Tuomisto and Marko Tainio, National Public Health Institute (KTL), Finland, 2005. The screening version was developed by Jouni Tuomisto, National Institute for Health and Welfare (THL), 2009. (c) CC-BY-SA.

How to use the method