Net Monetary Benefit Lines Augmented with Value-of-Information Measures to Present the Results of Economic Evaluations under Uncertainty

Abstract

Background

Methods to present the result of cost-effectiveness analyses under parameter uncertainty include cost-effectiveness planes (CEPs), cost-effectiveness acceptability curves/frontier (CEACs/CEAF), expected loss curves (ELCs), and net monetary benefit (NMB) lines. We describe how NMB lines can be augmented to present NMB values that could be achieved by reducing or resolving parameter uncertainty. We evaluated the ability of these methods to correctly 1) identify the alternative with the highest expected NMB and 2) communicate the magnitude of parameter and decision uncertainty.

Methods

We considered 4 hypothetical decision problems representing scenarios with high variance or correlated cost and effect estimates and alternatives with similar cost-effectiveness ratios. We used these decision problems to demonstrate the limitations of existing methods and the potential of augmented NMB lines to resolve these issues.

Results

CEPs and CEACs/CEAF could falsely imply the lack of sufficient evidence to identify the optimal option if cost and effect estimates have high variance, are correlated across alternatives, or when alternatives have similar cost-effectiveness ratios. The augmented NMB lines and ELCs can correctly identify the option with the highest expected NMB and communicate the potential benefit of resolving uncertainties. Like ELCs, the augmented NMB lines provide information about the value of resolving parameter uncertainties, but augmented NMB lines may be easier to interpret for decision makers.

Conclusions

Our analysis supports recommending the augment NMB lines as an important method to present the results of economic evaluation studies under parameter uncertainty.

Highlights

The results of cost-effectiveness analyses (CEAs) when the cost and effect estimates of alternatives are uncertain are commonly presented using cost-effectiveness planes (CEPs), cost-effectiveness acceptability curves/frontier (CEACs/CEAF), and expected loss curves (ELCs).

Although currently not often used, net monetary benefit (NMB) lines could present the results of cost-effectiveness to identify the alternative with the highest expected NMB values given the current level of uncertainty. Furthermore, NMB lines can be augmented to 1) show metrics of value of information, which measure the value of additional research to reduce or eliminate the decision uncertainty, and 2) display the confidence intervals along the NMB lines to ensure that NMB values are estimated accurately using a sufficiently large number of parameter samples.

Using several decision problems, we demonstrate the limitation of existing methods to present the results of CEAs under parameter uncertainty and how augmented NMB lines could resolve these issues.

Our analysis supports recommending augmented NMB lines as an important method to present the results of CEA under uncertainty since they 1) correctly identify the alternative with the highest expected NMB value given the current evidence, 2) provide information about the potential value of additional research to improve the decision by reducing or resolving uncertainty in model parameters, 3) assist the analysis to visually ensure that enough parameter samples are used to estimate the expected NMB of alternatives, and 4) are easier to interpret for decision makers compared with other methods.

Keywords

cost-effectiveness analysis probabilistic analysis probabilistic sensitivity analysis value of information

The main purpose of an economic evaluation is to identify the optimal option among a number of alternatives with different cost and health outcomes. In practice, the cost and effect of alternatives cannot be calculated with certainty. This is mainly due to the inherent uncertainties in model parameters and future factors influencing the cost and effect of alternatives.^1,2 For example, estimates for the quality of life in a disease state are obtained based on a limited number of participants, and hence, they are subject to sampling error. Consequently, a probabilistic analysis (PA), also known as a probabilistic sensitivity analysis (PSA), is recommended to propagate uncertainties from input parameters to the estimated costs and effects of alternatives.^3,4

The following approaches are currently often used to present the results of economic evaluations under uncertain estimates for the costs and effects of alternative: 1) cost-effectiveness planes (CEPs) with probability clouds presented for the projected cost and effect of alternatives, 2) cost-effectiveness acceptability curves (CEACs), 3) cost-effectiveness acceptability frontier (CEAF), and 4) expected loss curves (ELCs).⁵ For a risk-neutral decision maker whose objective is to maximize the population net monetary benefit (NMB)—an assumption held by all methods listed above—the optimal choice among a set of mutually exclusive alternatives is the one with the highest expected NMB give the current evidence.^6,7 Therefore, although less commonly used, NMB lines represent the fifth approach to present the results of economic evaluations under uncertainty.^7–9

Limitations of CEPs and CEACs have been rigorously discussed in the literature^6,10,11; CEPs provide visual presentations for the costs and effects of alternatives (along with the uncertainties in their estimates), and CEACs present the probability that each alternative has the highest NMB value, but neither is designed to identify the optimal alternative (i.e., the alternative with the highest expected NMB). To resolve some of these limitations, CEAF was developed, which marks the alternatives with the highest expected NMB on CEACs. CEAF, however, does not provide any information related to the magnitude of difference in the performance of different alternatives or the gain in NMB if parameter uncertainties are reduced or resolved. In contrast, ELCs show the alternatives with the highest expected NMB and the expected gain in NMB if decision uncertainty is resolved by additional research.^5,6 Despite their advantages, ELCs are rarely presented in cost-effectiveness studies.

In this study, we describe how NMB lines could be augmented to also present the potential value of additional research (e.g., the expected gain in NMB if uncertainties in parameters would be resolved). Using several hypothetical decision problems, we assess the ability of existing methods and the augmented NMB lines to identify the optimal option given the current evidence and to communicate the magnitude of parameter uncertainty and the potential gain from resolving the uncertainty. These decision problems represent scenarios with high variance of cost and effect estimates, correlated costs and effects across alternatives, and alternatives with similar cost-effectiveness ratios.

Health Care Resource Allocation under Uncertainty

We consider a resource allocation problem in which $M + 1$ alternatives, indexed by $i \in {0, 1, 2, \dots, M}$ , are available with $i = 0$ denoting the status quo (or the “do nothing” option). We denote the expected cost and effect of alternatives $i$ by $c_{i}$ and $e_{i}$ , respectively. We note that due to parameter uncertainty, the true cost and effect of alternatives (or the realized values once the alternatives are implemented) are not known. Hence, $c_{i}$ and $e_{i}$ are the expected cost and effect of this alternative $i$ given the current evidence about the parameters determining these outcomes. Mathematically, if random variables $C_{i}$ and $E_{i}$ denote the cost and effect of alternative $i$ in the presence of parameter uncertainty, then $c_{i} = E [C_{i}]$ and $e_{i} = E [E_{i}]$ . We also note that $c_{i}$ s and $e_{i}$ s could be defined at the individual level or at the population level depending on the decision problem under consideration. For example, $c_{i}$ could represent the overall population cost incurred during a pandemic under different mitigation strategies or represent the expected cost incurred during the lifetime of a person diagnosed with colon cancer under different cancer treatment options.

Relative to the status quo, the implementation of alternative $i$ is expected to increase the overall cost by $Δ c_{i} = c_{i} - c_{0}$ and to produce $Δ e_{i} = e_{i} - e_{0}$ units of effect. For a decision maker whose objective is to maximize the population NMB, the optimal alternative is the one with the highest expected incremental NMB value, which for alternative $i$ is defined as $ω Δ e_{i} - Δ c_{i}$ ; here, $ω \geq 0$ represents the decision maker’s willingness-to-pay (WTP) threshold for 1 additional unit of effect.^6,7 We formulate this problem as

\begin{matrix} Δ NM B^{*} (ω) = max_{i \in {0, 1, 2, \dots, M}} (ω Δ e_{i} - Δ c_{i}), \end{matrix}

(1)

where $Δ N M B^{*} (ω)$ is the increase in the expected population NMB if the optimal alternative is selected.

We note that this formulation assumes that the decision maker is risk neutral, an assumption held by all methods commonly used to present the results of CEA under parameter uncertainty. The risk-neutrality assumption suggests that the decision maker’s utility increases linearly in the accumulated NMB. For example, the objective function ( $1$ ) implies that the decision maker is indifferent between increasing the total NMB from 0 to 1 or from 1,000 to 1,001, since both represent an increase of 1 unit of NMB. For a risk-neutral decision maker, the variance of outcomes does not affect the decision maker’s preference over the available options, and the decision maker should choose the option with the highest expected incremental NMB, as formulated by the objective function ( $1$ ).

We also note that in the above formulation, alternatives are assumed to be mutually exclusive (i.e., only 1 alternative can be selected as the optimal choice). When alternatives are not mutually exclusive, a new set of mutually exclusive alternatives could be constructed that includes all subsets of the alternatives under consideration. For example, if compatible alternatives {status quo, A, B} are available, $2^{3} = 8$ subsets of mutually exclusive options could be considered: {status quo only, A only, B only, status quo + A, status quo + B, A + B, and status quo + A + B}.

Due to uncertainties in the model parameters, the true cost and effect of alternatives cannot be known with certainty. The good modeling practice guidelines recommend propagating that uncertainty in the mean value of input parameters by assigning appropriate probability distributions to all relevant input parameters.³ For example, cost parameters are assigned a gamma distribution, or health utility values are assigned a beta distribution. Next, the expected cost and effect of each alternative in the presence of parameter uncertainty (i.e., $c_{i}$ and $e_{i}$ ) are estimated through the Monte Carlo approach, where the cost and effect of each alternative are repeatedly calculated for many random draws from the assumed probability distributions for parameters; the expected cost and effect of each intervention in the presence of parameter uncertainty is then estimated by the mean of cost and effect samples: ${\bar{C}}_{i} = \sum_{j = 1}^{N} C_{i, j} / N$ and ${\bar{E}}_{i} = \sum_{j = 1}^{N} E_{i, j} / N$ , where ${C_{i, 1}, C_{i, 2}, \dots, C_{i, N}}$ and ${E_{i, 1}, E_{i, 2}, \dots, E_{i, N}}$ are $N$ cost and effect samples obtained for intervention $i$ . This approach is commonly referred to as a PA or PSA.^8,12

Since $c_{i}$ and $e_{i}$ are unknown and can only be estimated with some noise, the optimal alternative is selected based on the estimated expected incremental NMBs. That is, to identify the optimal alternative, the following problem is solved instead of problem ( $1$ ):

\begin{matrix} Δ {\bar{NMB}}^{*} (ω) = max_{i \in {0, 1, 2, \dots, M}} (ω {\bar{Δ E}}_{i} - {\bar{Δ C}}_{i}), \end{matrix}

(2)

where ${\bar{Δ C}}_{i} = {\bar{C}}_{i} - {\bar{C}}_{0}$ and ${\bar{Δ E}}_{i} = {\bar{E}}_{i} - {\bar{E}}_{0}$ are the estimates for the expected incremental cost and effect of alternative $i$ with respect to the status quo. Increasing $N$ would reduce the chance of choosing a suboptimal alternative (i.e., an alternative that does not have the highest $ω Δ e_{i} - Δ c_{i}$ ). This is more formally presented in the next subsection.

Consistency of Estimates Provided by PA

There are 2 important assumptions underlying PA:

Cost and effect estimates $C_{i, j}$ s and $E_{i, j}$ s are unbiased estimates for the expected cost and effect of alternative $i$ (i.e., $E [C_{i, j}] = c_{i}$ and $E [E_{i, j}] = e_{i}$ ). This means that the model is correctly specified and the choice of parameter distributions are justified such that the cost estimates ${C_{i, 1}, C_{i, 2}, \dots, C_{i, N}}$ and the effect estimates ${E_{i, 1}, E_{i, 2}, \dots, E_{i, N}}$ are unbiased estimates for the expected cost and effect of alternative $i$ .

For each alternative $i$ , the cost estimates ${C_{i, 1}, C_{i, 2}, \dots, C_{i, N}}$ are independent and identically distributed (iid) and the effect estimates ${E_{i, 1}, E_{i, 2}, \dots, E_{i, N}}$ are iid (or equivalently, pairs ${(C_{i, 1}, E_{i, 1}), (C_{i, 2}, E_{i, 2}), \dots, (C_{i, N}, E_{i, N})}$ are iid for each alternative $i$ ). We note that this assumption does not require cost and effect estimates to be independent from each other; that is, $corr (C_{i, j}, E_{i, j})$ could be nonzero for any $(i, j)$ .

Under these 2 assumptions, by the Law of Large Numbers, the sample means ${\bar{C}}_{i}$ and ${\bar{E}}_{i}$ are consistent estimators for the expected cost $c_{i}$ and the expected effect $e_{i}$ . That is, ${\bar{C}}_{i} \to c_{i}$ and ${\bar{E}}_{i} \to e_{i}$ as $N \to \infty$ . Hence, as $N \to \infty$ :

\begin{matrix} ω {\bar{Δ E}}_{i} - {\bar{Δ C}}_{i} \to ω Δ e_{i} - Δ c_{i}, \end{matrix}

(3)

which suggests that, under the above 2 assumptions, the estimated expected incremental NMB ( $ω {\bar{Δ E}}_{i} - {\bar{Δ C}}_{i}$ ) is a consistent estimator for the expected incremental NMB ( $ω Δ e_{i} - Δ c_{i}$ ). Therefore, the alternative with the highest estimated incremental NMB (i.e., $ω {\bar{Δ E}}_{i} - {\bar{Δ C}}_{i}$ ) would also be the one with the highest expected incremental NMB (i.e., $ω Δ e_{i} - Δ c_{i}$ ) when $N \to \infty$ .

In practice, the number of samples from parameter distributions (i.e., $N$ ) could not be infinitely large due to limited computing resources. Hence, the estimated expected incremental NMB ( $ω {\bar{Δ E}}_{i} - {\bar{Δ C}}_{i}$ ) could be different from $ω Δ e_{i} - Δ c_{i}$ , and the alternative identified as the optimal by problem ( $2$ ) may not necessarily maximize the population expected NMB. Increasing $N$ would reduce the chance of choosing a suboptimal alternative (i.e., an alternative that does not have the highest $ω Δ e_{i} - Δ c_{i}$ ).

Pairing the Cost and Effect Estimates of Alternatives

Since $ω {\bar{Δ E}}_{i} - {\bar{Δ C}}_{i}$ is used to estimate $ω Δ e_{i} - Δ c_{i}$ , we would like $ω {\bar{Δ E}}_{i} - {\bar{Δ C}}_{i}$ to have a small variance. A common approach to reduce the variance of estimates for the incremental cost and effect of alternatives is to pair the cost and effect estimates across all alternatives. This is motivated by noting that, for example, $Var [{\bar{Δ C}}_{i}]$ is

Var [{\bar{Δ C}}_{i}] = Var [{\bar{C}}_{i}] + Var [{\bar{C}}_{0}] - 2 Cov [{\bar{C}}_{i}, {\bar{C}}_{0}] .

Hence, by inducing a positive correlation between cost estimates for alternatives 0 and $i$ , we could lower $Var [{\bar{Δ C}}_{i}]$ since now, $Cov [{\bar{C}}_{i}, {\bar{C}}_{0}] > 0$ . A common approach to achieve this is to use the same parameter values for all alternatives when obtaining cost estimates $C_{0, j}, C_{1, j}, \dots, C_{M, j}$ and effect estimates $E_{0, j}, E_{1, j}, \dots, E_{M, j}$ during the $j$ ^th iteration of PA. That way, cost and effect estimates $(C_{i, j}, E_{i, j})$ are paired across all alternatives in the $j$ ^th iteration of PA because the same parameter set is used to obtain $(C_{0, j}, E_{0, j}), (C_{1, j}, E_{1, j}), \dots, (C_{M, j}, E_{M, j})$ .

Throughout the article, we hold the assumptions that cost and effect estimates of alternatives are paired using the method described above, and we define $Δ C_{i, i} = C_{i, j} - C_{0, j}$ and $Δ E_{i, i} = E_{i, j} - E_{0, j}$ as estimates for the incremental cost and effect of alternative $i$ obtained from the $j$ ^th iteration of PA.

Methods to Present the Results of Economic Evaluations under Uncertainty

We first describe a hypothetical resource allocation problem to illustrate and review the basics of methods that are commonly used to present the results of economic evaluations under uncertainty. We then assess the ability of these methods to identify and unambiguously communicate the optimal alternative and the magnitude of parameter uncertainty under different scenarios.

An Illustrative Example

We consider a decision problem in which, in addition to the status quo option, new alternatives ${A, B, C, D}$ are available with $(Δ c_{A}, Δ e_{A}) = ($ 250, 000, 20)$ , $(Δ c_{B}, Δ e_{B}) = ($ 500, 000, 10)$ , $(Δ c_{C}, Δ e_{C}) = ($ 750, 000, 25),$ and $(Δ c_{D}, Δ e_{D}) = ($ 1250, 000, 40)$ . Due to uncertainty in parameter values, the cost and effect estimates provided by PA (i.e., $Δ C_{i, j}$ s and $Δ E_{i, j}$ s) may deviate from their expected values. To model this deviation, we let $Δ C_{i, j} = Δ c_{i} + ϵ_{i, j}$ and $Δ E_{i, j} = Δ e_{i} + δ_{i, j}$ , where $ϵ_{i, j}$ and $δ_{i, j}$ represent the deviation from expected values $Δ c_{i}$ and $Δ e_{i}$ in the $j$ ^th iteration of PA. In this illustrative example, we assumed that $(ϵ_{i, j}, δ_{i, j})$ are normally distributed with mean $(0, 0)$ and standard division $($ 100, 000, 7.5)$ for alternative A, $($ 100, 000, 7.5)$ for alternative B, $($ 100, 000, 4)$ for alternative C, and $($ 150, 000, 5)$ for alternative D.

CEP and Cost-Effectiveness Frontier

A CEP is probably the most commonly used approach to present the results of an economic evaluation under uncertainty (Figure 1A).^8,13 The y-axis of a CEP represents the additional cost of each alternative with respect to the status quo option, and the x-axis represents the additional effect of each alternative with respect to the status quo option. CEPs display the estimated incremental cost and effect of each alternative (i.e., $({\bar{Δ C}}_{i}, {\bar{Δ E}}_{i})$ ) and the probability clouds produced by estimates ${Δ C_{i, 1}, Δ C_{i, 2}, \dots, Δ C_{i, N}}$ and ${Δ E_{i, 1}, Δ E_{i, 2}, \dots, Δ E_{i, N}}$ to present the uncertainty in cost and effect of alternatives. The integral element of these figures is the cost-effectiveness (or efficiency) frontier (gray lines in Figure 1A), which identifies the alternatives that could be considered by the decision makers.^6,14,15 Alternatives that are not on the frontier are considered dominated and hence are not recommended to consider. To easily identify the alternative with the highest expected incremental NMB for a given WTP value, CEPs are often accompanied by CEACs and CEAF, which we describe next.

Figure 1

Results of economic evaluation analyses under uncertainty: (A) cost-effectiveness plane (CEP) with probability clouds, (B) cost-effectiveness acceptability curves (CEACs) and cost-effectiveness acceptability frontier (CEAF), (C) expected loss curves (ELCs), and (D) net monetary benefit (NMB) lines with perfect information curve representing the highest incremental NMB that could be achieved by resolving parameter uncertainty. In B–D, the optimal alternative (i.e., the alternative with the highest estimated NMB) is represented by the bold curve for each willingness-to-pay threshold.

CEACs and CEAF

The CEAC of an alternative represents the probability that the alternative has the highest NMB, given the current parameter uncertainty, as the function of the decision maker’s WTP value ( $ω$ ).^6,16 The CEAC of the alternative $i$ is estimated by $CEA C_{i} (ω) = I_{i} (ω) / N$ , where $I_{i} (ω)$ is the number of PA iterations where the intervention $i$ has the highest NMB (i.e., number of $j$ s with $ω E_{i, j} - C_{i, j} > ω E_{i^{'}, j} - C_{i^{'}, j}$ for every $i^{'} \neq i$ ). The CEACs for the alternatives considered in our illustrative example are presented in Figure 1B.

The CEAF identifies the alternative with the highest estimated expected NMB and is calculated as $CEAF (ω) = CEA C_{i} (ω)$ if the alternative $i$ has the highest estimated expected NMB (i.e., $ω {\bar{E}}_{i} - {\bar{C}}_{i} > ω {\bar{E}}_{i^{'}} - {\bar{C}}_{i^{'}}$ for every $i^{'} \neq i$ ).⁶ In Figure 1B, CEAF is represented by bold curves.

ELCs

The ELCs display the expected forgone benefit if a suboptimal strategy is chosen (Figure 1C).^5,13,17 For strategy $i$ , ELC is calculated as

EL C_{i} (ω) = \frac{1}{N} \sum_{j = 1}^{N} (Δ {NMB}_{j}^{*} (ω) - Δ NM B_{i, j} (ω)),

where $Δ N M B_{j}^{*} (ω)$ is the estimated incremental NMB of the optimal alternative for the PA sample $j$ at the WTP threshold $ω$ (i.e., $Δ {NMB}_{j}^{*} (ω) = max_{i \in {0, 1, 2, \dots, M}} (ω Δ E_{i, j} - Δ C_{i, j})$ ) and $Δ NM B_{i, j} (ω) = ω Δ E_{i, j} - Δ C_{i, j}$ .⁵ The 4 important properties of ELCs are⁵

1. The strategy with the lowest ELC is the same as the strategy with the highest NMB.

2. The ELC of the strategy with the lowest ELC represents the expected value of perfect information (EVPI), which measures the expected gain in NMB if the uncertainty in the estimate of cost and effect of all alternatives would be resolved.^4,18 EVPI is calculated as:

EVPI (ω) = PI (ω) - Δ {\bar{NMB}}^{*} (ω),

where ${\bar{N M B}}^{*} (ω)$ is the estimated expected NMB value achieved in the presence of uncertainty (defined in Eq. ( $2)$ ) and $PI (ω)$ is the expected NMB value achieved when uncertainties are resolved (i.e., perfect information) and is estimated as

\begin{matrix} PI (ω) = \frac{1}{N} \sum_{j = 1}^{N} max_{i} (ω Δ E_{i, j} - Δ C_{i, j}) . \end{matrix}

(4)

We note that to determine the true magnitude of gain from resolving uncertainty, the EVPI value would need to be scaled to the population level.⁴

3. The distance between ELCs measures the difference in the estimated NMB of alternatives associated with each ELC.

4. The WTP values at which the curves with the lowest expected loss (Figure 1C) intersect represent the incremental cost-effectiveness ratios (ICERs) of the alternative on the right-hand side of the intersect with respect to the alternative on the left-hand side of the intersect. For example, the green curve (alternative O) and blue curve (alternative A) in Figure 1C cross at the WTP value of $12,289 per effect, which is the estimated ICER of alternative A with respect to alternative O.

NMB Lines

For a given alternative $i$ , the expected incremental NMB can be estimated as $ω {\bar{Δ E}}_{i} - {\bar{Δ C}}_{i}$ for the WTP threshold $ω$ (Figure 1D). In Figure 1D, the bold lines represent the alternative with the highest NMB for each WTP threshold.

Since NMB lines show the magnitude of gain in NMB under each alternative considered compared with the status quo, they may be easier to understand compared with other methods discussed so far. For a given WTP threshold, the line with the highest value represents the optimal strategy, and the difference between that line and other lines represents the expected loss in NMB if a suboptimal alternative is selected. From that perspective, NMB lines have an advantage over CEACs/CEAF since the difference between CEACs does not provide any meaningful information.

NMB lines allow for visualizing the confidence intervals for the estimates of the expected incremental NMB under each alternative.^7,9 If the incremental cost and effect estimates ${\bar{Δ C}}_{i}$ s and ${\bar{Δ E}}_{i}$ s follow normal distributions, then the $100 (1 - α) %$ standard t confidence regions can be calculated for the estimated expected incremental NMB under alternative $i$ (i.e.,) as

\begin{matrix} ω {\bar{Δ E}}_{i} - {\bar{Δ C}}_{i} \pm t_{N - 1, \frac{α}{2}} \sqrt{\frac{S^{2} (ω)}{N}}, \end{matrix}

(5)

where $t_{N - 1, \frac{α}{2}}$ is the upper $\frac{α}{2}$ critical point for the t distribution with $N - 1$ degrees of freedom and $S^{2} (ω)$ is the variance of $ω {\bar{Δ E}}_{i} - {\bar{Δ C}}_{i}$ , which can be calculated as

\begin{array}{l} S^{2} (ω) = ω^{2} Var [{\bar{Δ E}}_{i}] + Var [{\bar{Δ C}}_{i}] - Cov (ω {\bar{Δ E}}_{i}, {\bar{Δ C}}_{i}) \\ = ω^{2} σ_{Δ U_{i}}^{2} + σ_{Δ C_{i}}^{2} - 2 ω ρ_{i} σ_{Δ E_{i}} σ_{Δ C_{i}} . \end{array}

(6)

Here, $σ_{Δ C_{i}}^{2}$ is the sample variance of ${Δ C_{i, 1}, Δ C_{i, 2}, \dots, Δ C_{i, N}}$ , $σ_{Δ E_{i}}^{2}$ is the sample variance of ${Δ E_{i, 1}, Δ E_{i, 2}, \dots, Δ E_{i, N}}$ , and $ρ_{i}$ is the correlation between samples ${Δ C_{i, 1}, Δ C_{i, 2}, \dots, Δ C_{i, N}}$ and ${Δ E_{i, 1}, Δ E_{i, 2}, \dots, Δ E_{i, N}}$ .

In Figure 1D, the t confidence intervals for each line are displayed by colored regions around each line. Displaying the confidence intervals along incremental NMB lines is a straightforward approach to ensure that enough parameter samples are used to obtain stable estimates for the expected incremental NMBs in the presence of parameter uncertainty. That is, $N$ is large enough such that the NMB lines are not expected to change meaningfully if recalculated using a new set of parameter samples. To this end, $N$ should be selected such that the confidence interval around the incremental NMB lines is narrow.

NMB Lines Augmented with Value-of-Information Measures

In this study, we propose further augmenting NMB lines to display value-of-information measures. For example, Figure 1D displays $PI (ω)$ curve (Eq. ( $4)$ ) on the figure of NMB lines. The $PI (ω)$ curve provides an upper bound for the maximum gain in NMB value if the uncertainty in the estimates of the costs and effects of all alternatives is resolved (for our illustrative example, this is represented by the dotted curve in Figure 1D). Therefore, the figure of the NMB line could provide the same information as in ELCs, although potentially in a more understandable fashion as they present the gain in NMB as opposed to the loss.

The distance between the $PI (ω)$ curve and the highest NMB line measures the EVPI (i.e., $EVPI (ω)$ ). If this gap is small, resolving the uncertainty in parameters would not lead to a considerable gain in NMB, and the decision maker could confidently select the alternative with the highest NMB line based on the currently available evidence. In contrast, when the distance between the $PI (ω)$ curve and the highest NMB line is large, attempting to resolve the uncertainties in parameters before selecting an alternative might be warranted. Nonetheless, if a decision must be immediately made, the optimal choice is the alternative with the highest NMB line, independent of the magnitude of gain in NMB if parameter uncertainty is resolved or reduced in the future.

We note that in addition to $PI (ω)$ , other measures capturing the value of resolving or reducing uncertainties could be displayed on this figure. These measures are defined in detail by Rothery et al.¹⁸ and Heath et al.¹⁹:

the partial perfect information curve is defined by Eq. (6) in Rothery et al.¹⁸ and measures the expected incremental NMB when the uncertainty in an individual parameter or a subset of parameters is fully resolved;

the sample information curve (SI) is defined by Eq. (9) in Rothery et al.¹⁸ and measures the expected incremental NMB when the uncertainty in an individual parameter or a subset of parameters is reduced (but not fully resolved); and

the net benefit of sample information curve is equal to the SI curve minus the expected cost of data collection to reduce parameter uncertainty. It allows accounting for the cost of conducting additional research to reduce parameter uncertainty.

While NMB lines allow for the display of all these value-of-information metrics, in this analysis, we display only $PI (ω)$ curves on NMB figures, as EVPI is the first step to assess decision uncertainty as indicated by the Value of Information Task Force papers.^4,18 This also allows us to present a fair comparison with ELCs and to maintain focus on illustrating how augmented NMB lines could resolve the limitations of existing methods to present the results of PA.

Illustrative Decision Problems

To assess the ability of methods to present the PA results to identify the optimal alternative and communicate the impact of parameter uncertainty and value of resolving decision uncertainty, we consider 4 decision problems with different characteristics. These decision problems are described below and summarized in Table 1.

Table 1

Description of Illustrative Decision Problems to Evaluate the Ability of Methods for Presenting the Results of Probabilistic Analyses to Identify the Optimal Alternative and Communicate the Impact of Parameter Uncertainty

Decision Problem	Alternatives Involved	Characteristics	Expected Incremental Cost and Effect with Respect to the Status Quo $(Δ c_{i}, Δ e_{i})$	Deviation from $Δ c_{i}$ and $Δ e_{i}$ Induced by Parameter Uncertainty $(ϵ_{i, j}, δ_{i, j})$	Optimal Alternative for a Decision Maker Whose Objective Is to Maximize the Population NMB
1	E	An alternative with high variance of cost and effect induced by parameter uncertainty	$(Δ c_{E}, Δ e_{E}) = ($ 25, 000, 0.5)$	$ϵ_{E, j} ~ N (0, 2, 500)$ $δ_{E, j} ~ N (0, 3)$	Alternative E if $ω \geq $ 50, 000$ and the status quo, otherwise
2	F	An alternative with low variance of cost and effect induced by parameter uncertainty	$(Δ c_{F}, Δ e_{F}) = ($ 5, 000, 0.1)$	$ϵ_{F, j} ~ N (0, 500)$ $δ_{F, j} ~ N (0, 0.6)$	Alternative F if $ω \geq $ 50, 000$ and the status quo, otherwise
3	G, H, I	Alternatives with the same cost-effectiveness ratio	$(Δ c_{G}, Δ e_{G}) = ($ 50, 000, 1)$ $(Δ c_{H}, Δ e_{H}) = ($ 100, 000, 2)$ $(Δ c_{I}, Δ e_{I}) = ($ 150, 000, 3)$	$ϵ_{G, j}, ϵ_{H, j}, ϵ_{I, j} ~ N (0, 5, 000)$ $δ_{G, j}, δ_{H, j}, δ_{I, j} ~ N (0, 1)$	Alternative I if $ω > $ 50, 000$ , the status quo if $ω < $ 50, 000$ , and any of 4 alternatives, otherwise
4	J, K	Alternatives with corrected cost and effect	$(Δ c_{J}, Δ e_{J}) = (- $ 1, 000, 0.1)$ $(Δ c_{K}, Δ e_{K}) = (- $ 950, 0.101)$	$ϵ_{J, j} ~ N (0, 200)$ $δ_{J, j} ~ N (0, 0.4)$ $ϵ_{K, j} ~ ϵ_{J, j} + N (0, 50)$ $δ_{K, j} ~ δ_{J, j} + N (0, 0.001)$	Alternative K if $ω \geq $ 50, 000$ and alternative J, otherwise

NMB, net monetary benefit.

As before, we use $(Δ c_{i}, Δ e_{i})$ to denote the expected incremental cost and effect of alternative $i$ with respect to the status quo in the present of parameter uncertainty. Due to uncertainty in parameter values, the cost and effect estimates provided by PA (i.e., $Δ C_{i, j}$ s and $Δ E_{i, j}$ s) may deviate from the true expected incremental cost and effect $(Δ c_{i}, Δ e_{i})$ . To model this deviation, we let $Δ C_{i, j} = Δ c_{i} + ϵ_{i, j}$ and $Δ E_{i, j} = Δ e_{i} + δ_{i, j}$ , where $ϵ_{i, j}$ and $δ_{i, j}$ represent the deviation from expected values $Δ c_{i}$ and $Δ e_{i}$ in the $j$ ^th iteration of PA. We note that the variance of $ϵ_{i, j}$ and $δ_{i, j}$ describe the uncertainty in cost and effect induced by parameter uncertainty. As discussed before, a decision-maker whose objective is to maximize the population NMB must select the alternative with the highest expected NMB (i.e., $ω Δ e_{i} - Δ c_{i}$ ) for the WTP threshold $ω$ . Hence, the variance of $ϵ_{i, j}$ and $δ_{i, j}$ should not impact the preference of a risk-neutral decision-maker over available options.

The first decision problem involves alternative E with the expected incremental cost and effect $(Δ c_{E}, Δ e_{E}) = ($ 25, 000, 0.5)$ . The ICER of this alternative with respect to the status quo (or equivalently, the WTP threshold at which this alternative becomes the optimal choice) is $Δ c_{E} / Δ e_{E} = $ 50, 000$ per effect. Hence, a decision maker with a WTP threshold equal to or greater than $$ 50, 000$ per effect should choose this alternative. In this decision problem, we assume that the uncertainties in the model parameters would generate substantial uncertainty in the estimates of incremental cost and effect for this alternative with $ϵ_{E, j} ~ N (0, 2, 500)$ and $δ_{E, j} ~ N (0, 3)$ . However, a decision maker should choose the alternative with the highest expected NMB, which is alternative E if $ω \geq $ 50, 000$ per effect and the status quo, otherwise.

The second decision problem involves alternative F with expected incremental cost and effect that is $1 / 5$ of those of alternative E: $(Δ c_{F}, Δ e_{F}) = ($ 5, 000, 0.1)$ . Alternative F has the same ICER of alternative E and hence should be chosen if $ω \geq $ 50, 000$ per effect. We assume that the uncertainty induced by parameter uncertainty in this decision problem is also $1 / 5$ of that in the previous decision problem, that is, the standard deviation of $ϵ_{F, j}$ and $δ_{F, j}$ is $1 / 5$ of the standard deviation of $ϵ_{E, j}$ and $δ_{E, j}$ : $ϵ_{F, j} ~ N (0, 500)$ and $δ_{F, j} ~ N (0, 0.6)$ . Therefore, we expect the gain in NMB by resolving uncertainties in this decision problem to be less compared to the previous decision problem. We assess the ability of the methods described above to reveal this relationship.

The third decision problem involves 3 alternatives, G, H, and I, with $Δ c_{I} > Δ c_{H} > Δ c_{G}$ and $Δ e_{I} > Δ e_{H} > Δ e_{G}$ but having the same cost-effectiveness ratio $\frac{Δ c_{G}}{Δ e_{G}} = \frac{Δ c_{H}}{Δ e_{H}} = \frac{Δ c_{I}}{Δ e_{I}} = $ 50, 000$ (Table 1). For a decision maker whose objective is to maximize the population NMB, the status quo is the optimal choice for $ω < $ 50, 000$ , alternative I is the optimal choice for $ω > $ 50, 000$ , and all 4 options (alternatives G, H, and I in additional to the status quo) are equally good for $ω = $ 50, 000$ . In this example, we assume that the variance of deviation from expected cost and effect induced by parameter uncertainty is relatively low. Therefore, the gain in NMB if parameter uncertainty is resolved is minimal. We assess whether the methods described above can identify the optimal option and correctly communicate the value of resolving parameter uncertainty in this example.

In our fourth and final decision problem, we consider the scenario presented by Sadatsafavi et al.²⁰ involving a set of alternatives with correlated costs and effects. In this example, $Δ c_{J} = - 1, 000, Δ c_{K} = - 950$ , and $Δ e_{J} = 0.1, Δ e_{K} = 0.101$ . Clearly, both alternatives result in lower cost and higher effect compared with the status quo. Therefore, the status quo should never be selected, and the optimal option is either alternative J or alternative K—the optimal option is alternative K if $ω > (Δ c_{J} - Δ c_{I}) / (Δ e_{J} - Δ e_{I}) = $ 50, 000$ and is alternative J, otherwise. To induce correlation between the cost and effect of alternatives, Sadatsafavi et al.²⁰ assumed that estimates of cost and effect for alternative K were correlated with the estimates of cost and effect for alternative J through the following: $C_{K, j} = C_{J, j} + χ_{j}$ and $E_{K, j} = E_{J, j} + ϒ_{j}$ for PA iteration $j$ , where $χ_{j}$ and $ϒ_{j}$ follow normal distributions with means $(50, 0.001)$ and standard deviations $(50, 0.001)$ , respectively. We use this decision problem to investigate whether the methods described above can correctly inform the optimal option under correlated cost and effect outcomes.

Results

In the first decision problem, the alternative with the highest incremental expected NMB is alternative E if $ω \geq $ 50, 000$ and the status quo, otherwise (Table 1). However, the CEP for this decision problem (Figure 2A) might create the wrong impression that alternative E is an undesirable option as it is expected to increase the cost (all points lie above the x-axis) and is expected to improve health with about 50% probability (only about half of the points lie to the right of the y-axis). While CEAF in Figure 2B correctly marks alternative E as the optimal alternative for WTP value greater than $$ 50, 000$ per effect, CEACs in Figure 2B create the impression that evidence is insufficient to confidently select the optimal option—CEACs in Figure 2B suggest that alternative E would improve the population’s NMB with only about 50% probability for $ω \geq $ 40, 000$ . As such, Figure 2A and B could mislead the decision maker to believe that alternative E should not be considered given the current level of parameter uncertainties.

Figure 2

Deciding if an alternative with highly uncertain cost and effect is the optimal option. Alternative E has the expected incremental cost and effect of ($25,000, 0.5); hence, it is the optimal choice for willingness-to-pay (WTP) thresholds greater than $50,000 per effect. However, the cost-effectiveness plane (A) and cost-effectiveness acceptability curve (CEAC) (B) could create the wrong impression that this alternative should not be considered given the current level of uncertainty as this alternative is expected to increase the cost and to improve the population health with about 50% probability. Cost-effectiveness acceptability frontier (CEAF), expected loss curves, and net monetary benefit (NMB) lines (B–D) correctly identify the optimal options.

In contrast, both ELCs (Figure 2C) and augmented NMB lines (Figure 2D) focus on the expected NMB under each of the alternatives and hence remove the emphasis that CEPs and CEACs/CEAF placed on the variance of costs and effects. This is important because the optimal option, given the current evidence, does not depend on the variance of cost and effect outcomes if the decision maker is risk neutral and aims to maximize the population NMB. In both C and D of Figure 2, alternative E has the highest estimated incremental NMB compared with the status quo for $ω \geq $ 50, 000$ . Figure 2C and D also highlights the substantial expected gain from resolving the uncertainties in estimates of costs and effects. Therefore, for a decision maker who needs to make an immediate decision based on the currently available evidence, panels C and D in Figure 2 provide an unambiguous recommendation about which alternative to choose. We note that the alternative that is deemed optimal given the current evidence may not be the optimal option when the uncertainty is resolved or reduced. If the decision can be postponed, Figure 2C and D provide an upper bound for the potential gain in the expected NMB if additional research can be conducted to reduce the current level of parameter uncertainty.

Our second decision problem involves alternative F, which is similar to alternative E in the previous decision problem except that the magnitude and the variance of incremental cost and effect is reduced by the factor of 5. The reductions in the incremental cost and effect and their variance are clearly communicated by CEPs (comparing panels A in Figures 2 and 3). However, CEPs and CEACs/CEAF do not provide any information about the expected change in the population expected NMB from implementing alternatives E or F. The smaller variance of incremental cost and effect for alternative F compared with E means a reduction in the expected NMB gain that could be achieved by resolving decision uncertainty. This is clearly communicated by the ELCs and the augmented NMB lines (comparing panels C and D in Figures 2 and 3). While the value of resolving decision uncertainty is substantially different in these decision problems, the CEACs/CEAF are identical for both problems (comparing Figures 2B and 3B). This suggests that CEACs/CEAF do not provide information about the level of decision uncertainty or whether it would be beneficial to collect additional evidence before selecting an option.

Figure 3

Evaluating the ability of methods to present the results of probabilistic analysis to communicate the impact of reduction in parameter uncertainty. Alternative F in this decision problem is similar to alternative E in Figure 2, except that the magnitude and the variance of incremental cost and effect are reduced by a factor of 5 (comparing panel (A) in this figure and Figure 2). Expected loss curves (C) and augmented net monetary benefit (NMB) lines (D) are able to correctly communicate the impact of the reduction in parameter uncertainty on the gain in NMB that could be achieved if parameter uncertainty is resolved (comparing panels C and D in this figure with those in Figure 2). However, the cost-effectiveness acceptability curve (CEAC)/cost-effectiveness acceptability frontier (CEAF) curves are identical for these decision problems (comparing panel (B) in this figure with that in Figure 2). This suggests that that CEACs/CEAF do not provide information about the level of parameter uncertainty. WTP, willingness to pay.

Our third decision problem involved alternatives with the same cost-effectiveness ratio, but only the status quo and alternative I could result in the highest expected NMB (Table 1). The CEP for this decision problem (Figure 4A) creates the wrong impression that these 3 alternatives are equally cost-effective as they all seem to be on the cost-effectiveness frontier with equal ICERs. However, the CEAF, ELCs, and NMB lines correctly identify the optimal alternative (i.e., status quo for $ω < $ 50, 000$ and alternative I for $ω > $ 50, 000$ ). In the vicinity of $ω = $ 50, 000$ , CEACs of all alternatives are about 25%, which might be mistakenly interpreted as the lack of sufficient evidence in identifying the optimal alternative. In contrast, the ELCs and NMB lines in Figure 4C and D correctly indicate that the expected gain from resolving the decision uncertainty in this example is negligible. Hence, the current evidence related to parameter uncertainty allows the decision maker to confidently select the optimal option since waiting to resolve the uncertainty in the future would only minimally improve the expected population NMB.

Figure 4

Choosing among alternatives with the same average cost-effectiveness ratio. Alternatives G, H, and I have the same cost-effectiveness ratio of $50,000, but only alternatives O and I could result in the highest net monetary benefit (NMB) value. The cost-effectiveness acceptability curve (CEAC), expected loss curves, and NMB lines (B–D) correctly identify the optimal option. The cost-effectiveness plane (A) may create the false impression that all alternatives could be desirable as they all appear to lie on the cost-effectiveness frontier; CEACs (B) may create the false impression that around the willingness-to-pay (WTP) threshold of $50,000 per effect, additional research to reduce the uncertainties in cost and effect estimate would be beneficial. Yet, the perfect information curve in (D) indicates that the benefit of eliminating uncertainties in this example is minimal. CEAF, cost-effectiveness acceptability frontier.

In our final decision problem, which involves alternatives with correlated cost and effect outcomes (Table 1), the CEP creates the impression that the uncertainties are too high to select one alternative over the other (Figure 5A). While the CEAF in Figure 5B correctly marks the optimal choice for different WTP thresholds, the CEAC of the status quo, which is a dominated alternative, remains higher than the CEAC of alternatives J and K for a wide range of WTP thresholds (Figure 5B). This again could create the false impression that the status quo could be a viable option or that the evidence is insufficient to identify the optimal alternative in this context. In contrast, ELCs and NMB lines (Figure 5C–D) correctly mark the optimal choice (Figure 5D).

Figure 5

Choosing among alternatives with correlated cost and effects (borrowed from Sadatsafavi et al.²⁰). The expected costs and effects of alternatives J and K are determined such that status quo can never be the optimal choice for any willingness-to-pay value (A). However, cost-effectiveness acceptability curve (CEACs) (B) estimate that the status quo is more likely to be the optimal choice than alternatives J and K for a wide range of willingness-to-pay (WTP) values. CEAF, expected loss curves, and net monetary benefit (NMB) lines (B–D) correctly identify the alternative with the highest expected incremental NMB. CEAF, cost-effectiveness acceptability frontier.

Discussion

Several methods are available to present the results of economic evaluations when the cost and effect of alternatives considered cannot be estimated with certainty, including CEPs, CEACs, CEAF, ELCs, and NMB lines.^{5,7,8,16,21–23} In this article, we propose augmenting NMB lines with value-of-information measures. Here, we focus on presenting the perfect information curve, which measures the potential gain in the expected NMB if uncertainties in parameters are fully resolved. This feature allows us to present the same information provided by ELCs but in a way that may be easier to communicate to decision makers; that is, each line presents the expected incremental NMB under each alternative with respect to the status quo, the optimal option for a given WTP value is the alternative with the highest expected incremental NMB, and the curve of perfect information represents the maximum gain in the expected NMB if uncertainties in cost and effect estimates would be resolved through additional research (e.g., Figure 1D).

NMB lines also communicate the difference in the expected NMB value associated with each alternative. Hence, they allow the decision maker to understand the loss in the expected NMB if instead of the alternative with the highest expected NMB, the second-best alternative is selected, for example, in pursuit of equity or reducing disparity. Presenting NMB lines also mitigates the reliance of decision makers to use ICER to select among alternatives. For many decades, ICER has been the most reported measure to present the results of CEAs. However, ICER has several major limitations such as being easily subject to misinterpretation, inappropriately used in sensitivity analyses, or inappropriately used to rank alternatives.^24,25

For a risk-neutral decision maker, the optimal option is the alternative with the highest expected incremental NMB given the current uncertainty in parameters. ELCs and augmented NMB lines could clearly communicate the option with the highest expected NMB. However, since the shape of a CEP and CEACs depends on the variance of cost and effect outcomes, they could create the false impression that evidence is insufficient or the uncertainty in cost and effect estimates is too high to confidently identify the optimal option. For example, in each decision problem presented in Figures 2 and 3, the alternative under consideration has the highest expected incremental NMB value for $ω > $ 50, 000$ ; however, CEPs and CEAC in both figures may cast doubt on the optimality of alternative E and F for $ω > $ 50, 000$ , given the current level of parameter uncertainty. If a decision needs to be made immediately, the alternative with the highest expected NMB should be selected. This alternative may not turn out to be the optimal choice once the decision uncertainty is resolved. Therefore, if making a decision could be postponed, value-of-information measures (including the perfect information curves in Figures 2D and 3D, partial perfect information curves, and sample information curves) could be used to decide whether obtaining additional data to reduce parameter uncertainty and decision uncertainty is warranted.

Our numerical analysis indicates that the shape of CEACs is not influenced by the level of parameter uncertainty. Yet, CEACs could falsely suggest the lack of sufficient evidence to identify the optimal option (this was also suggested in prior studies^11,26). For example, our first and second decision problems resulted in identical CEACs (panels B in Figures 2 and 3), although the value of resolving parameter uncertainty is substantially reduced in the second decision problem, as also verified by ELCs and the perfect information curves in Figures 2 and 3.

CEACs could assign a high probability of being cost-effective for alternatives that are clearly dominated (for example, in decision problems presented in Figures 4 and 5). Decision uncertainty arises when the alternative that has the highest expected NMB given the current evidence may not have the highest NMB once the uncertainties in parameter values are reduced or resolved. Therefore, decision uncertainty is best characterized by value-of-information analysis and not by CEACs (as illustrated here). Presenting metrics estimating the value of reducing/resolving decision uncertainty on NMB lines (such as perfect information curves) allows the decision maker to understand the level of parameter and decision uncertainty and the potential benefit from addressing it.

We note that achieving the value indicated by the perfect information curves may not be feasible in practice, as it requires resolving all parameter uncertainties. To calculate the perfect information curves, we also assumed that resolving uncertainty is costless, which is never the case. Yet, presenting the perfect information curves could be the first step to determine whether further research could be potentially worthwhile and could help the analyst decide whether to proceed with a value-of-information analysis. If EVPI (expressed at a population level) is lower than the cost of conducting any feasible research study to reduce the decision uncertainty by providing additional data to inform the model parameters, then the decision can confidently be made based on the currently available evidence.

Decision makers are usually interested in specific evidence-generation schemes (e.g., reducing the uncertainty about the effectiveness or side effects of the new alternative) rather than eliminating the uncertainty in all model parameters entirely. As discussed under the ‘Methods” section, NMB lines allow the presentation of other value-of-information measures such as expected value of partial perfect information and sample information or net benefit of sample information curves, which are more useful in practice.^4,18 These methods allow for examining the drivers of the decision uncertainty (e.g., treatment effectiveness, cost, safety) and assist the decision maker in making an informed decision about whether resolving or reducing that uncertainty is warranted.

While CEAF, ELCs, and NMB lines can identify the alternative with the highest expected incremental NMB, they do not provide any insight about the budget impact of each alternative or the contribution of cost and effect of each alternative to their expected incremental NMB. Hence, presenting a CEP could still be of value as it provides information about the change in the overall cost expected under each alternative and the uncertainty in the estimates for the expected cost and effect of each alternative given the current level of parameter uncertainty. After excluding the unaffordable alternatives, a decision maker with the objective of maximizing the population NMB should choose the option with the highest expected incremental NMB value.

The methods commonly considered to present the results of PA assume that the decision maker is risk neutral, suggesting that the decision maker’s utility increases linearly in the accumulated cost, effect, or NMB. Although whether a social planner is risk neutral or risk averse has been debated,^13,27 the augment NMB lines could be easily modified to identify the optimal option for a risk-averse decision maker whose preference over available alternatives also depends on the variance of outcomes. One approach is to use a concave function to represent the decision maker’s preference over NMB values; for example, the utility of alternative $i$ could be measured as $u (Δ C_{i}, Δ E_{i}) = 1 - \exp (- γ (ω Δ E_{i} - Δ C_{i}))$ , where $γ > 0$ represents the decision maker’s coefficient of risk aversion.²⁸ Another approach is to adjust the NMB of each alternative such that uncertainties in estimates (or extreme outcomes) are penalized²⁷; for example, alternatives could be evaluated based on $b (C_{i}, E_{i}) = (ω Δ E_{i} - Δ C_{i}) - β Var [ω Δ E_{i} - Δ C_{i}],$ where $β > 0$ is to penalize the variance in NMB estimates induced by parameter uncertainty.²⁷ Nevertheless, the optimal option is the one that maximizes the decision maker’s expected utility, whether it is defined as NMB (Eq. ( $1$ )), a concave function of NMB (e.g., $u (\cdot)$ as defined above), or an NMB function with penalized variance (e.g., $b (\cdot)$ as defined above). Hence, CEACs, which show the probabilities that each alternative has the highest utility, do not provide meaningful information for either a risk-neutral or risk-averse decision maker.

CEAF, ELCs, and augmented NMB lines are guaranteed to identify the optimal alternative for a given WTP value if the estimates for cost and effect of the alternative are unbiased (i.e., they are representative of the cost and effect of the alternative if implemented in the target population) and the expected cost and effect of each alternative are calculated using enough number of parameter samples. If not enough parameter samples are obtained, the recommendation made by these approaches could substantially change if a new set of parameter samples is used. To ensure the robustness of conclusions with respect to the set of parameter samples used to produce CEAF, ELCs, and NMB lines, NMB lines can display the confidence region along each NMB lines (e.g., in Figure 1D). Increasing the number of parameter samples to minimize the confidence regions along NMB lines could ensure the conclusions are not sensitive to the selected set of sampled parameters. We note that incremental NMB lines are calculated with respect to the status quo; hence, although the confidence intervals along these lines account for the correlation between the outcomes of each alternative and the status quo (Eq. ( $6$ )), they do not reflect the correlation among the outcomes of a given alternative and the other alternatives under consideration. Therefore, confidence intervals along incremental NMB lines should be interpreted with caution. To illustrate, consider the decision problem presented in Figure 1. If the decision maker’s WTP value is close to where the incremental NMB lines for alternatives A and D cross, then it would be helpful to create the NMB plot for only these 2 alternatives, with A being treated as the status quo. That way, the confidence interval along the incremental NMB line of alternatives D would also account for the correlation between the outcomes of alternatives A and D and could guide decisions on whether enough parameter samples are obtained to compare the NMB of these 2 alternatives.

Although we assumed that the decision maker’s objective is to maximize the population NMB, our conclusions would not change when the objective is to maximize the population net health benefit (NHB),²⁹ which is defined as $e_{i} - c_{i} / ω$ for alternative $i$ (we note that the alternative with the highest NMB value is the same as the alternative with the highest NHB value). However, since NHB would become numerically unstable for small values of $ω$ , we chose to use the NMB framework.

For many health systems, robust empirical estimates for the values of WTP thresholds might be available or become available in the future.^30–34 If the WTP threshold can be estimated with high level of accuracy, the need for methods such as CEACs/CEAF, ELCs, and NMB lines, which present the optimal option for a range of WTP thresholds, would be minimal. However, since estimates for WTP thresholds are also subject to uncertainty, these methods are expected to be used to present the results of CEAs for the foreseeable future.

In conclusion, our work supports the use of augmented NMB lines as one of the main methods to present the results of economic evaluations under uncertain cost and effect estimates. Like ELCs and CEAF, augmented NMB lines could identify the optimal alternative if the expected cost and effect of alternatives are estimated accurately in the presence of uncertainty using a sufficiently large number of parameter samples. Furthermore, like ELCs, this approach provides information about the value of additional research to resolve decision uncertainty. Augmented NMB lines, however, present the results of PA in a way that may be easier to communicate to decision makers who wish to identify the alternative that maximizes the population NMB.

Footnotes

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Reza Yaesoubi

Natalia Kunst

References

Meltzer

Addressing uncertainty in medical cost-effectiveness analysis implications of expected utility maximization for methods to perform sensitivity analysis and the use of cost-effectiveness analysis to set priorities for medical research. J Health Econ. 2001;20(1):109–29.

Claxton

Exploring uncertainty in cost-effectiveness analysis. Pharmacoeconomics. 2008;26(9):781–98.

Briggs

Weinstein

Fenwick

, et al. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6. Med Decis Making. 2012;32(5):722–32.

Fenwick

Steuten

Knies

, et al. Value of information analysis for research decisions-an introduction: report 1 of the ISPOR value of information analysis emerging good practices task force. Value Health. 2020;23(2):139–50.

Alarid-Escudero

Enns

Kuntz

Michaud

Jalal

“Time traveling is just too dangerous” but some methods are worth revisiting: the advantages of expected loss curves over cost-effectiveness acceptability curves and frontier. Value Health. 2019;22(5):611–8.

Barton

Briggs

Fenwick

EA.

Optimal cost-effectiveness decisions: the role of the cost-effectiveness acceptability curve (CEAC), the cost-effectiveness acceptability frontier (CEAF), and the expected value of perfection information (EVPI). Value Health. 2008;11(5):886–97.

Claxton

The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. J Health Econ. 1999;18(3):341–64.

Bilcke

Beutels

Generating, presenting, and interpreting cost-effectiveness results in the context of uncertainty: a tutorial for deeper knowledge and better practice. Med Decis Making. 2022;42(4):421–35.

Coyle

Who’s better not best: appropriate probabilistic uncertainty analysis. Int J Technol Assess Health Care. 2003;19(3):540–5.

10.

van Hout

Gordon

Rutten

FF.

Costs, effects and C/E-ratios alongside a clinical trial. Health Econ. 1994;3(5):309–19.

11.

Groot Koerkamp

Hunink

Stijnen

Hammitt

Kuntz

Weinstein

MC.

Limitations of acceptability curves for presenting uncertainty in cost-effectiveness analysis. Med Decis Making. 2007;27(2):101–11.

12.

Briggs

Goeree

Blackhouse

O’Brien

BJ.

Probabilistic analysis of cost-effectiveness models: choosing between treatment strategies for gastroesophageal reflux disease. Med Decis Making. 2002;22(4):290–308.

13.

Eckermann

Willan

AR.

Presenting evidence and summary measures to best inform societal decisions when comparing multiple strategies. Pharmacoeconomics. 2011;29(7):563–77.

14.

Johannesson

Weinstein

MC.

On the decision rules of cost-effectiveness analysis. J Health Econ. 1993;12(4):459–67.

15.

Suen

Goldhaber-Fiebert

JD.

An efficient, noniterative method of identifying the cost-effectiveness frontier. Med Decis Making. 2016;36(1):132–6.

16.

Fenwick

O’Brien

Briggs

Cost-effectiveness acceptability curves—facts, fallacies and frequently asked questions. Health Econ. 2004;13(5):405–15.

17.

Eckermann

Karnon

Willan

AR.

The value of value of information: best informing research design and prioritization using current methods. Pharmacoeconomics. 2010;28(9):699–709.

18.

Rothery

Strong

Koffijberg

, et al. Value of information analytical methods: Report 2 of the ISPOR value of information analysis emerging good practices task force. Value Health. 2020;23(3):277–86.

19.

Heath

Kunst

Jackson

Value of Information for Healthcare Decision-Making. Boca Raton (FL): CRC Press; 2024.

20.

Sadatsafavi

Najafzadeh

Marra

Technical note: acceptability curves could be misleading when correlated strategies are compared. Med Decis Making. 2008;28(3):306–7.

21.

Wolff

Qendri

Kunst

Alarid-Escudero

Coupe

VMH

. Methods for communicating the impact of parameter uncertainty in a multiple-strategies cost-effectiveness comparison. Med Decis Making. 2022;42(7):956–68.

22.

Fenwick

Claxton

Sculpher

Representing uncertainty: the role of cost-effectiveness acceptability curves. Health Econ. 2001;10(8):779–87.

23.

Black

WC.

The CE plane: a graphic representation of cost-effectiveness. Med Decis Making. 1990;10(3):212–4.

24.

Paulden

Why it’s time to abandon the ICER. Pharmacoeconomics. 2020;38(8):781–4.

25.

Paulden

Calculating and interpreting ICERs and net benefit. Pharmacoeconomics. 2020;38(8):785–807.

26.

Spieker

Illenberger

Roy

Mitra

Net benefit separation and the determination curve: a probabilistic framework for cost-effectiveness estimation. Stat Methods Med Res. 2021;30(5):1306–19.

27.

Zivin

JG.

Cost-effectiveness analysis with risk aversion. Health Econ. 2001;10(6):499–508.

28.

Elbasha

EH.

Risk aversion and uncertainty in cost-effectiveness analysis: the expected-utility, moment-generating function approach. Health Econ. 2005;14(5):457–70.

29.

Stinnett

Mullahy

Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analysis. Med Decis Making. 1998;18(2 suppl):S68–80.

30.

Kazibwe

Gheorghe

Wilson

Ruiz

Chalkidou

Chi

YL.

The use of cost-effectiveness thresholds for evaluating health interventions in low- and middle-income countries from 2015 to 2020: a review. Value Health. 2022;25(3):385–9.

31.

Chi

Blecher

Chalkidou

, et al. What next after GDP-based cost-effectiveness thresholds? Gates Open Res. 2020;4:176.

32.

Leech

Kim

Cohen

Neumann

PJ.

Use and misuse of cost-effectiveness analysis thresholds in low- and middle-income countries: trends in cost-per-DALY studies. Value Health. 2018;21(7):759–61.

33.

Thokala

Ochalek

Leech

Tong

Cost-effectiveness thresholds: the past, the present and the future. Pharmacoeconomics. 2018;36(5):509–22.

34.

Woods

Revill

Sculpher

Claxton

Country-level cost-effectiveness thresholds: initial estimates and the need for further research. Value Health. 2016;19(8):929–35.