Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

Abstract

Background

The expected value of sample information (EVSI) calculates the value of collecting additional information through a research study with a given design. However, standard EVSI analyses do not account for the slow and often incomplete implementation of the treatment recommendations that follow research. Thus, standard EVSI analyses do not correctly capture the value of the study. Previous research has developed measures to calculate the research value while adjusting for implementation challenges, but estimating these measures is a challenge.

Methods

Based on a method that assumes the implementation level is related to the strength of evidence in favor of the treatment, 2 implementation-adjusted EVSI calculation methods are developed. These novel methods circumvent the need for analytical calculations, which were restricted to settings in which normality could be assumed. The first method developed in this article uses computationally demanding nested simulations, based on the definition of the implementation-adjusted EVSI. The second method is based on adapting the moment matching method, a recently developed efficient EVSI computation method, to adjust for imperfect implementation. The implementation-adjusted EVSI is then calculated with the 2 methods across 3 examples.

Results

The maximum difference between the 2 methods is at most 6% in all examples. The efficient computation method is between 6 and 60 times faster than the nested simulation method in this case study and could be used in practice.

Conclusions

This article permits the calculation of an implementation-adjusted EVSI using realistic assumptions. The efficient estimation method is accurate and can estimate the implementation-adjusted EVSI in practice. By adapting standard EVSI estimation methods, adjustments for imperfect implementation can be made with the same computational cost as a standard EVSI analysis.

Highlights

• Standard expected value of sample information (EVSI) analyses do not account for the fact that treatment implementation following research is often slow and incomplete, meaning they incorrectly capture the value of the study.

• Two methods, based on nested Monte Carlo sampling and the moment matching EVSI calculation method, are developed to adjust EVSI calculations for imperfect implementation when the speed and level of the implementation of a new treatment depends on the strength of evidence in favor of the treatment.

• The 2 methods we develop provide similar estimates for the implementation-adjusted EVSI.

• Our methods extend current EVSI calculation algorithms and thus require limited additional computational complexity.

The expected value of sample information (EVSI) can be a tool for research prioritization and trial design as it calculates the value of collecting additional information through a proposed study with a specific design.^1,2 When coupled with a health economic decision model, ³ EVSI calculates the value of reducing the statistical uncertainty in the parameters underlying this model before making a decision. The information collected in a study has value if it indicates that the current optimal treatment is, in fact, nonoptimal. This is because the information has prevented decision makers from implementing the incorrect treatment and thereby incurring an opportunity loss. ⁴

EVSI can prioritize studies with the highest expected net economic benefit by computing EVSI for a range of proposed studies and subtracting the study costs. ⁵ This prioritization process requires an estimate of the population-level EVSI from which the study costs are subtracted to compute the expected net benefit of sampling (ENBS). ⁶ Population-level EVSI is usually estimated by multiplying the individual-level EVSI, the output of standard calculations, by the number of people who would be affected by the decision in a given year and the decision horizon. ⁶ This decision horizon is defined as the length of time before the decision will be reassessed (i.e., due to the development of a new treatment option). ⁷

This estimation of the population-level EVSI assumes that any treatment recommended following the study is implemented for all future patients.^8,9 In practice, this assumption is unrealistic, as treatment recommendations are often slow to diffuse into clinical practice. ¹⁰ Thus, standard estimates of the population-level EVSI will result in biased ENBS estimates, although the direction of this bias is dependant on the underlying decision model and the definition of the counterfactual.^9,11 Thus, ENBS can be estimated accurately only if the population-level EVSI is adjusted for realistic expectations about the implementation of the recommended treatment, following study completion. ¹²

Several frameworks consider the interplay between the value of information and imperfect implementation,^8–10,12 although Fenwick et al. ¹² considered only the value of perfect, rather than study-specific, information. Andronis and Barton ⁹ extended the Fenwick et al. framework to define an implementation-adjusted EVSI but facilitated their calculations by making the unrealistic assumption that the speed of adoption and the saturation level of the most cost-effective treatment is not related to the future data. In contrast, Willan and Eckermann ⁸ suggested that the implementation dynamics depend on the probability that a given treatment is cost effective; that is, treatments with a higher probability of cost-effectiveness achieve a higher saturation level. ⁸ Finally, Grimm et al. ¹⁰ split the value of a research study into 2 components that compute the research’s impact on implementation and information separately.

Willan and Eckermann ⁸ computed the implementation-adjusted EVSI by assuming that both the value of each treatment and the data collected in the study follow a normal distribution, allowing for analytical results. Elsewhere, Grimm et al. ¹⁰ did not present an algorithm to estimate the value of the research’s impact on implementation. Thus, the value of a research study, adjusting for imperfect implementation, can currently be achieved only in a small number of models that meet these normality assumptions. To overcome the practical challenge of adjusted analyses, we develop 2 algorithms to estimate the implementation-adjusted EVSI, irrespective of the underlying model structure and study design.

First, we define EVSI and how it can be adjusted for imperfect implementation. ⁸ A general purpose nested simulation algorithm is then adjusted to estimate the implementation-adjusted EVSI. ⁴ This algorithm can be applied irrespective of the model complexity and the data-generation process. However, as is the case with nonadjusted EVSI estimation, ¹³ this method is computationally intensive, acting as a significant barrier for the proposed analyses.

Recent computation methods have been developed to efficiently compute individual-level unadjusted EVSI irrespective of the structural form of the underlying health economic decision model and study design.^14–19 However, these methods cannot directly estimate the implementation-adjusted EVSI, as the probability that a given intervention is cost-effective cannot be estimated. Thus, one of these methods, known as the moment matching method,^18,19 is adapted so this probability can be estimated. From this, the implementation-adjusted EVSI can then be computed. This novel method allows efficient estimation of the implementation-adjusted EVSI, based on realistic model structures and trial designs.

Following the development of these 2 methods, the implementation-adjusted EVSI is estimated for 3 proposed studies based on a previously published example. The computationally intensive nested simulation method and the adapted efficient computation method give similar estimates of the implementation-adjusted EVSI. As expected, the efficient computation method is significantly faster than the nested simulation method. Importantly, as these 2 methods are adjusted from currently available computation methods, estimating the implementation-adjusted EVSI does not increase the computational complexity of the EVSI analysis. Thus, the implementation-adjusted EVSI can be easily estimated, and the implementation-adjusted net value of research can be used to determine the optimal study design for future data collection, irrespective of model structure.

EVSI

Health economic decision models estimate the costs and benefits of different treatment options to help decision makers select the optimal treatment from $D$ potential alternatives. These models are based on a set of model parameters $\mathbf{\theta }$ that represent real-world quantities (e.g., prevalence, quality of life weights, relative effects, and treatment costs). The statistical uncertainty in the estimates of these parameters is usually characterized through a joint probability distribution $p(\mathbf{\theta })$ in a process known as probabilistic analysis (or probabilistic sensitivity analysis). The costs and benefits estimated from a probabilistic health economic decision model can be combined to measure the net benefit of a given treatment (measured in monetary or health units), denoted ${\mathrm{NB}}_d(\mathbf{\theta })$ , $d=1,\dots ,D$ . ²⁰ Given the current evidence about the parameters, defined in $p(\mathbf{\theta })$ , the best treatment is the one that maximizes the expected net benefit, $d^{*}=\arg \underset{d}{max}{\mathrm{E}}_{\mathbf{\theta }}[{\mathrm{NB}}_d(\mathbf{\theta })]$ . ²¹ The net benefit measures the average benefit across the whole population, implying that ${\mathrm{NB}}_d(\mathbf{\theta })$ , $d=1,\dots ,D$ , would be known if all parameter uncertainty could be resolved. ²²

EVSI calculates the value of collecting additional information about the model parameters $\mathbf{\theta }$ to improve decision making. Assume that this information is collected through a proposed research study that aims to collect data $\mathit{X}$ . ²³ If the data $\mathit{X}$ were collected and realized to a specific data set $\mathit{x}$ , then it would be combined with the current evidence to update the distribution of $\mathbf{\theta }$ , $p(\mathbf{\theta }|\mathit{x})$ . This updated distribution for $\mathbf{\theta }$ would change the distribution for the net benefits ${\mathrm{NB}}_d(\mathbf{\theta })$ , $d=1,\dots ,D$ . The (potentially new) optimal treatment would again be found by taking the expectation of the net benefit $\underset{d}{max}{E}_{\mathbf{\theta }|\mathit{x}}[{\mathrm{NB}}_d(\mathbf{\theta })]$ . From this, the value of observing $\mathit{x}$ is

$$\underset{d}{max}{\mathrm{E}}_{\mathbf{\theta }|\mathit{x}}[{\mathrm{NB}}_d(\mathbf{\theta })]-\underset{d}{max}{E}_{\mathbf{\theta }}[{\mathrm{NB}}_d(\mathbf{\theta })].$$

(1)

However, as the data have not been collected, the EVSI is calculated by taking an expectation over all possible studies ²³ :

$$\mathrm{EVSI}={\mathrm{E}}_{\mathit{X}}\underset{d}{max}{\mathrm{E}}_{\mathbf{\theta }|\mathit{X}}[{\mathrm{NB}}_d(\mathbf{\theta })]-\underset{d}{max}{\mathrm{E}}_{\mathbf{\theta }}[{\mathrm{NB}}_d(\mathbf{\theta })].$$

(2)

The distribution of all potential data sets $\mathit{X}$ is defined through the joint distribution

$$p(\mathit{X},\mathbf{\theta })=p(\mathit{X}|\mathbf{\theta })p(\mathbf{\theta }),$$

(3)

where $p(\mathit{X}|\mathbf{\theta })$ is the sampling distribution of the data conditional on the parameters and $p(\mathbf{\theta })$ is the current distribution of the model parameters, defined for the probabilistic analysis.

Adjusting for Imperfect Implementation

Willan and Eckermann ⁸ defined the implementation-adjusted EVSI ( ${\mathrm{EVSI}}^{\mathrm{IM}}$ ) as the difference between the expected opportunity loss under the current decision and the expected opportunity loss of the decision made with additional information. The supplementary material demonstrates that this definition can be expressed as the difference between the “value of the current decision” and the “expected value of the decision made with additional information” to more closely mimic the standard EVSI definition. The value of the current decision is equal to

$$\mathcal{C}=\sum _{d=1}^D{m}_d{\mathrm{E}}_{\mathbf{\theta }}[{\mathrm{NB}}_d(\mathbf{\theta })],$$

(4)

where $m_d$ is the current market share of the $d$ th intervention. The phrase market share indicates the proportion of eligible patients who receive intervention $d$ , even in settings where there is not a traditional market (e.g., for a public health intervention that is rolled out over time).

The value of the decision made after a specific additional data set $\mathit{x}$ has been collected is

$${\mathcal{F}}^{\mathit{x}}=\sum _{d=1}^D{m}_d(\mathit{x}){\mathrm{E}}_{\mathbf{\theta }|\mathit{x}}[{\mathrm{NB}}_d(\mathbf{\theta })],$$

(5)

where $m_d(\mathit{x})$ is the market share of the $d$ th intervention, conditional on the observed data $\mathit{x}$ . The value $\mathit{x}$ is then the difference between $\mathcal{C}$ and ${\mathcal{F}}^{\mathit{x}}$ . However, the data have not been observed, and so ${\mathrm{EVSI}}^{\mathrm{IM}}$ is defined by taking the expectation of ${\mathcal{F}}^{\mathit{x}}$ over all possible data sets $\mathit{X}$

$${\mathrm{EVSI}}^{\mathrm{IM}}={\mathrm{E}}_{\mathit{X}}\left[{\mathcal{F}}^{\mathit{X}}\right]-\mathcal{C}={\mathrm{E}}_{\mathit{X}}[\sum _{d=1}^D{m}_d(\mathit{X}){\mathrm{E}}_{\mathbf{\theta }|\mathit{X}}[{\mathrm{NB}}_d(\mathbf{\theta })]]-\mathcal{C}.$$

(6)

The value of the current decision, $\mathcal{C}$ , is unlikely to be equal to the value of the optimal treatment, because adjustments must be made for current issues in implementation. ¹¹ Note that ${\mathrm{EVSI}}^{\mathrm{IM}}$ is estimated at the individual level, with the population-level ${\mathrm{EVSI}}^{\mathrm{IM}}$ calculated by multiplying by the number of people affected yearly by the decision and the decision horizon, as in standard EVSI analyses. ⁶

${\mathrm{EVSI}}^{\mathrm{IM}}$ , defined in Eq. 6, is equal to the expected value of a specific implementation measure, defined by Grimm et al. ¹⁰ in Appendix Equation A.3. Grimm et al. used this wording as ${\mathrm{EVSI}}^{\mathrm{IM}}$ calculates the expected value the decision maker would gain by changing the implementation of the treatments based on the sample information. Thus, it is unclear whether ${\mathrm{EVSI}}^{\mathrm{IM}}$ should be considered a value of information measure or a value of implementation measure. In settings in which the collected data do not affect the implementation levels, that is, where $m_d(\mathit{X})=m_d$ for all $\mathit{X}$ , then ${\mathrm{EVSI}}^{\mathrm{IM}}$ computes the value of improving implementation. However, provided the future market share is related to the collected data, ${\mathrm{EVSI}}^{\mathrm{IM}}$ computes the value of collecting information to improve implementation. In this case, information has positive value when it increases the market share of the most cost-effective treatment, based on the available evidence.

Defining the Sample Specific Market Share

All EVSI calculation methods estimate ${\mu }^{\mathit{X}}=E_{\mathbf{\theta }|\mathit{X}}[{\mathrm{NB}}_d(\mathbf{\theta })]$ , across the distribution of plausible data sets $p(\mathit{X})$ . ²⁴ Thus, ${\mathrm{EVSI}}^{\mathrm{IM}}$ is calculated by estimating the sample-specific market share $m_d(\mathit{X})$ . Willan and Eckermann defined $m_d(\mathit{X})$ , $d=1,\dots ,D$ , as a function of the probability that a given intervention is the most cost-effective, $p_d(\mathit{X})$ , ⁸

$$m_d(\mathit{X})=f_d^m(p_d(\mathit{X})),$$

(7)

with the functional form of $f_d^m(·)$ dependent on the decision problem. For example, if clinical practice is reticent to move away from current practice ( $d=1$ ), then a higher probability of cost-effectiveness would be required to implement the novel treatment ( $d=2$ ). Thus, to calculate ${\mathrm{EVSI}}^{\mathrm{IM}}$ , $p_d(\mathit{X})$ must be estimated.

Estimating the Implementation-Adjusted EVSI

Nested Monte Carlo Simulation

Unless restrictive assumptions are made, EVSI is estimated using simulation-based methods. ²⁴ The first general purpose simulation method for estimating EVSI is based on a nested simulation procedure ⁴ ; first, $S$ plausible data sets ${\mathit{X}}_s$ , $s=1,\dots ,S$ are simulated from $p(\mathit{X})$ . Simulations from the marginal distribution of $\mathit{X}$ can be obtained by simulating a parameter set ${\mathbf{\theta }}_s$ and then simulating a data set from $p(\mathit{X}|{\mathbf{\theta }}_s)$ for, $s=1,\dots ,S$ . ²³ Following this, $R$ simulations from $p(\mathbf{\theta }|{\mathit{X}}_s)$ are required for each $s=1,\dots ,S$ , denoted ${\mathbf{\theta }}_{r,s}$ . The net benefit for each treatment must be computed for each simulated parameter set ${\mathbf{\theta }}_{r,s}$ , $r=1,\dots ,R$ , $s=1,\dots ,S$ , requiring $R\times S$ evaluations of each net benefit function. The expected net benefit for treatment $d=1,\dots ,D$ , conditional on the sample ${\mathit{X}}_s$ is estimated by calculating the sample average net benefit

$$\frac{1}{R}\sum _{r=1}^R{\mathrm{NB}}_d\left({\mathbf{\theta }}_{r,s}\right).$$

(8)

The EVSI is then approximated by

$$\widehat{\mathrm{EVSI}}=\frac{1}{S}\sum _{s=1}^S\underset{d}{\max }\left\{\frac{1}{R}\sum _{r=1}^R\mathrm{NB}{}_d\left({\theta }_{r,s}\right)\right\}-\underset{d}{\max }\frac{1}{S}\sum _{s=1}^S\frac{1}{R}\sum _{r=1}^{R}NB{}_d\left({\theta }_{r,s}\right).$$

(9)

This algorithm can be adapted to estimate ${\mathrm{EVSI}}^{\mathrm{IM}}$ by estimating the sample-specific probability of cost-effectiveness for treatment $d=1,\dots ,D$ as the proportion of simulations in which treatment $d$ is optimal;

$$\widehat{p_d({\mathit{X}}_s)}=\frac{1}{R}\sum _{r=1}^{R}1[{\mathrm{NB}}_d\left({\mathbf{\theta }}_{r,s}\right)=\underset{d}{max}{\mathrm{NB}}_d\left({\mathbf{\theta }}_{r,s}\right)],$$

(10)

where $1[·]$ equals 1 if the condition is true and 0 otherwise. From this, the market share for each treatment can be estimated $\widehat{m_d({\mathit{X}}_s)}=f_d^m\left(\widehat{p_d({\mathit{X}}_s)}\right)$ , and ${\mathrm{EVSI}}^{\mathrm{IM}}$ ,

$$\widehat{\mathrm{EVSI}{}^{IM}}=\frac{1}{S}\sum _{s=1}^S\sum _{d=1}^D\widehat{m_d(X_s)}\left\{\frac{1}{R}\sum _{r=1}^R\mathrm{NB}{}_d\left({\theta }_{r,s}\right)\right\}-\sum _{d=1}^D{m}_d\frac{1}{S}\sum _{s=1}^S\frac{1}{R}\sum _{r=1}^R\mathrm{NB}{}_d\left({\theta }_{r,s}\right)$$

(11)

However, this algorithm is computationally intensive, as it requires $S\times R$ evaluations of the $D$ net benefit functions. Thus, in practical health economic decision models in which the net benefit function is nontrivial to compute, this method cannot compute ${\mathrm{EVSI}}^{\mathrm{IM}}$ within a feasible time frame.

The Moment Matching Method

The moment matching method is an efficient nested simulation method for EVSI that reduces the number of data sets required to compute EVSI from $S$ , usually at least 1000, to $Q$ , which is usually between 30 and 50. ²⁵ The standard moment matching method approximates ${\mu }_d(\mathit{X})=E_{\mathbf{\theta }|\mathit{X}}[{\mathrm{NB}}_d(\mathbf{\theta })]$ by reducing the variance of simulated values that derive from a function of the net benefit. ²⁵ The required function of ${\mathrm{NB}}_d(\mathbf{\theta })$ is defined by noting that the sampling distribution of $\mathit{X}$ is typically dependent on a subset of the model parameters $\mathbf{\varphi }\subset \mathbf{\theta }$ . The moment matching method then rescales the conditional expectation of the net benefit, conditional on $\mathbf{\varphi }$ , ${\mathrm{E}}_{\mathbf{\theta }|\mathbf{\varphi }}[{\mathrm{NB}}_d(\mathbf{\theta })]$ . In general, ${\mathrm{E}}_{\mathbf{\theta }|\mathbf{\varphi }}[{\mathrm{NB}}_d(\mathbf{\theta })]$ can be estimated by fitting a nonparametric regression between the simulated values of $\mathbf{\varphi }$ and the simulated net benefit values that were calculated with the specific value of $\mathbf{\varphi }$ and extracting the fitted values from this regression. ²⁶ If the sampling distribution of the data is defined using all the model parameters, then $\mathbf{\varphi }=\mathbf{\theta }$ , and EVSI can then be approximated by rescaling ${\mathrm{NB}}_d(\mathbf{\theta })$ directly.

To determine the variance reduction factor, $Q$ data sets are simulated from the distribution of plausible data sets, ${\mathit{X}}_q$ , $q=1,\dots ,Q$ . ²³ For each of these $Q$ data sets, $R$ values are simulated from $p(\mathbf{\theta }|{\mathit{X}}_q)$ , ${\mathbf{\theta }}_{r,q}$ and compute the net benefit for each treatment option, ${\mathrm{NB}}_d({\mathbf{\theta }}_{r,q})$ . For each simulated data set, the sample variance of the net benefits is calculated, before computing the average variance across these $Q$ estimates. Finally, the simulated values for ${\mathrm{E}}_{\mathbf{\theta }|\mathbf{\varphi }}[{\mathrm{NB}}_d(\mathbf{\theta })]$ are rescaled so their variance is equal to the difference between the variance of the initial net benefit simulations and the average variance from the nested simulations. ²⁵ This rescaling provides $S$ simulations ${\mu }_d({\mathit{X}}_s)$ , $s=1,\dots ,S$ , that approximate ${\mu }_d(\mathit{X})$ for each decision option $d=1,\dots ,D$ . To accurately estimate the EVSI, the $Q$ data sets should be generated so they cover the complete range of possible data sets. This can be achieved by extracting the $Q$ quantiles from the simulated values of $\mathbf{\varphi }$ from the probailistic analysis. A separate data set is then generated for each quantile. ²⁵ Functions in R are provided in the supplementary material to estimate ${\mathrm{E}}_{\mathbf{\theta }|\mathbf{\varphi }}[{\mathrm{NB}}_d(\mathbf{\theta })]$ and specify the appropriate values of $\mathbf{\varphi }$ to generate the required data sets.

Adjusting the Moment Matching Method for Imperfect Implementation

As the standard moment matching procedure estimates only the distribution of the mean net benefit, conditional on the future data, it cannot estimate the probability of cost-effectiveness $p_d(\mathit{X})$ , which depends on the full distribution of the net benefit. Thus, currently the moment matching method can estimate only $p_d(\mathit{X})$ for the $Q$ data sets used in the nested simulation procedure and is now extended to compute $p_d(\mathit{X})$ for all plausible data sets.

The standard moment matching method produces simulations of ${\mu }_d({\mathit{X}}_s)$ , $s=1,\dots ,S$ . In general, $p_d(\mathit{X})$ is related to the value of ${\mu }_d(\mathit{X})$ , as the larger the expected net benefit, the more likely the treatment is to be cost-effective. Therefore, this extension for the moment matching method aims to estimate the probability of cost-effectiveness, $p_d(\mathit{X})$ , as a function of ${\mu }_d(\mathit{X})$ . To achieve this, $p_d(\mathit{X})$ is estimated for each of the $q=1,\dots ,Q$ data sets, ${\mathit{X}}_q$ , from the nested simulations using Eq. 10, with ${\mu }_d({\mathit{X}}_q)$ estimated from Eq. 8. From these estimates, nonlinear regression is used to approximate the function $h_d(·)$ ;

$$p_d({\mathit{X}}_q)=h_d({\mu }_d({\mathit{X}}_q))+{\epsilon }_q,$$

(12)

where ${\epsilon }_q~N(0,{\sigma }^2)$ is the error due to estimating $p_d({\mathit{X}}_q)$ by simulation.

The functional form for $h_d(·)$ is selected by noting that 1) $p_d(\mathit{X})$ is a probability and thus constrained between 0 and 1; 2) as ${\mu }_d(\mathit{X})$ increases, $p_d(\mathit{X})$ also increases as the treatment is becoming more valuable; and 3) $p_d(\mathit{X})$ increases smoothly as ${\mu }_d(\mathit{X})$ increases, that is, if the expected net benefits are similar than the probability of cost-effectiveness will also be similar. The generalized logistic function is a flexible function that exhibits these 3 features, ²⁷ so

$$h_d({\mu }_d(\mathit{X}))={(A+e^{-B{\mu }_d(\mathit{X})})}^{-v},$$

(13)

where $A$ , $B$ , and $v$ are from the data $p_d({\mathit{X}}_q)$ and ${\mu }_d({\mathit{X}}_q)$ for $q=1,\dots ,Q$ . $A$ , $B$ , and $v$ can be estimated using either Bayesian or frequentist methods. However, maximum likelihood methods can struggle to converge in some settings, while all 3 model parameters can be estimated in a Bayesian framework using weakly informative priors to improve convergence. These priors are discussed, along with model code, in the supplementary material.

Once estimates have been obtained for $A$ , $B$ , and $v$ , the regression model can calculate the fitted values of $p_d(\mathit{X})$ for each of the simulations for ${\mu }_d({\mathit{X}}_s)$ , $s=1,\dots ,S$ , denoted $p_d({\mathit{X}}_s)$ . The market share of each treatment can then be computed using $m_d({\mathit{X}}_s)=f_d^m(p_d({\mathit{X}}_s)$ ). From this, ${\mathrm{EVSI}}^{\mathrm{IM}}$ can be estimated using the moment matching method as

$$\widehat{{\mathrm{EVSI}}^{\mathrm{IM}}}=\frac{1}{S}\sum _{s=1}^S\sum _{d=1}^D{m}_d({\mathit{X}}_s){\mu }_d({\mathit{X}}_s)-\sum _{d=1}^D{m}_d\frac{1}{S}\sum _{s=1}^S{\mu }_d({\mathit{X}}_s).$$

(14)

Estimating Implementation-Adjusted EVSI across Sample Size

The moment matching method can be extended to estimate EVSI across a range of sample sizes of the proposed study ( $N_{min},N_{max})$ . ¹⁹ This is achieved by creating a sequence of sample sizes, $N_q$ , $q=1,\dots ,Q$ , between $N_{min}$ and $N_{max}$ . Each simulated data set is then generated with a different sample size, that is, the data set ${\mathit{X}}_q$ contains data from $N_q$ simulated individuals. The variance reduction factor for a given sample size $n$ is then found using nonlinear regression. ¹⁹ Specifically, a nonlinear regression is fit between the posterior variance of the net benefit conditional on the sample ${\mathit{X}}_q$ and the sample size $N_q$ . The variance reduction factor for the sample size $n$ is then estimated by calculating the fitted value from this regression for $n$ . The Moment matching estimation method for ${\mathrm{EVSI}}^{\mathrm{IM}}$ is now extended so it can be used across sample size.

To achieve this, the method proceeds, similar to before, by first calculating $p_d({\mathit{X}}_q)$ and ${\mu }_d({\mathit{X}}_q)$ for each of the nested simulations. As each of these pairs has a different sample size, the regression equation is adapted so $h_d(·)$ is also a function of the sample size $N_q$ ,

$$p_d({\mathit{X}}_q)=h_d^N({\mu }_d({\mathit{X}}_q),N_q)+{\epsilon }_q,$$

(15)

where ${\epsilon }_q~N(0,{\sigma }^2)$ . In this setting, $h_d^N(·)$ can still be represented by a generalized logistic function but is adjusted to account for the fact that the larger the sample size, the faster the probability of cost-effectiveness will increase from 0 and 1,

$$h_N({\mu }_d({\mathit{X}}_q),N_q)={(A+e^{-{\mathrm{BN}}_q^u{\mu }_d({\mathit{X}}_q)})}^{-v},$$

(16)

where $u$ is an additional parameter defining the rate at which the probability of cost-effectiveness increases due to the sample size. The 4 parameters in this model, $A$ , $B$ , $u$ , and $v$ , can be estimated in a Bayesian or frequentist framework, with the best performance seen using Bayesian methods with weakly informative priors.

Once these parameters have been estimated, ${\mathrm{EVSI}}^{\mathrm{IM}}$ for a specific sample size $n$ can be estimated. To achieve this, the values for ${\mu }_d(\mathit{X})$ for the sample size $n$ are estimated using the moment matching method. The probability of cost-effectiveness is then estimated by computing $h_d^N({\mu }_d(\mathit{X}),n)$ . From this, the market share is estimated, and ${\mathrm{EVSI}}^{\mathrm{IM}}$ is calculated from Eq. 14.

Calculating EVSI Adjusted for Imperfect Implementation

This section estimates ${\mathrm{EVSI}}^{\mathrm{IM}}$ for a previously developed health economic model ⁴ to compare the 2 estimation methods and demonstrate their accuracy. The computational efficiency of the augmented moment matching method is also investigated.

A Health Economic Decision Model for Reduced Risk of a Critical Event

The case study is based on a previously developed decision tree model. ⁴ This decision model has 2 interventions, $d=1$ (standard care) and $d=2$ (novel treatment). Individuals are at risk of a critical event that would lead to a reduced quality of life ( $Q_C$ ) for the $L$ remaining years of their life and incur a yearly treatment cost ( $C_C$ ). The novel treatment has a fixed cost ( $C_T$ ) and reduces the probability of experiencing this critical event but with a risk of side effects. These side effects give a short-term reduction in quality of life ( $Q_{\mathrm{SE}})$ and incur a one-off cost ( $C_{\mathrm{SE}}$ ). The model has 4 uncertain parameters, the baseline probability of critical event ( $P_C$ ), the odds ratio of the critical event under the novel treatment ( $\mathrm{OR}$ ), the probability of side effects on treatment ( $P_{\mathrm{SE}})$ , and the quality-of-life detriment due to the critical event. These 4 parameters are modeled using independent probability distributions with the distributions, and the values of the fixed parameters, given in Table 1.

Table 1

Parameter Specification for the Decision Model Adapted from Ades et al. ⁴ and Strong et al. ¹⁴

Description	Parameter	Mean	Distribution
Probability of critical event with no treatment	$P_C$	0.15	Beta(15,85)
Odds ratio of critical event with treatment	$\mathrm{OR}$	0.2636	$\log (\mathrm{OR})~N(-1.5,\frac{1}{3})$
Probability of critical event with treatment	$P_T$	0.0440	$P_T=\frac{P_C\mathrm{OR}}{1-P_c+P_C\mathrm{OR}}$
Probability of side effects on treatment	$P_{\mathrm{SE}}$	0.25	Beta(3,9)
Quality of life after critical event	$Q_C$	0.6405	logit $(Q_C)~N(0.6,\frac{1}{6})$
Remaining years of life	$L$	30	Fixed
Cost of treating critical event	$C_C$	$\$200,000$	Fixed
New treatment cost	$C_T$	$\$15,000$	Fixed
Cost of treating side effects	$C_{\mathrm{SE}}$	$\$100,000$	Fixed
Quality-of-life detriment due to side effects	$Q_{\mathrm{SE}}$	1	Fixed
Willingness to pay for 1 quality-of-life unit	$\lambda$	$\$75,000$	Fixed

The decision tree structure of the model implies that the net benefit is calculated as follows for $d=1$ and $d=2$ , respectively:

$${\mathrm{NB}}_1(\mathbf{\theta })=\lambda (P_{C}L\left(\frac{1+Q_C}{2}\right)+(1-P_C)L)-P_C{C}_C$$

(16)

$$\begin{array}{c}{\mathrm{NB}}_2(\mathbf{\theta })=\lambda (P_T{P}_{\mathrm{SE}}(L\frac{1+Q_C}{2}-Q_{\mathrm{SE}})+P_T(1-P_{\mathrm{SE}})L\frac{1+Q_C}{2}+\hfill \\ (1-P_T)P_{\mathrm{SE}}(L-Q_{\mathrm{SE}})+(1-P_T)(1-P_{\mathrm{SE}})L)-\hfill \\ (C_T+P_T{C}_C+P_{\mathrm{SE}}{C}_{\mathrm{SE}}).\hfill \end{array}$$

(16)

Proposed Future Studies

Three alternative proposed studies are considered. ⁴ The first study aims to reduce uncertainty in the probability of side effects with the new treatment by offering 60 individuals the treatment and observing the number who experience side effects. The data are modeled using a binomial distribution, $X~\mathrm{Bin}(60,p_{\mathrm{SE}})$ . The second study aims to reduce uncertainty in the quality of life after the critical event by recording the quality of life for 100 individuals who have experienced the critical event. The individual-level variation in the logit of quality of life is modeled using a normal distribution with variance 2, $\mathrm{logit}(X)~N(\mathrm{logit}(Q_E),2)$ . Finally, study 3 aims to reduce uncertainty in the odds ratio of effectiveness of the new treatment compared with the standard of care. This study undertakes a randomized controlled trial with 200 patients on each arm with the data simulated from 2 binomial distribution, one for each treatment arm, $X_1~\mathrm{Bin}(200,P_C)$ and $X_2~\mathrm{Bin}(200,P_T)$ .

Dynamics of Implementation

Based on the current information, standard care has an average net benefit of $\$2,159,300$ , and the novel treatment has an average net benefit of $\$2,164,900$ . Thus, the current optimal treatment is the novel treatment. However, there is substantial uncertainty about this result, with only a 57% chance that the novel treatment is the most cost-effective. To adjust for imperfect implementation, the risk of side effects for the novel treatment is currently assumed to make clinicians reluctant to implement it. Thus, the novel treatment is not currently used, and the value of the current decision is $\mathcal{C}=\$2,159,300.$

It is then assumed that clinicians will begin to adopt the novel treatment when the probability of cost-effectiveness is greater than 60%. However, some clinicians will have higher levels of risk aversion and will therefore avoid the novel treatment until the evidence of cost-effectiveness is clearer. The uptake of the novel treatment is assumed to relate linearly to the probability of cost-effectiveness. Furthermore, the uptake is assumed to be instantaneous and static, with full treatment switching achieved if the probability of cost-effectiveness is 1. This gives a functional form of

$$m_2(\mathit{X})=f_d^m(p_2(\mathit{X}))=\{\begin{array}{cc}0\hfill & p_2(\mathit{X})<0.6\hfill \\ \multicolumn{1}{c}{\frac{10}{4}(p_2(\mathit{X})-0.6)}& p_2(\mathit{X})\ge 0.6\hfill \end{array}.$$

(17)

As the example has 2 potential decision options, $m_1(\mathit{X})=1-m_2(\mathit{X})$ is defined as to estimate ${\mathcal{F}}^{\mathbf{X}}$ across the range of plausible data sets $\mathit{X}$ .

Assessing the Performance of the Moment Matching Method

To assess the estimation methods for ${\mathrm{EVSI}}^{\mathrm{IM}}$ , the implementation-adjusted EVSI will be calculated for the 3 studies. To determine the expected value of the standard of care and the novel treatment under current information, $S$ was set to 10,000. To ensure a feasible computation time for the nested Monte Carlo method, $S=5,000$ and $R=10,000$ , and for the moment matching method, $Q=50$ and $R=10,000$ .

The 2 methods will be compared in terms of the estimates of ${\mathrm{EVSI}}^{\mathrm{IM}}$ across the 3 examples. The estimates of the relationship between the probability that the novel treatment is cost-effective and the sample-specific incremental net benefit (i.e., the difference between the net benefit for no treatment and the novel treatment) will also be compared graphically to determine whether the functional form chosen for the regression is sufficiently flexible to capture the varied relationships. Finally, the computational time required to generate the ${\mathrm{EVSI}}^{\mathrm{IM}}$ estimate will be computed.

Results

Implementation-Adjusted EVSI

Table 2 contains the ${\mathrm{EVSI}}^{\mathrm{IM}}$ estimates from the nested Monte Carlo method and the augmented moment matching method. The 2 methods are very similar, with the largest discrepancy observed for study 3, which collects additional information to estimate the odds ratio of the critical event with treatment. However, the discrepancy is only about 6% of the ${\mathrm{EVSI}}^{\mathrm{IM}}$ estimate. Note that as both of these estimates are obtained using simulation methods, some differences between the 2 estimates are expected. Thus, it seems likely that both methods were able to accurately estimate ${\mathrm{EVSI}}^{\mathrm{IM}}$ in this example without making restrictive assumptions about the distribution of the net benefit or the data-generating mechanism.

Table 2

Estimated Implementation-Adjusted EVSI ( ${\mathrm{EVSI}}^{\mathrm{IM}}$ ) and the Computational Time Required to Obtain These Estimates for the 3 Studies Considered for the Ades et al. ⁴ Example ^a

Study	Estimate of ${\mathrm{EVSI}}^{\mathrm{IM}}$		Computational Time (s)
	Nested Monte Carlo	Moment Matching	Nested Monte Carlo	Moment Matching
1: Updating $p_{\mathrm{SE}}$	6,086	6,013	12	2.1
2: Updating $Q_C$	1,924	1,849	12	2.1
3: Updating $\mathrm{OR}$	1,778	1,669	272	4.7

a

All estimates are obtained using both the nested Monte Carlo method and the moment matching method.

Estimating the Relationship between the Probability of Cost-Effectiveness and the Expected Net Benefit

Figure 1 plots the relationship between $p_2(\mathit{X})$ , the probability that the novel treatment is cost-effective, and ${\mu }_2(\mathit{X})-{\mu }_1(\mathit{X})$ , the expected posterior incremental net benefit. In this case, positive values of the incremental net benefit indicate that the novel treatment is optimal, so $p_2(\mathit{X})$ would be expected to be about 0.5 when ${\mu }_2(\mathit{X})-{\mu }_1(\mathit{X})$ is equal to 0. The gray dashed line represents the relationship estimated with the moment matching method, and the solid black line represents the relationship estimated with the nested Monte Carlo method. For all 3 studies, the relationship between $p_2(\mathit{X})$ and ${\mu }_2(\mathit{X})-{\mu }_1(\mathit{X})$ is similar across the 2 methods. The shape of the relationship changes across the 3 studies but is well captured by the generalized logistic function proposed for the regression. Discrepancies between the 2 curves always occur in areas of low density for the expected incremental net benefit, shown by the density plots at the top in Figure 1. As ${\mathrm{EVSI}}^{\mathrm{IM}}$ is the product of ${\mu }_2(\mathit{X})-{\mu }_1(\mathit{X})$ and a function of $p_2(\mathit{X})$ , these sections where the curve is poorly estimated have very limited impact on the overall results. These functions are displayed for a given sample size. However, the moment matching method can estimate ${\mathrm{EVSI}}^{\mathrm{IM}}$ for different alternative sample sizes. In this case, the comparison values estimated using nested Monte Carlo would have to be recomputed and so these results are not shown.

Figure 1

Estimated functional relationship between the probability of cost-effectiveness and the sample-specific expected incremental net benefit between the 2 treatment options for the 3 examples. The left-hand panel is for study 1, which updates $p_{\mathrm{SE}}$ ; the middle panel is for study 2, which updates $Q_C$ ; and the right-hand panel is for study 3, which updates $\mathrm{OR}$ . The black line represents the estimates generated by the nested Monte Carlo (NMC) method, and the gray dashed line represents the estimates generated by the moment matching (MM) method. The density of the sample-specific expected incremental net benefit is represented above each plot, with a gray density plot (estimated by the MM method) plotted over a black density plot from the NMC method.

Computational Time

Table 2 displays the time taken (in seconds) to compute the ${\mathrm{EVSI}}^{\mathrm{IIM}}$ for all 3 examples for the 2 methods. The moment matching method is between 6 and 60 times faster than the nested Monte Carlo method. For this example, the moment matching method requires 100 times fewer model runs than the nested Monte Carlo method to achieve the same accuracy. Fitting the regression model requires a fixed computational cost of about 2 s. Thus, in decision models in which each model run has a nonnegligible computational cost, the moment matching method will be about 100 times faster than the nested Monte Carlo method, if $S$ , $R$ , and $Q$ are set to the same values used in this article. Finally, note that the moment matching method estimates ${\mathrm{EVSI}}^{\mathrm{IM}}$ across sample size, so the reported computational times also allow us to recompute ${\mathrm{EVSI}}^{\mathrm{IM}}$ for alternative sample sizes.

Discussion

It has been suggested that value-of-information analyses should consider realistic assumptions about the implementation of new health care technologies.^8–10,12 ${\mathrm{EVSI}}^{\mathrm{IM}}$ measures the expected value of changing the implementation levels of the treatments through a research study by assuming that treatment implementation is likely to be more complete and faster if stronger evidence exists in favor of that treatment.^8,10 However, the only method that discussed the computation of ${\mathrm{EVSI}}^{\mathrm{IM}}$ relied on restrictive assumptions to obtain analytic formulas. ⁸ Thus, it was unclear how to compute ${\mathrm{EVSI}}^{\mathrm{IM}}$ in complex models that did not respect these assumptions. ²⁸

This article addresses this gap by developing 2 methods to estimate the probability that a given treatment is cost-effective across the range of plausible data sets. A computationally expensive nested simulation method to estimate ${\mathrm{EVSI}}^{\mathrm{IM}}$ is developed, based on the standard nested EVSI calculation method. ⁴ The moment matching method for EVSI calculation is then extended to efficiently estimate ${\mathrm{EVSI}}^{\mathrm{IM}}$ by using nonlinear regression to estimate the probability of cost-effectiveness from the expected posterior net benefit. These 2 methods provide similar estimates of ${\mathrm{EVSI}}^{\mathrm{IM}}$ , although the adjusted moment matching method is substantially faster. An extended moment matching method is also introduced to compute ${\mathrm{EVSI}}^{\mathrm{IM}}$ across different sample sizes for the proposed future study. Thus, ${\mathrm{EVSI}}^{\mathrm{IM}}$ can now be computed for a range of models to support research prioritization and design, either alone ⁸ or combined with the standard EVSI to compute the expected value of research. ¹⁰

A limitation of this work is that to calculate ${\mathrm{EVSI}}^{\mathrm{IM}}$ using these methods, the function $f_d^m(p_d(\mathit{X}))$ that calculates the market share based on the probability of cost-effectiveness must be specified. The example in this article assumed that market share increased linearly to 100%, when the probability of cost-effectiveness for the novel treatment is greater than 0.6. However, $f_d^m(p_d(\mathit{X}))$ is likely to be more complex in practice and may be challenging to determine. Grimm et al. ¹⁰ used diffusion models to make realistic assumptions about the implementation changes over time, but these would need to be reestimated to determine how the strength of evidence affects diffusion.

If the functional form of $f_d^m(p_d(\mathit{X}))$ is unknown, it would be possible to undertake a sensitivity analysis to its functional form. Using these methods, this sensitivity analysis would be relatively inexpensive, as the probability of cost-effectiveness would not need to be recomputed. However, it may be challenging to determine the appropriate range of functional forms that should be considered in this sensitivity analyses, especially if the possibility of implementation levels changing over time was also included.

Another limitation is the assumption that implementation is related to the outcome of a cost-effectiveness analysis (i.e., the probability of cost-effectiveness). Implementation could be more closely related to results based on the primary clinical outcome alone, rather than the cost-effectiveness, or based on safety concerns. These methods could be adapted to estimate the probability that a given treatment is effective (i.e., the primary clinical outcome is largest for a specific treatment) or safe (i.e., adverse events are lower). The market share could then be estimated based on this probability of effectiveness or safety. This analysis would jointly consider potential complementary aspects of clinical decision making (i.e., cost-effectiveness and clinical efficacy or safety) in study design.

Supplemental Material

sj-pdf-1-mdm-10.1177_0272989X211073098 – Supplemental material for Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

Supplemental material, sj-pdf-1-mdm-10.1177_0272989X211073098 for Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation by Anna Heath in Medical Decision Making

Supplemental Material

sj-r-1-mdm-10.1177_0272989X211073098 – Supplemental material for Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

Supplemental material, sj-r-1-mdm-10.1177_0272989X211073098 for Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation by Anna Heath in Medical Decision Making

Supplemental Material

sj-r-2-mdm-10.1177_0272989X211073098 – Supplemental material for Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

Supplemental material, sj-r-2-mdm-10.1177_0272989X211073098 for Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation by Anna Heath in Medical Decision Making

Supplemental Material

sj-r-3-mdm-10.1177_0272989X211073098 – Supplemental material for Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

Supplemental material, sj-r-3-mdm-10.1177_0272989X211073098 for Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation by Anna Heath in Medical Decision Making

Supplemental Material

sj-r-4-mdm-10.1177_0272989X211073098 – Supplemental material for Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

Supplemental material, sj-r-4-mdm-10.1177_0272989X211073098 for Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation by Anna Heath in Medical Decision Making