Abstract
INTRODUCTION:
Cost-utility analyses (CUA) are useful when the treatment conditions depend on patient preferences that are in turn dependent on health state utility value. Spina bifida (SB) is an example of such a preference-sensitive condition. Historically, the SB utility value for CUA has been gathered via a traditional face-to-face interview. However, due to funding and time constrains, utility estimation via online crowdsourcing has recently gained popularity. Our aim was to estimate the utility value for a generic SB health state using a validated online tool.
METHODS:
A cross-sectional survey of American adults was conducted using the time-trade-off (TTO) method. Participants were recruited from an online crowdsourcing interface, Amazon’s Mechanical Turk (mTurk). Demographic information and prior knowledge of SB were assessed. Respondents were provided a written passage and an online video explaining SB and its potential associated comorbidities. Participants were queried on hypothetical ascending time-trades from a child-parent dyad perspective to determine the utility of a SB health state in an affected 6-year-old child. Respondents were also asked to indicate the percentage of time traded from their life in relation to their child’s. Utility estimates were then calculated and compared using bivariate and multivariate analyses.
RESULTS:
We obtained 503 responses (85% response rate). Mean respondent age was 34
(
CONCLUSIONS:
Utility estimation for SB is feasible through crowdsourcing, and the resultant values are similar to previous estimates using traditional techniques. Subjects view the SB health state to be inferior to perfect health.
Introduction
Cost utility analysis (CUA) is a type of economic analysis that may be useful when more than one acceptable treatment option is available. In these situations, the ultimate treatment decision typically depends on patient preferences that are in turn dependent on the relative value of the possible health states that can occur after any treatment decision is made. This relative value, usually estimated as a “utility” value, is a number between 0 (representing death) and 1(representing perfect health) that quantifies the impact of a health state on the person’s overall quality of life. Utility estimation is a crucial component of any CUA; if the utility estimate is unsound, then the resulting CUA will be irredeemably biased. Perspectives on the impact of disease progression and treatment on quality of life can subsequently be represented as numerical values within this scale [1, 2, 3].
Spina bifida (SB) is an example of such a preference-sensitive condition. SB is the most common permanently disabling birth defect in the U.S., occurring in approximately 3/10,000 live birth [4, 5]. Patients with SB range from minimally symptomatic to severely disabled. Thus, a range of management options exists for SB patients. Improved treatment and management of SB have greatly increased life expectancy and quality of life for affected patients.
With many SB patients living well into adulthood, the need to better understand patient/family perspectives and values as they pertain to treatment options is becoming increasingly important for all physicians involved in their care. Furthermore, demand for SB health state utilities is increasing as more disease-specific cost-effective modeling is undertaken. At the core of SB CUA is the ability to provide an accurate estimation of SB utility.
Estimates for SB utility value via traditional interview approaches have been previously reported to be from 0.74 to 0.84 [6]. However, to date, no SB utility score has been estimated via online surveys or crowdsourcing methods. Our aim was to determine the estimated utility value for a generic SB health state using a previously validated online crowdsourcing tool [7]. Crowdsourcing is an online problem-solving and production model that utilizes the collective intelligence of online communities for specific purposes. One type of crowdsourcing is called Distributed Human Intelligence Tasking (DHIT), where a large scale of information that is too complex to be processed in an automated way can be broken down and processed individually [8]. Our current study utilizes the advantages of DHIT to estimate utility in SB. We hypothesize that the SB utility derived from online surveys will be similar to those estimated via traditional interview approaches.
Patients and methods
Study design
A cross-sectional survey study was conducted amongst the general public to elicit utility values for SB in an affected, generic 6-year-old child. A time-trade off (TTO) approach was used to determine utility value.
Study participants
We used Amazon.com’s ‘Mechanical Turk’ (MTurk,
Because all payments were made through an intermediary (Amazon.com), study
participants remained strictly anonymous. Exclusion criteria included residence outside
the United States, age
Instrument development
The initial survey questionnaire (see Appendix) contains demographic questions and a TTO assessment of SB health state utility. Participants were also asked to view a 3-minute video that illustrates the condition of SB (see Appendix).
After completing the SB-related passage and video, each participant was asked his or her perceived effect of SB on himself/herself, his/her child, and on the family. Participants were asked hypothetically about what quantity of life they would be willing to trade to get their child back to perfect health. Specifically, participants were asked whether they were willing to trade off from the end of their combined dyadic life (both parent & child) to restore their child back to a state of perfect health.
Utility elicitation
A 10-year timeframe was used for all scenarios, with the respondent offered a variable amount of time spent in ‘perfect health’ compared to 10 years in the disease state, in exchange for hypothetical trades of fixed amounts of time from their overall lifespan. Health durations changed in 1-year intervals for responses from 1 to 9 year TTO, and could be narrowed to between 0–365 days for respondents willing to trade less than one year’s time. This process allows only utilities between 0 and 1, since negative utility values are not possible with this experimental design.
We collected respondent demographics including age, gender, marital status, parental status, race/ ethnicity, education achievement, annual income, and prior SB or TTO related knowledge. Given our hypothesis that previous experience with SB or other chronic health conditions might affect an individual’s preferences for given health states, we also collected data on the health of respondents and their friends.
Prior to release, each survey instrument was internally validated using a convenience sample of medical students and residents, none of whom had a personal or family history of SB or previous experience with TTO models. This study was reviewed by our Institutional Review board and deemed to not be human subject research. No formal consent process was thus required, but the principles of the Declaration of Helsinki were followed.
Statistical analysis
Mean (
An alpha of 0.05 and 95% confidence intervals (CI) were used as criteria for statistical significance. All analyses were performed using SAS 9.4 (SAS Institute, Cary, NC).
Results
Respondent characteristics
A total of 591 participants were given the survey. Of the participants, 503 completed the task for an 85% response rate. The average age of the participants was 34 years. There were 245 female participants (48.8%), and the majority of the participants were Caucasian (76.8%). There was a similar representation of single vs. married respondents (45% vs. 38%). Other marital status includes sep/div (5.6%), widowed (0.4%), and living w/partner (11.4%). Almost half of the respondents had children (46%). The majority of participants had attained some level of college education (87%), and 42% of the respondents reported a median household income between $20,000 and $49,000.
Of the respondents, 491 (97%) did not have SB, 498 (99%) did not have a child with SB, and 433 (86%) did not have any friends with SB. Almost half (44%) of the complete respondents had some knowledge of SB. However, the majority of the participants (49%) had no prior knowledge of cost utility analysis (Table 1).
SB utility value and Bi-variate analysis
Compared to a perfect health state of 1.0, we found a mean utility value
of 0.85 (
Respondent demographics
Respondent demographics
Bivariate analysis on whether respondent characteristics may affect utility estimate
On multivariate analysis, none of the included factors such as prior NTD knowledge, race, marital status, education, or whether self/child/friend with SB was significantly associated with the determination of SB utility score (Table 3).
Multivariate analysis on whether respondent characteristics may affect utility
estimate
Multivariate analysis on whether respondent characteristics may affect utility estimate
CUA is a useful tool to consider in the management of conditions, such as SB, when multiple treatment options exist. These conditions typically depend to some extent on patient preferences. These patient preferences are typically dependent in turn on health state utility value, or how patients (or families) value the quality of life for a given health state. Spina bifida (SB) is an example of such a preference-sensitive condition. To accurately conduct CUA research, accurate utility values are crucially important. If a baseline utility value incorrectly represents the value of that health state, then the resulting CUA will be irredeemably biased and essentially worthless. However, utility elicitation, which is usually done via lengthy face-to-face interviews, can be time-consuming and expensive. To our knowledge, utility estimation for SB via online survey has not previously been performed.
We aimed to estimate the utility score for a typical 6-year-old child with SB using a validated online platform [3, 7]. We found this annual utility to be 0.85. Surprisingly, factors such as whether respondents/their child/their friend had SB did not significantly change their estimates of utility score. Prior knowledge of neural tube defect and SB were seen as significant factors in utility score determination on bivariate analysis, although this effect did not persist on multivariate analysis. This is consistent with our previous work on utility estimation [3, 7]. This result can be explained by several factors such as the small percentage of respondents with SB/NTD and possible co-linearity of the analyzed variables. Importantly, however, our crowd-sourced, calculated utility value is firmly in line with other estimates derived from more traditional methods.
A recent review by Tarride et al. showed that the utility estimate in children with chronic health conditions to be 0.80 from a utility survey done among parents, patients, and pediatricians. Interestingly, the average utility score derived from pediatricians tend to be higher than those derived from parents [2]. In another study using the National Health Interview Survey, the mean and median SB specific health utility score were 0.72 and 0.84, respectively [6]. Thus, our reported SB utility value developed from online survey is similar to the established values in literature. The consistency of these estimated utility values provides further support in the use of this online platform as a valuable tool to estimate utility for other urological conditions. This significantly lowers the barrier to deriving health state utility estimates for specific health states specific to the various treatments that patients with SB often undergo (e.g., shunted vs. not, continent vs. not, reconstructed vs. not, catheterizing vs. not).
At present, utility estimation in urology has been limited. There have been reported utility estimates for ureteroscopy, shock wave lithotripsy, and congenital ureteropelvic junction obstruction [12, 13]. These mentioned utility estimates have been conducted in the traditional approach via interviews. However, more recently, utility estimation via online platform has gained popularity as a valid and efficient method. Our group previously reported the utility estimation of childhood vesicoureteral reflux (VUR) using this online survey method and found that it was comparable to the utility score obtained via traditional interviews [7]. Methodological limitations of this online platform have also been evaluated recently. We previously reported that utility estimate fluctuates based on whether it was parent vs. child vs. dyad perspective. In addition, utility weight was surprisingly unaffected by anchoring bias, which is a common human tendency where subsequent decisions are made based on the initial piece of information [3]. Therefore, this online crowdsourcing tool not only provides precise utility estimation of VUR, but also more chronic pediatric urological conditions such as SB. The online tool’s success in deriving utility values for these two disease states suggests that it is a valid, reliable, and efficient tool to estimate CUA utilities.
The result of our study should be viewed in the context of its limitations. First, our sampled respondents may not provide a true representation of the U.S. population. The demographics in our study were 48.8% F, 86.8% Caucasian, and 37.6% married. In a study by Dworkin et al., they recruited subjects for family dynamic research. In their study, the demographics of US population were 50.8% F, 74.8% Caucasian, and 52.4% married. Age was not part of their demographics analysis [14]. Compared to national data, we showed similar gender demographics. However, there were less respondents who are married and more Caucasian respondents in our study. Given that the majority of our respondents were Caucasian, it is uncertain how precise our current utility value can estimate other racial ethnicities. However, race was not found to be a significant contributing factor on our multivariate analysis. Furthermore, previous literatures have shown that MTurk is more representative than other commonly used samples (U.S. undergraduates), and the survey result obtained through MTurk is similar to the traditional interview survey [15, 16, 17]. Nonetheless, a wide geographic and racial representation remains a major challenge in utility estimation.
Second, our current study estimates the utility of an overall SB state but does not provide specific distinction for myelomeningocele vs. other forms of SB (e.g., SB occulta or lipomeningocele). We would hypothesize that the utility value for myelomeningocele would be expected to be lower than other “less severe” forms, but we did not explore this point in this study. Furthermore, our analysis was based on trading from a parent/child dyad perspective. A prior study by our group showed that utility estimate was highest from child’s perspective followed by dyad’s perspective, and lowest from a parent’s perspective. Thus, the utility estimated from the dyad perspective likely provides an average from the other two health states. In addition, we estimated utility value based on a generic 6-year-old patient. This may not reflect the true utility estimate in much older or younger patients. However, the purpose of our current study is to validate the effectiveness and feasibility of crowdsourcing methods in estimating utility value in SB in general. This could now be adapted to a nearly infinite array of patient archetypes – preoperative patients with urinary leakage, postop patients with complications, older patients, younger patients, etc.
Third, we limited respondents’ age to older than 18 years. However, we did include a diverse cohort of participants such as parent/non-parent, familiar/unfamiliar with SB, and single/married. It is uncertain how much impact the age limit will affect utility estimation on a mostly pediatric condition. However, due to improvement in the care of SB, majority of patients live well into adulthood, and we would anticipate that the impact of this age limit would likely be minimal.
In addition, our survey respondents may not have a full spectrum of understanding about living with spina bifida. The brief Youtube video and description paragraph serve as a general overview of SB, but they do not necessarily provide an emphasis on some of the specific urological complications related to SB. However, the goal of this project is to determine the feasibility of MTurk for assessing utility in SB health states, but the fundamental structure of crowdsourcing can be applied to other health states as well (mobility, shunt malfunctions, fecal continence, decubiti, etc).
Finally, due to the paucity of literature on SB utility values, it is difficult to draw definitive conclusion on the accuracy of any one published estimate (i.e. “gold standard”). However, the consistency among the currently published SB utility values and our estimate is reassuring. By establishing online crowdsourcing as a valid tool for SB utility estimate, it provides a solid foundation for future studies to better define SB utility values in varies health states.
Conclusions
Using a validated online survey platform, we estimated that the utility value for a typical 6-year-old child with SB is 0.85, which is consistent with prior published utility scores obtained via more traditional approaches. Our study further supports the reliability of MTurk and crowdsourcing in general as a viable technique in spina bifida utility research.
Footnotes
Acknowledgments
Dr. Routh is supported in part by grant K08-DK100534 from the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK). The funding sources had no role in the collection, analysis and interpretation of data; in the writing of the manuscript; or in the decision to submit the manuscript for publication.
Conflict of interest
No authors have any financial and personal relationships with other people or organizations that could inappropriately influence (bias) our work, including employment, consultancies, stock ownership, honoraria, paid expert testimony, patent applications/registrations, and grants or other funding.
Abbreviations
Spina Bifida (SB)
Cost-utility analysis (CUA)
Time-trade-off (TTO)
Amazon’s Mechanical Turk (MTurk)
Neural Tube Defect (NTD)