Expert elicitation (EE) has been used across disciplines to estimate input parameters for computational modeling research when information is sparse or conflictual.
Objectives
We conducted a systematic review to compare EE methods used to generate model input parameters in health research.
Data Sources
PubMed and Web of Science.
Study Eligibility
Modeling studies that reported the use of EE as the source for model input probabilities were included if they were published in English before June 2021 and reported health outcomes.
Data Abstraction and Synthesis
Studies were classified as “formal” EE methods if they explicitly reported details of their elicitation process. Those that stated use of expert opinion but provided limited information were classified as “indeterminate” methods. In both groups, we abstracted citation details, study design, modeling methodology, a description of elicited parameters, and elicitation methods. Comparisons were made between elicitation methods.
Study Appraisal
Studies that conducted a formal EE were appraised on the reporting quality of the EE. Quality appraisal was not conducted for studies of indeterminate methods.
Results
The search identified 1520 articles, of which 152 were included. Of the included studies, 40 were classified as formal EE and 112 as indeterminate methods. Most studies were cost-effectiveness analyses (77.6%). Forty-seven indeterminate method studies provided no information on methods for generating estimates. Among formal EEs, the average reporting quality score was 9 out of 16.
Limitations
Elicitations on nonhealth topics and those reported in the gray literature were not included.
Conclusions
We found poor reporting of EE methods used in modeling studies, making it difficult to discern meaningful differences in approaches. Improved quality standards for EEs would improve the validity and replicability of computational models.
Highlights
We find extensive use of expert elicitation for the development of model input parameters, but most studies do not provide adequate details of their elicitation methods.
Lack of reporting hinders greater discussion of the merits and challenges of using expert elicitation for model input parameter development.
There is a need to establish expert elicitation best practices and reporting guidelines.
MaglioPPSepulvedaMJMabryPL.Mainstreaming modeling and simulation to accelerate public health innovation. Am J Public Health. 2014;104(7):1181–6.
2.
O’DonnellEAtkinsonJAFreebairnLRychetnikL.Participatory simulation modelling to inform public health policy and practice: rethinking the evidence hierarchies. J Public Health Policy. 2017;38(2):203–15.
3.
PetittiDBLinJSOwensDKCroswellJMFeuerEJ.Collaborative modeling: experience of the U.S. Preventive Services Task Force. Am J Prev Med. 2018;54(1s1):S53–62.
4.
HabbemaJDWiltTJEtzioniR, et al. Models in the development of clinical practice guidelines. Ann Intern Med. 2014;161(11):812–8.
5.
MorganMG.Use (and abuse) of expert elicitation in support of decision making for public policy. Proc Natl Acad Sci U S A. 2014;111(20):7176–84.
6.
VennixJAMAndersenDFRichardsonGPRohrbaughJ. Model-building for group decision support: Issues and alternatives in knowledge elicitation. Eur J Oper Res. 1992;59(1):28–41.
7.
RefsgaardJCvan der SluijsJPHøjbergALVanrolleghemPA.Uncertainty in the environmental modelling process: a framework and guidance. Environ Model Softw. 2007;22(11):1543–56.
8.
KoppREDeContoRMBaderDA, et al. Evolving understanding of Antarctic ice-sheet physics and ambiguity in probabilistic sea-level projections. Earth’s Future. 2017;5(12):1217–33.
9.
IglesiasCPThompsonARogowskiWHPayneK.Reporting guidelines for the use of expert judgement in model-based economic evaluations. Pharmacoeconomics. 2016;34(11):1161–72.
10.
GrigoreBPetersJHydeCSteinK.Methods to elicit probability distributions from experts: a systematic review of reported practice in health technology assessment. Pharmacoeconomics. 2013;31(11):991–1003.
11.
SoaresMOSharplesLMortonAClaxtonKBojkeL.Experiences of structured elicitation for model-based cost-effectiveness analyses. Value Health. 2018;21(6):715–23.
12.
SankatsingVDVJuraniecKGrimmSE, et al. Cost-effectiveness of Digital Breast Tomosynthesis in Population-based Breast Cancer Screening: A Probabilistic Sensitivity Analysis. Radiology. 2020;297(1):40–8.
13.
VeermanJLVan BeeckEFBarendregtJJMackenbachJP.By how much would limiting TV food advertising reduce childhood obesity?Eur J Public Health. 2009;19(4):365–9.
14.
ApelbergBJFeirmanSPSalazarE, et al. Potential public health effects of reducing nicotine levels in cigarettes in the United States. N Engl J Med. 2018;378(18):1725–33.
15.
LevyDTMumfordEACummingsKM, et al. The relative risks of a low-nitrosamine smokeless tobacco product compared with smoking cigarettes: estimates of a panel of experts. Cancer Epidemiol Biomarkers Prev. 2004;13(12):2035–42.
16.
LevyDTMumfordEACummingsKM, et al. The potential impact of a low-nitrosamine smokeless tobacco product on cigarette smoking in the United States: estimates of a panel of experts. Addict Behav. 2006;31(7):1190–200.
17.
PeelAJenksMChoudhuryM, et al. Use of expert judgement across NICE guidance-making programmes: a review of current processes and suitability of existing tools to support the use of expert elicitation. Appl Health Econ Health Policy. 2018;16(6):819–36.
18.
BojkeLSoaresMOClaxtonK, et al. Reference case methods for expert elicitation in health care decision making. Med Decis Making. doi:0272989X211028236.
19.
CookeRM.Experts in Uncertainty: Opinion and Subjective Probability in Science. New York: Oxford University Press; 1991.
20.
ButlerATThomasMKPintarKDM. Systematic review of expert elicitation methods as a tool for source attribution of enteric illness. Foodborne Pathog Dis. 2015;12(5):367–82.
21.
GordonTJ.The delphi method. Futures Res Methodol. 1994;2(3):1–30.
22.
O’HaganABCaitlinEDaneshkhahA, et al. Uncertain Judgements: Eliciting Experts’ Probabilities. New York: Wiley; 2006.
23.
GarthwaitePHKadaneJBO’HaganA.Statistical methods for eliciting probability distributions. J Am Stat Assoc. 2005;100(470):680–701.
24.
HemmingVBurgmanMAHaneaAMMcBrideMFWintleBC.A practical guide to structured expert elicitation using the IDEA protocol. Methods Ecol Evol. 2018;9(1):169–80.
25.
JohnsonSRTomlinsonGAHawkerGAGrantonJTFeldmanBM.Methods to elicit beliefs for Bayesian priors: a systematic review. J Clin Epidemiol. 2010;63(4):355–69.
26.
SpeightPPalmerSMolesD, et al. The cost-effectiveness of screening for oral cancer in primary care. Health Technol Assess. 2006;10(14):1–144, iii.
27.
BrownBB.Delphi Process: A Methodology Used for the Elicitation of Opinions of Experts. Santa Monica (CA): RAND Corporation; 1968.
28.
GoslingJP. SHELF: the Sheffield Elicitation Framework. In: DiasLCMortonAQuigleyJ, eds. Elicitation: The Science and Art of Structuring Judgement. Cham (UK): Springer International Publishing; 2018:61–93.
29.
GrigoreBPetersJHydeCSteinK.EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment. BMC Med Inform Decis Mak. 2017;17(1):131.
30.
GarberAMPhelpsCE.Economic foundations of cost-effectiveness analysis. J Health Econ. 1997;16(1):1–31.
31.
MeltzerD.Accounting for future costs in medical cost-effectiveness analysis. J Health Econ. 1997;16(1):33–64.
32.
NeumannPJSandersGDRussellLBSiegelJEGaniatsTG.Cost-Effectiveness in Health and Medicine. Oxford (UK): Oxford University Press; 2016.
33.
RichardsonJ.Cost utility analysis: what should be measured?Soc Sci Med. 1994;39(1):7–21.
34.
PageMJMcKenzieJEBossuytPM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.
35.
KuntzKSainfortFButlerM, et al. Decision and simulation modeling alongside systematic reviews. In: Decision and Simulation Modeling in Systematic Reviews [online]. Rockville (MD): Agency for Healthcare Research and Quality; 2013.
36.
AkpoEIHJansenIRMaesESimoensS. Cost-utility analysis of lipegfilgrastim compared to pegfilgrastim for the prophylaxis of chemotherapy-induced neutropenia in patients with stage II-IV breast cancer. Front Pharmacol. 2017;8:614.
37.
AnnemansLBresseXGobboCPapageorgiouM.Health economic evaluation of a vaccine for the prevention of herpes zoster (shingles) and post-herpetic neuralgia in adults in Belgium. J Med Econ. 2010;13(3):537–51.
38.
BanzK, Eucomed CRT Steering Committee. Cardiac resynchronization therapy (CRT) in heart failure: a model to assess the economic value of this new medical technology. Value Health. 2005;8(2):128–39.
39.
BojkeLClaxtonKBravo-VergelYSculpherMPalmerSAbramsK.Eliciting distributions to populate decision analytic models. Value Health. 2010;13(5):557–64.
40.
ChiêmJCMacqJSpeybroeckN.Rule-based modeling of chronic disease epidemiology: elderly depression as an illustration. PLoS One. 2012;7(8):e41452.
41.
De GraeveDSmetAMehnertA, et al. Long-acting risperidone compared with oral olanzapine and haloperidol depot in schizophrenia: a Belgian cost-effectiveness analysis. Pharmacoeconomics. 2005;23(suppl 1):35–47.
42.
DewildeSVerdianLMaclaineGD.Cost-effectiveness of ziconotide in intrathecal pain management for severe chronic pain patients in the UK. Curr Med Res Opin. 2009;25(8):2007–19.
43.
DubinskyMCJohansonJFSeidmanEGOfmanJJ.Suspected inflammatory bowel disease: the clinical and economic impact of competing diagnostic strategies. Am J Gastroenterol. 2002;97(9):2333–42.
44.
Duenas-MezaEGiraldo-CadavidLFKarpfE, et al. Cost-utility analysis of an integrated care program for children with asthma in a medium-income country. Pediatr Pulmonol. 2020;55(11):3110–8.
45.
EkaetteELeeRCCookeDLIftodySCraigheadP.Probabilistic fault tree analysis of a radiation treatment system. Risk Anal. 2007;27(6):1395–410.
46.
FirozBFLeeSJNghiemPQureshiAA.Role of skin biopsy to confirm suspected acute graft-vs-host disease: results of decision analysis. Arch Dermatol. 2006;142(2):175–82.
47.
GarayOUPalaciosAPichon-RiviereA, et al. The cost-effectiveness of continuous versus intermittent renal replacement therapies in acute kidney injury: perspective of the social services for the elderly in Argentina. Value Health Reg Issues. 2019;20:142–8.
48.
GarthwaitePHChilcottJBJenkinsonDJTappendenP.Use of expert knowledge in evaluating costs and benefits of alternative service provisions: a case study. Int J Technol Assess Health Care. 2008;24(3):350–7.
49.
GordonLGHirstNGMayneGC, et al. Modeling the cost-effectiveness of strategies for treating esophageal adenocarcinoma and high-grade dysplasia. J Gastrointest Surg. 2012;16(8):1451–61.
50.
GrandTSBasarirHJacksonLJ.The cost-effectiveness of oral contraceptives compared to ‘no hormonal treatment’ for endometriosis-related pain: an economic evaluation. PLoS One. 2019;14(1):e0210089.
51.
GrigoreBPetersJHydeCSteinK.A comparison of two methods for expert elicitation in health technology assessments. BMC Med Res Methodol. 2016;16:85.
52.
GyllenstenHHakkarainenKMJönssonAK, et al. Modelling drug-related morbidity in Sweden using an expert panel of pharmacists’. Int J Clin Pharm. 2012;34(4):538–46.
53.
HandelsRLHWimoADodelR, et al. Cost-utility of using Alzheimer’s disease biomarkers in cerebrospinal fluid to predict progression from mild cognitive impairment to dementia. J Alzheimers Dis. 2017;60(4):1477–87.
54.
HindeSMcKennaCWhyteS, et al. Modelling the cost-effectiveness of public awareness campaigns for the early detection of non-small-cell lung cancer. Br J Cancer. 2015;113(1):135–41.
55.
HitimanaRLindholmLMogrenI, et al. Incremental cost and health gains of the 2016 WHO antenatal care recommendations for Rwanda: results from expert elicitation. Health Res Policy Syst. 2019;17.
56.
HsuTCChenHHYangMC, et al. Pharmacoeconomic analysis of capecitabine versus 5-fluorouracil/leucovorin as adjuvant therapy for stage III colon cancer in Taiwan. Value Health. 2011;14(5):647–51.
57.
KaufTLFarkouhRAEarnshawSRWatsonMEMaroudasPChambersMG.Economic efficiency of genetic screening to inform the use of abacavir sulfate in the treatment of HIV. Pharmacoeconomics. 2010;28(11):1025–39.
58.
KerenRHelfandMHomerCMcPhillipsHLieuTA.Projected cost-effectiveness of statewide universal newborn hearing screening. Pediatrics. 2002;110(5):855–64.
59.
KipMMASteutenLMGKoffijbergHIjzermanMJKustersR. Using expert elicitation to estimate the potential impact of improved diagnostic performance of laboratory tests: a case study on rapid discharge of suspected non-ST elevation myocardial infarction patients. J Eval Clin Pract. 2018;24(1):31–41.
60.
LoeveFBrownMLBoerRvan BallegooijenMvan OortmarssenGJHabbemaJD.Endoscopic colorectal cancer screening: a cost-saving analysis. J Natl Cancer Inst. 2000;92(7):557–63.
61.
LubellYStaedkeSGGreenwoodBM, et al. Likely health outcomes for untreated acute febrile illness in the tropics in decision and economic models; a Delphi survey. PLoS One. 2011;6(2):e17439.
62.
McKennaCHawkinsNClaxtonK, et al. Cost-effectiveness of enhanced external counterpulsation (EECP) for the treatment of stable angina in the United Kingdom. Int J Technol Assess Health Care. 2010;26(2):175–82.
63.
MeeyaiAPraditsitthikornNKotirumS, et al. Seasonal influenza vaccination for children in Thailand: a cost-effectiveness analysis. PLoS Med. 2015;12(5):e1001829.
64.
PetersohnSGrimmSERamaekersBLTten Cate-HoekAJJooreMA.Exploring the feasibility of comprehensive uncertainty assessment in health economic modeling: a case study. Value Health. 2021;24(7):983–94.
65.
PoncetAGencerBBlondonM, et al. Electrocardiographic screening for prolonged QT interval to reduce sudden cardiac death in psychiatric patients: a cost-effectiveness analysis. PLoS One. 2015;10(6):e0127213.
66.
ProsserLAKongCYRusinakDWaisbrenSL.Projected costs, risks, and benefits of expanded newborn screening for MCADD. Pediatrics. 2010;125(2):e286–94.
67.
SchumockGTWaltonSMGuptaSRCramerJA.Cost analysis of divalproex sodium extended-release compared to valproic acid in the treatment of bipolar disorder. Curr Med Res Opin. 2007;23(8):1997–2004.
68.
SoaresMODumvilleJCAshbyRL, et al. Methods to assess cost-effectiveness and value of further research when data are sparse: negative-pressure wound therapy for severe pressure ulcers. Med Decis Making. 2013;33(3):415–36.
69.
StevensonMDOakleyJELloyd JonesM, et al. The cost-effectiveness of an RCT to establish whether 5 or 10 years of bisphosphonate treatment is the better duration for women with a prior fracture. Med Decis Making. 2009;29(6):678–89.
70.
van HoevenLRJanssenMPLieshout-KrikkeRWMolenaar-de BackerMW.An assessment of the risk, cost-effectiveness, and perceived benefits of anti-parvovirus B19 tested blood products. Transfusion. 2019;59(7):2352–60.
71.
WaltonSMSchumockGTMcLainDA.Cost analysis of flavocoxid compared to naproxen for management of mild to moderate OA. Curr Med Res Opin. 2010;26(9):2253–61.
72.
WielageRCBansalMAndrewsJSKleinRWHappichM.Cost-utility analysis of duloxetine in osteoarthritis: a US private payer perspective. Appl Health Econ Health Policy. 2013;11(3):219–36.
73.
CookeRMendelMThijsW.Calibration and information in expert resolution: a classical approach. Automatica. 1988;24(1):87–93.
74.
ColsonARCookeRM.Expert elicitation: using the classical model to validate experts’ judgments. Rev Environ Econ Policy. 2020;12(1): 113--132.
75.
KarnonJStahlJBrennanACaroJJMarJMöllerJ.Modeling using discrete event simulation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-4. Value Health. 2012;15(6):821–7.
76.
NeumannPJSandersGDRussellLBGaniatsTGSiegelJE.Cost-Effectiveness in Health and Medicine. Oxford (UK): Oxford University Press; 2017.
77.
HungerTSchnell-InderstPSahakyanNSiebertU.Using expert opinion in health technology assessment: a guideline review. Int J Technol Assess Health Care. 2016;32(3):131–9.
78.
Peñaloza RamosMCBartonPJowettSSuttonAJ.A systematic review of research guidelines in decision-analytic modeling. Value Health. 2015;18(4):512–29.
79.
RotheryCStrongMKoffijbergH, et al. Value of information analytical methods: report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value Health. 2020;23(3):277–86.
80.
LogulloPMacCarthyAKirtleySCollinsGS.Reporting guideline checklists are not quality evaluation forms: they are guidance for writing. Health Sci Rep. 2020;3(2):e165.
81.
TverskyAKahnemanD.Judgment under uncertainty: heuristics and biases. Science. 1974;185(4157):1124–31.
82.
KynnM.The ‘heuristics and biases’ bias in expert elicitation. J R Stat Soc Ser A Stat Soc. 2008;171(1):239–64.
83.
CaulleyLCatalá-LópezFWhelanJ, et al. Reporting guidelines of health research studies are frequently used inappropriately. J Clin Epidemiol. 2020;122:87–94.
84.
GlasziouPChalmersI.Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers. BMJ. 2018;363.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
