Expert elicitation is a useful tool to explore sources of uncertainty and to answer questions where data are expensive or difficult to collect. It has been used across a variety of disciplines and represents an important method for estimating source attribution for enteric illness. A systematic review was undertaken to explore published expert elicitation studies, identify key considerations, and to make recommendations for designing an expert elicitation in the context of enteric illness source attribution. Fifty-nine studies were reviewed. Five key themes were identified: the expert panel including composition and recruitment; the pre-elicitation material, which clarifies the research question and provides training in uncertainty and probability; the choice of elicitation tool and method (e.g., questionnaires, surveys, and interviews); research design; and analysis of elicited data. Careful consideration of these themes is critical in designing and implementing an expert elicitation in order to reduce bias and produce the best possible results. While there are various epidemiological and microbiological methods available to explore source attribution of enteric illness, expert elicitation provides an opportunity to identify gaps in our understanding and where such studies are not feasible or available, represents the only possible method for synthesizing knowledge about transmission.
Get full access to this article
View all access options for this article.
References
1.
AcostaH, WuDR, ForrestBM. Fuzzy experts on recreational vessels, a risk modelling approach for marine invasions. Ecol Model, 2010; 221:850–863.
2.
ArnellNW, TompkinsEL, AdgerWN. Eliciting information from experts on the likelihood of rapid climate change. Risk Anal, 2005; 25:1419–1431.
3.
AshtonRH. Combining the judgments of experts: How many and which ones?. Organ Behav Hum Decis Process, 1986; 38:405–414.
4.
AspinallW. A route to more tractable expert advice. Nature, 2010; 463:294–295.
5.
AyyubBM. Elicitation of Expert Opinions for Uncertainty and Risks. Boca Raton, FL: CRC Press, 2001.
6.
BakerE, KeislerJM. Cellulosic biofuels: Expert views on prospects for advancement. Energy, 2011; 36:595–605.
7.
BartonovaA. How can scientists bring research to use: The HENVINET experience. Environ Health, 2012; 11(Suppl 1):S2-069X-11-S1-S2.
8.
BoumaJA, KuikO, DekkerAG. Assessing the value of Earth Observation for managing coral reefs: An example from the Great Barrier Reef. Sci Total Environ, 2011; 409:4497–4503.
CagnoE, CaronF, ManciniM, RuggeriF. Using AHP in determining the prior distributions on gas pipeline failures in a robust Bayesian approach. Reliab Eng Syst Saf, 2000; 67:275–284.
11.
ClemenRT, WinklerRL. Combining probability distributions from experts in risk analysis. Risk Anal, 1999; 19:187–203.
12.
CookeRM. Experts in Uncertainty: Opinion and Subjective Probability in Science. New York: Oxford University Press, 1991.
13.
CookeRM, GoossensLLHJ. TU Delft expert judgment data base. Reliab Eng Syst Saf, 2008; 93:657–674.
14.
CookeRM, WilsonAM, TuomistoJT, MoralesO, TainioM, EvansJS. A probabilistic characterization of the relationship between fine particulate matter and mortality: Elicitation of European experts. Envir Sci Tech, 2007; 41:6598–6605.
15.
CoppersmithKJ, YoungsRR, SprecherC. Methodology and main results of seismic source characterization for the PEGASOS Project, Switzerland. Swiss J Geosci, 2009; 102:91–105.
16.
CresseyP, LakeR. Ranking Food Safety Risks: Development of NZFSA Policy 2004–2005. Client Report FW0563:23, 2005.
17.
CrossP, RigbyD, Edwards-JonesG. Eliciting expert opinion on the effectiveness and practicality of interventions in the farm and rural environment to reduce human exposure to Escherichia coli O157. Epidemiol Infect, 2012; 140:643–654.
18.
DavidsonVJ, RavelA, NguyenTN, FazilA, RuzanteJM. Food-specific attribution of selected gastrointestinal illnesses: Estimates from a Canadian expert elicitation survey. Foodborne Pathog Dis, 2011; 8:983–995.
19.
de JongA, WardekkerJA, van der SluijsJP. Assumptions in quantitative analyses of health risks of overhead power lines. Environ Sci Policy, 2012; 16:114–121.
20.
DelbecqAL, van de VenAH. A group process model for problem identification and program planning. J Appl Behav Sci, 1971; 7:466–492.
21.
DonlanCJ, WingfieldDK, CrowderLB, WilcoxC. Using expert opinion surveys to rank threats to endangered species: A case study with sea turtles. Conserv Biol, 2010; 24:1586–1595.
22.
DoriaMdF, BoydE, TompkinsEL, AdgerWN. Using expert elicitation to define successful adaptation to climate change. Environ Sci Policy, 2009; 12:810–819.
23.
ElmerF, SeifertI, KreibichH, ThiekenAH. A Delphi method expert survey to derive standards for flood damage data collection. Risk Anal, 2010; 30:107–124.
24.
EvansJS, GrayGM, SielkenRLJr, SmithAE, Valdez-FloresC, GrahamJD. Use of probabilistic expert judgment in uncertainty analysis of carcinogenic potency. Regul Toxicol Pharmacol, 1994; 20:15–36.
25.
FaussE, GormanME, SwamiN. Using expert elicitation to prioritize resource allocation for risk identification for nanosilver. J Law Med Ethics, 2009; 37:770–780.
26.
FitchK, BernsteinSJ, AguilarMS, BurnandB, LaCalleJR, LazaroP, van het LooM, McDonnellJ, VaderJ, KahanJP. The RAND/UCLA Appropriateness Method User's Manual. Santa Monica, CA: The Rand Corporation, 2001; MR1269.
27.
FlanderL, DixonW, McbrideM, BurgmanM. Facilitated expert judgment of environmental risks: Acquiring and analysing imprecise data. Int J Risk Assess Manag, 2012; 16:199–212.
ForsbergB, BrabackL, KeuneH, KobernusM, von KraussMK, YangA, BartonovaA. An expert assessment on climate change and health—with a European focus on lungs and allergies. Environ Health, 2012; 11(Suppl 1):S4.
30.
GarabedRB, PerezAM, JohnsonWO, ThurmondMC. Use of expert opinion for animal disease decisions: An example of foot-and-mouth disease status designation. Prev Vet Med, 2009; 92:20–30.
GordonSN, GalloK. Structuring expert input for a knowledge-based approach to watershed condition assessment for the Northwest Forest Plan, USA. Environ Monit Assess, 2011; 172:643–661.
33.
GoslingJP, HartA, MouatDC, SabirovicM, ScanlanS, SimmonsA. Quantifying experts' uncertainty about the future cost of exotic diseases. Risk Anal, 2012; 32:881–893.
34.
HavelaarAH, GalindoAV, KurowickaD, CookeRM. Attribution of foodborne pathogens using structured expert elicitation. Foodborne Pathog Dis, 2008; 5:649–659.
35.
Helmer-HirschbergO. The Use of the Delphi Technique in Problems of Educational Innovations, P-3499. Santa Monica, CA: The RAND Corporation, 1966.
36.
HetesB, RichmondH, PekarZ, The Expert Elicitation Task Force. US EPA Expert Elicitation Task Force White Paper. Washington, DC: The US Environmental Protection Agency, 2011; 2013.
37.
HirschKG, PodurJJ, JanserRF, McAlpineRS, MartellDL. Productivity of Ontario initial-attack fire crews: Results of an expert-judgement elicitation study. Can J Forest Res, 2004; 34:705–715.
38.
Hnottavange-TelleenK, ChaboraE, FinleyRJ, GreenbergSE, MarstellerS. Risk management in a large-scale CO2 geosequestration pilot project, Illinois, USA. 10th International Conference on Greenhouse Gas Control Technologies, 2011; 4:4044–4051.
39.
HoekG, BoogaardH, KnolA, de HartogJ, SlottjeP, AyresJG, BormP, BrunekreefB, DonaldsonK, ForastiereF, HolgateS, KreylingWG, NemeryB, PekkanenJ, StoneV, WichmannHE, van der SluijsJ. Concentration response functions for ultrafine particles and all-cause mortality and hospital admissions: Results of a European expert panel elicitation. Environ Sci Technol, 2010; 44:476–482.
40.
HoelzerK, OliverHF, KohlLR, HollingsworthJ, WellsMT, WiedmannM. Structured expert elicitation about Listeria monocytogenes cross-contamination in the environment of retail deli operations in the United States. Risk Anal, 2012; 32:1139–1156.
41.
HoffmannSA, FischbeckPS, KrupnickAJ, McWilliamsM. Attributing foodborne illnesses to their food sources: Using large expert panels to capture variability in expert judgment. Washington, DC: Resources for the Future, 2006; Discussion Paper 06-17-REV.
42.
HoffmannS, FischbeckP, KrupnickA, McwilliamsM. Elicitation from large, heterogeneous expert panels: Using multiple uncertainty measures to characterize information quality for decision analysis. Decision Anal, 2007; 4:91–109.
43.
HorstHS, DijkhuizenAA, HuirneRBM, De LeeuwPW. Introduction of contagious animal diseases into The Netherlands: Elicitation of expert opinions. Livest Prod Sci, 1998; 53:253–264.
44.
KahnemanD, TverskyA. On the psychology of prediction. Psychol Rev, 1973; 80:237–251.
KnolAB, de HartogJJ, BoogaardH, SlottjeP, van der SluijsJP, LebretE, CasseeFR, WardekkerJA, AyresJG, BormPJ, BrunekreefB, DonaldsonK, ForastiereF, HolgateST, KreylingWG, NemeryB, PekkanenJ, StoneV, WichmannHE, HoekG. Expert elicitation on ultrafine particles: Likelihood of health effects and causal pathways. Part Fibre Toxicol, 2009; 6:19-8977-6-19.
47.
KnolAB, SlottjeP, van der SluijsJP, LebretE. The use of expert elicitation in environmental health impact assessment: A seven step procedure. Environ Health, 2010; 9:19-069X-9-19.
48.
KoehlerJJ. The base rate fallacy reconsidered: Descriptive, normative, and methodological challenges. Behav Brain Sci, 1996; 19:1–53.
49.
KuhnertPM, MartinTG, GriffithsSP. A guide to eliciting and using expert knowledge in Bayesian ecological models. Ecol Lett, 2010; 13:900–914.
50.
KynnM. The ‘heuristics and biases’ bias in expert elicitation. J Roy Stat Soc A Sta, 2008; 171:239–264.
51.
LealJ, WordsworthS, LegoodR, BlairE. Eliciting expert opinion for economic models: An applied example. Value Health, 2007; 10:195–203.
52.
LeonCJ, VazquezPoloFJ, GonzalezRL. Elicitation of expert opinion in benefit transfer of environmental goods. Environ Resour Econ, 2003; 26:199–210.
53.
LieuTA, NguyenMD, BallR, MartinDB. Health outcomes of interest for evaluation in the Post-Licensure Rapid Immunization Safety Monitoring Program. Vaccine, 2012; 30:2824–2830.
54.
MarcotBG, AllenCS, MoreyS, ShivelyD, WhiteR. An expert panel approach to assessing potential effects of bull trout reintroduction on federally listed salmonids in the Clackamas River, Oregon. N Am J Fish Manage, 2012; 32:450–465.
55.
McdanielsT, MillsT, GregoryR, OhlsonD. Using expert judgments to explore robust alternatives for forest management under climate change. Risk Anal, 2012; 32:2098–2112.
56.
MoherD, LiberatiA, TetzlaffJ, AltmanDG, The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLOS Med, 2009; 6:e1000097.
57.
MontangeroA, BeleviH. Assessing nutrient flows in septic tanks by eliciting expert judgement: A promising method in the context of developing countries. Water Res, 2007; 41:1052–1064.
58.
MoonJH, KangCS. Use of fuzzy set theory in the aggregation of expert judgments. Ann Nucl Energy, 1999; 26:461–469.
59.
MorganG, HenrionM. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge, UK: Cambridge University Press, 1992.
60.
NorringtonL, QuigleyJ, RussellA, van der MeerR. Modelling the reliability of search and rescue operations with Bayesian Belief Networks. Reliab Eng Syst Saf, 2008; 93:940–949.
61.
O'HaganA. Probabilistic uncertainty specification: Overview, elaboration techniques and their application to a mechanistic model of carbon flux. Environ Modell Softw, 2012; 36:35–48.
62.
PageT, HeathwaiteAL, ThompsonLJ, PopeL, WillowsR. Eliciting fuzzy distributions from experts for ranking conceptual risk model components. Environ Modell Softw, 2012; 36:19–34.
63.
PiresSM. Assessing the applicability of currently available methods for attributing foodborne disease to sources, including food and food commodities. Foodborne Pathog Dis, 2013; 10:206–213.
64.
PiresSM, EversEG, van PeltW, AyersT, ScallanE, AnguloFJ, HavelaarA, HaldT, Med-Vet-Net Workpackage 28 Working Group. Attributing the human disease burden of foodborne infections to specific sources. Foodborne Pathog Dis, 2009; 6:417–424.
65.
PodofilliniL, DangV, ZioE, BaraldiP, LibrizziM. Using expert models in human reliability analysis—A dependence assessment method based on fuzzy logic. Risk Anal, 2010; 30:1277–1297.
66.
RavelA, DavidsonVJ, RuzanteJM, FazilA. Foodborne proportion of gastrointestinal illness: Estimates from a Canadian expert elicitation survey. Foodborne Pathog Dis, 2010; 7:1463–1472.
67.
RavnumS, ZimmerKE, KeuneH, GutlebAC, MurkAJ, KoppeJG, MagnantiB, LycheJL, EriksenGS, RopstadE, SkaareJU, KobernusM, YangA, BartonovaA, Krayer von KraussM. Policy relevant results from an expert elicitation on the human health risks of decabromodiphenyl ether (decaBDE) and hexabromocyclododecane (HBCD). Environ Health, 2012; 11(Suppl 1):S7-069X-11-S1-S7.
68.
RomanHA, HammittJK, WalshTL, StiebDM. Expert elicitation of the value per statistical life in an air pollution context. Risk Anal, 2012; 32:2133–2151.
69.
SantoriG, ValenteR, CambiasoF, GhirelliR, Gianelli CastiglioneA, ValenteU. Preliminary results of an expert-opinion elicitation process to prioritize an informative system funded by Italian ministry of health for cadaveric donor management, organ allocation, and transplantation activity. Transplant Proc, 2004; 36:433–434.
70.
SaundersM, MagnantiBL, Correia CarreiraS, YangA, Alamo-HernandezU, Riojas-RodriguezH, CalamandreiG, KoppeJG, Krayer von KraussM, KeuneH, BartonovaA. Chlorpyrifos and neurodevelopmental effects: A literature review and expert elicitation on research and policy. Environ Health, 2012; 11(Suppl 1):S5-069X-11-S1-S5.
71.
Speirs-BridgeA, FidlerF, McBrideM, FlanderL, CummingG, BurgmanM. Reducing overconfidence in the interval judgments of experts. Risk Anal, 2010; 30:512–523.
72.
StiberNA, PantazidouM, SmallMJ. Expert system methodology for evaluating reductive dechlorination at TCE sites. Environ Sci Technol, 1999; 33:3012–3020.
73.
TanK, BaxterG, NewellS, SmyeS, DearP, BrownleeK, DarlingJ. Knowledge elicitation for validation of a neonatal ventilation expert system utilising modified Delphi and focus group techniques. Int J Hum-Comput St, 2010; 68:344–354.
74.
ThomasMK, MurrayR, FlockhartL, PintarK, PollariF, FazilA, NesbittA, MarshallB. Estimates of the burden of foodborne illness in Canada for 30 specified pathogens and unspecified agents, circa 2006. Foodborne Pathog Dis, 2013; 10:639–648.
75.
TuomistoJT, WilsonA, EvansJS, TainioM. Uncertainty in mortality response to airborne fine particulate matter: Combining European air pollution experts. Reliab Eng Syst Saf, 2008; 93:732–744.
76.
TverskyA, KahnemanD. Judgment under uncertainty: Heuristics and biases. Science, 1974; 185:1124–1131.
77.
TyshenkoMG, ElSaadanyS, OrabyT, DarshanS, AspinallW, CatfordA, KrewskiD. Expert elicitation and probabilistic inversion for the judgment of prion disease risk uncertainties using the classical model, EXCALIBUR and UNIBALANCE. Prion, 2010; 4:165–166.
78.
TyshenkoMG, ElSaadanyS, OrabyT, DarshanS, AspinallW, CookeR, CatfordA, KrewskiD. Expert elicitation for the judgment of prion disease risk uncertainties. J Toxicol Environ Health A, 2011; 74:261–285.
79.
VallyH, GlassK, FordL, HallG, KirkMD, ShadboltC, VeitchM, FullertonKE, MustroJ, BeckerN. Proportion of illness acquired by foodborne transmission for nine enteric pathogens in Australia: An expert elicitation. Foodborne Pathog Dis, 2014; 11:727–733.
80.
van der Fels-KlerxHJ, HorstHS, DijkhuizenAA. Risk factors for bovine respiratory disease in dairy youngstock in The Netherlands: The perception of experts. Livest Prod Sci, 2000; 66:35–46.
81.
van der Fels-KlerxHJ, KandhaiMC, BrynestadS, DreyerM, BorjessonT, MartinsHM, UiterwijkM, MorrisonE, BooijCJH. Development of a European system for identification of emerging mycotoxins in wheat supply chains. World Mycotoxin J, 2009; 2:119–127.
82.
van der Fels-KlerxIH, GoossensLH, SaatkampHW, HorstSH. Elicitation of quantitative data from a heterogeneous expert panel: Formal process and application in animal health. Risk Anal, 2002; 22:67–81.
83.
van LentheJ. ELI: An interactive elicitation technique for subjective probability distributions. Organ Behav Hum Decis Process, 1993; 55:379–413.
84.
WalkerKD, CatalanoP, HammittJK, EvansJS. Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene. J Expo Anal Environ Epidemiol, 2003; 13:1–16.
85.
WalkerKD, EvansJS, MacIntoshD. Use of expert judgment in exposure assessment part I. Characterization of personal exposure to benzene. J Expo Anal Environ Epidemiol, 2001; 11:308–322.
86.
WardekkerJA, De JongA, Van BreeL, TurkenburgWC, Van Der SluijsJP. Health risks of climate change: An assessment of uncertainties and its implications for adaptation policies. Environ Health, 2012; 11:67.
87.
[WHO] World Health Organization. The Global Burden of Disease: 2004 Update. Geneva, Switzerland: The World Health Organization, 2008.
WreathallJ, BleyD, RothE, MulterJ, RaslearT. Using an integrated process of data and modeling in HRA. Reliab Eng Syst Safe, 2004; 83:221–228.
90.
ZimmerKE, GutlebAC, RavnumS, Krayer von KraussM, MurkAJ, RopstadE, SkaareJU, EriksenGS, LycheJL, KoppeJG, MagnantiBL, YangA, BartonovaA, KeuneH. Policy relevant results from an expert elicitation on the health risks of phthalates. Environ Health, 2012; 11(Suppl 1):S6-069X-11-S1-S6.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
