Sage Journals: Discover world-class research

Abstract

Background

While structured expert elicitation (SEE) is gaining traction in health technology assessment in situations in which data are scarce, its application in practice remains limited. Co-designing a practical and fit-for-purpose SEE with experts could enhance its acceptability and feasibility in clinical research.

Objectives

An SEE was co-designed with clinicians to elicit expert opinions on 3 uncertain quantities of interest (QoIs) for a decision-analytic model in exercise oncology.

Methods

A series of co-design meetings was convened to design 6 elicitation stages. Individual elicitation was conducted using the variable interval method (VIM), via videoconferencing. Linear pooling was adopted to generate group estimates. Semi-structured interviews were conducted after the elicitation exercise to gather the experts’ first-hand experience of the elicitation process and to identify areas for improvement. Qualitative data were transcribed and content analyzed.

Results

Twelve experts participated in the co-designed SEE. Three beta distributions were derived and estimated from the experts’ responses: the relative risk reduction of cardiovascular events of exercise for women who survived early-stage endometrial cancer (Mean: 0.362, SD: 0.15), the probability that a clinician would refer a patient to the exercise program (Mean: 0.457, SD: 0.218), and the probability that a cancer patient would use such a health service upon referral (Mean: 0.446, SD: 0.203). Most of the experts’ first-hand experience of the co-designed SEE was positive. The qualitative feedback highlighted critical aspects of the elicitation process that should be designed and executed with caution when targeting clinicians with no prior experience of SEE.

Conclusions

This is the first expert elicitation conducted in exercise oncology. Engaging diverse stakeholders through co-design meetings and incorporating qualitative feedback proved effective and practical in introducing expert elicitation into clinical research.

Highlights

Recent SEE guidelines aim to facilitate the conduct of expert elicitation in model-based economic evaluation, but its application in practice remains limited.

Engaging experts in the design of SEE could enhance its acceptability and feasibility in clinical research.

This is the first co-designed expert elicitation involving clinicians in the field of exercise oncology.

This practical approach to conducting SEE could promote a wider adoption to inform health care policy decisions when the evidence is lacking or uncertain.

Keywords

expert elicitation exercise oncology co-design economic evaluation

Introduction

Performing economic evaluations using decision-analytic modeling is crucial to inform health care reimbursement decisions.¹ Decision-analytic modeling plays a vital role in improving the efficiency of randomized controlled trials (RCTs) and projecting cost and outcomes beyond the initial trial period.² Furthermore, early economic evaluation of new medical devices, treatments, or health services can be conducted during early-stage development to inform research, development, and early market access.^3–5 However, the evidence to inform model parameters is often of insufficient quality or nonexistent, which can lead to uncertainty in decision making.⁶ Failure to quantify this uncertainty results in wrong decisions that are associated with undesirable consequences (e.g., adopting an intervention that is not cost-effective).

Exercise is considered a beneficial adjunct therapy for adults during and after cancer treatment. Past economic evaluations in exercise oncology have assessed the value for money of various exercise programs for different cancer types.⁷ Given the paucity of data, however, it has been challenging to populate decision-analytic models to account for the long-term benefits of exercise in certain cancers.^8–11 This results in high uncertainty when populating these decision-analytic models.^7,12 Considering the lack of suitable data, decision makers often rely on expert judgments to bridge these evidence gaps.

Structured expert elicitation (SEE) is a formal process of extracting expert judgments to facilitate the characterization of uncertainty in decision-analytic modeling. By leveraging experts’ prior beliefs, this approach enhances the understanding of uncertainties surrounding model inputs where clinical evidence is scarce.¹³ Three well-established SEE protocols are available: the Sheffield Elicitation Framework (SHELF) protocol,^14,15 Cooke’s classical model (CM) protocol,¹⁶ and the Investigate, Discuss, Estimate and Aggregate (IDEA) protocol.¹⁷ While these protocols have similar elicitation procedures,¹⁸ their primary distinctions lie in the elicitation technique applied and the aggregation approach.¹⁹ Cooke’s CM and the IDEA protocol recommend mathematical aggregation with nonlinear or linear pooling,¹⁶ while the SHELF protocol favors behavioral aggregation.²⁰ Systematic reviews have identified strengths and weaknesses in these elicitation approaches.^21,22 For example, nonlinear aggregation requiring seed questions for expert opinion weighting presents challenges in identifying the right questions and imposes a burden on experts by requiring them to answer additional questions.²¹ In contrast, linear pooling is recognized for its simplicity and comparable performance to complex pooling methods.²³ Behavioral aggregation, on the other hand, involves multiple rounds of workshops for addressing disagreements between experts and reducing biases,²⁴ but it is time- and resource-intensive and often difficult to coordinate.^19,21 Past expert elicitation conducted in health technology assessment (HTA) has exhibited significant variation in the elicitation approaches employed.^11,22,25 The selection of the elicitation approaches propounded by past systematic reviews often stemmed from pragmatic considerations, such as available resources, the experts’ understanding of the elicitation concept, and their acceptance of the elicitation concept.^26,27

Considering the heterogeneity of elicitation approaches used in HTA,²⁸ there is a lack of clarity and guidance as to the most appropriate elicitation approach in the setting of exercise oncology. Furthermore, current elicitation approaches, similar to many other health economics methods, are not well understood by participating experts. Perhaps due to these methodological considerations, there is a notable underutilization of SEE in Australian clinical research. Qualitative research methods have been used by health economists to guide and enhance health care modeling practices by improving the validity and reliability of the model construction process.^29,30 Such a participatory approach is not commonly used in conducting SEE. Involving clinicians (e.g., oncologists, oncology nurses, exercise physiologists) in the design and conduct of SEE could provide valuable insights for the development of the SEE process, facilitating its improvement and refinement in terms of practicality in the clinical setting.

The objective of this study was to conduct and evaluate the acceptability of an SEE co-designed with clinicians in the field of exercise oncology by eliciting their expert opinions on 3 uncertain model parameters from a decision-analytic model assessing the cost-effectiveness of supervised exercise for women with early-stage endometrial cancer. Qualitative responses from the co-design team and the participating clinical experts were collected to directly shape the development of the SEE and to pinpoint opportunities for enhancing future SEE practices.

Method: The Co-design Process

The expert elicitation undertaken in this study was collaboratively designed, leveraging the first-hand experiences of clinicians involved in the ongoing clinical trial EnhAnCing treatment oUtcoMes after gynaEcological caNcer (ACUMEN) registered under the Australian and New Zealand Clinical Trials Registry (ACTRN12621000050853). The clinical experts included in the co-design process included 2 oncology nurses, 1 psychologist, 2 exercise physiologists, and 2 health economists, all of whom work in exercise oncology. The elicitation process was categorized into 6 broad elicitation stages: 1) defining the quantity of interest (QoI), 2) selection of experts, 3) elicitation approach, 4) aggregation approach, 5) elicitation format, and 6) elicitation training and piloting, adapted from an existing elicitation reference case protocol by Bojke et al. (2021) and current elicitation guidelines in health care decision making.^14,31,32 Each elicitation stage was then thoroughly discussed with the clinical team in 4 co-design meetings convened between November 2022 and May 2023. These meetings were platforms to navigate the decision paths of the elicitation process and to identify potential implementation barriers.

The 6 elicitation stages were then developed with the clinicians based on the feedback gathered during the co-design meetings. These meetings were audio recorded and later professionally transcribed verbatim by 2 independent analysts. Inductive content analysis of these transcriptions was undertaken, meaning that no preconceived theory was used to analyze or interpret the data. The main themes (listed above each elicitation stage) and subthemes (outlined under each elicitation stage) are summarized in Figure 1.³³ The themes and subthemes encapsulate the primary feedback and suggestions provided for each stage of the SEE process.

Figure 1.

Key considerations categorized across different stages of the elicitation process.

Research Problem: Define the Elicitation Questions

The elicitation focused on 3 QoIs ( $X_{1}$ , $X_{2}, X_{3}$ ) identified from a previously constructed decision-analytic model that aimed to assess the early cost-effectiveness evidence of supervised exercise for women treated for endometrial cancer in the Australian health care setting.³⁴ This existing decision-analytic model was populated using data sourced from available published studies; one of the key model parameters was based on a breast cancer study. Following extensive systematic searches in the databases without finding suitable evidence in the endometrial cancer population, the first QoI ( $X_{1})$ was defined, aiming to use expert elicitation to bridge this evidence gap. QoIs $X_{2}$ and $X_{3}$ were proposed by the health economists to address the uncertainties surrounding the total uptake of exercise by the eligible Australian population and health professionals, should it be implemented as policy on a national level in the future.

$X_{1}$ : The relative risk reduction of a cardiovascular disease event (such as stroke, heart failure, or coronary heart disease) from participating in routine exercise for women with early-stage endometrial cancer.

$X_{2}$ : The probability that a clinician (e.g., oncologists, general medical practitioners, oncology nurses) would refer eligible women with endometrial cancer to an exercise program.

$X_{3}$ : The probability that an eligible consumer (patient) would enroll in the exercise program upon referral.

In the first round of discussion with the co-design team, the clinicians identified that certain technical terms and phrasings of the QoIs were abstract and difficult to understand. Several changes were made to improve the communication with the target experts and enhance the clarity of the QoIs. The term “Quantity of Interest” was rephrased as “questions” for experts who are unfamiliar with SEE. Furthermore, terms such as “probability” were replaced with “likelihood,” where the experts had issues understanding the elicitation questions.

Expert Selection

A total of 15 experts from various professional disciplines (medical doctors, accredited exercise physiologists, oncology nurses, and professorial exercise researchers) were selected for recruitment. A formal personalized letter was drafted to invite the experts identified. A second round of recruitment was then conducted aiming to ensure diversity of experts, which was based on recent relevant publications in the field, a track record of similar clinical trials, and peer recommendation by the identified experts.

In conjunction with the co-design team, a set of inclusion criteria was devised to select suitable experts, focusing on their professional, clinical, and research backgrounds. The sample size of target experts adhered to current recommendations within existing elicitation guidelines.^24,33 While health economists in the team initially suggested including only medical professionals (i.e., oncologists and general practitioners), the larger team stressed the importance of including accredited exercise physiologists and oncology nurses due to their predominant role in exercise oncology research and their awareness of the value of exercise. The clinicians also expressed reservations about the feasibility of recruiting medical doctors, citing their competing clinical burdens. This collaborative approach to expert selection resulted in a more diverse and representative panel.

Elicitation Approach

The variable interval method (VIM) was chosen to seek expert judgments regarding the 2.5th percentile (L), the 50th percentile (M), and the 97.5th percentile (U) for each QoI.³⁵ The elicitation questions were structured to be straightforward and comprehensible to the average clinician. If any expert encountered difficulty in grasping the concept of percentiles, they were encouraged to envision these percentiles as representing the least likely, most likely, and highest likely possible values for each QoI on a scale of 0% to 100%. A reflection step was included to improve the quality of the estimates and reduce any bias. After the initial elicitation of the 2.5th and 97.5th percentile for QoI $X_{1}$ , QoI $X_{2}$ , and QoI $X_{3}$ , the experts were asked to imagine a hypothetical scenario in which someone in the future tells them that the true value of X has been determined and it falls outside of their estimated range and how they would react to that information. This facilitated the experts to reflect on the credible interval that they had stated to reduce any bias and heuristics.¹⁵

Alternative elicitation approaches of the VIM and fixed-interval methods (FIM) were also presented to the co-design team for consideration.¹¹ However, these methods were not preferred due to their perceived complexity by the clinicians. Unlike the VIM, in which experts are provided fixed percentiles to indicate the ranges for the possible values of the QoI, the FIM requires experts to indicate the probability of the true value of the QoI that fall within a fixed range provided to them. For FIM, the clinicians questioned the feasibility of the “chips and bins” method in terms of execution via teleconferences and its time-consuming nature. For VIM, the quartile technique was most preferred by health economists; however, clinicians deemed it implausible for experts to provide accurate judgments on the lower (25th percentile) and upper (75th percentile) quartiles. This caused considerable debate in the team, and eventually, the VIM to elicit on the 2.5th, 50th, and 97.5th percentiles was selected for its lower level of cognitive complexity and more practical application.

Aggregation and Sensitivity Analyses

The individual prior probability distribution elicited for QoIs ( $X_{1}$ , $X_{2}, X_{3}$ ) was aggregated using linear pooling.³² Equal weights were assigned to each expert. The prior distributions were assumed to follow a beta distribution based on past evidence in literature.^36,37 The mean $(μ)$ and variance $(σ^{2})$ for QoI $X_{1}$ , QoI $X_{2}$ , and QoI $X_{3}$ were estimated following the method of moments approach using the beta distribution shape parameters $α$ and $β$ .^38,39

E (X) = \frac{α}{α + β}

Var (X) = \frac{α β}{{(α + β)}^{2} (α + β + 1)}

All analysis was done in R programming language, and R SHELF Package was used.

Other aggregation methods were also considered and discussed with the co-design team. The health economists on the team initially favored the differential mathematical aggregation, following Cooke’s classical method.^30,34 However, no consensus could be reached regarding appropriate calibration questions due to the diversity of the professional backgrounds of the experts. Concerns also arose regarding the time burden imposed on experts to respond to these additional questions. Behavioral aggregation, as outlined in the Sheffield Elicitation Protocol (SHELF), was considered impractical and resource intensive due to coordination challenges. Clinicians also highlighted the difficulty of open sharing in a setting in which experts were unfamiliar with one another. In this context, linear pooling and equal-weighting aggregation emerged as the most practical option, streamlining the process for easier navigation.

Sensitivity analyses were planned to investigate potential heterogeneity or between-expert uncertainty. The first sensitivity analysis examined how elicited estimates vary based on different disciplines or fields of research of the experts. The second sensitivity analysis was conducted to investigate the potential bias of the elicited prior judgments. Based on past elicitation studies conducted, there is a tendency that experts, sometimes, might believe that their estimates are more accurate than is justified; as a result, the elicited confidence interval often tends to be too narrow.^35,36 Therefore, the elicited estimates were also fitted within an 80% confidence interval.

Implementing the Elicitation Exercise

Prior to the individual elicitation, an evidence dossier outlining the aim and context of the study was sent to the experts. During the elicitation sessions, which were conducted by a health economist between May and August 2023, participants received training through a step-by-step demonstration of the elicitation method, followed by a discussion on the semi-structured interview questions. All elicitation sessions were conducted individually with each expert to ensure that the elicited judgments remained confidential and were not shared among the participants. Experts were provided with an answer booklet to record their responses, which also included a questionnaire to be completed and returned to the facilitator. The questionnaire aimed to gather feedback on the overall experience of this co-designed expert elicitation process. The sessions, conducted via videoconferencing, used PowerPoint slides featuring graphical illustrations of a moving scale and prompts to guide the experts’ thought process. The deck of PowerPoint slides used in the elicitation sessions was based on the online template by SHELF: the Sheffield Elicitation Framework (version 4) and Horscroft et el. (2022).^14,40 All sessions were video recorded and transcribed verbatim.

The elicitation materials were developed with incorporation of the feedback from the co-design team. To ensure robustness, 1 clinician and a health economist provided external validation by reviewing these materials for comprehensibility and feasibility. Prior to full implementation, piloting was also conducted by the designated elicitation facilitator, involving clinicians from diverse professional backgrounds, to refine and finalize the elicitation process. Elicitation online software was initially included; however, there was mixed feedback from the piloting clinicians. Some deemed the graphic illustration useful, while others perceived the curves generated from the elicited values as distracting.

Results

Experts’ Quantitative Estimates

Of the 15 invited experts, 12 responded and participated in the elicitation session via videoconferencing. All 12 elicitation sessions, which took place between July 2023 and September 2023, were audio recorded. The mean duration of each videoconference was 45 min. Of the 12 who completed the elicitation session, 11 subsequently completed the elicitation interview and survey questionnaire.

The 12 participants comprised 3 accredited exercise physiologists who specialized in oncology care in private hospitals, 1 accredited exercise physiologist from a private rehabilitation center, 1 physiotherapist who specialized in female cancer rehabilitation in a public hospital, 3 oncology nurses from chemotherapy clinics, 2 oncology-trained ACUMEN research investigators who conducted exercise outcome assessments for participants, senior clinical researcher who specialized in exercise oncology and clinical trials, and 1 professorial researcher who specialized in women’s cancer.

The general characteristics of the elicited estimates of 2.5th percentile, 50th percentile, and 97.5th percentiles for QoI $X_{1}$ , QoI $X_{2}$ , and QoI $X_{3}$ were presented as box and whisker plots (Figure 2a–c).

Figure 2

Box plots of individual prior estimations for quantities of interest (QoIs) $X_{1}, X_{2}, X_{3}$ . (a) Boxplot of prior estimation of relative cardiovascular event (CVD) risk reduction for women with early-stage endometrial cancer. (b) Boxplot of prior estimation of clinicians’ uptake of supervised exercise. (c) Boxplot of prior estimation of cancer consumer uptake of supervised exercise.

No distinct characteristics were observed between different professional backgrounds. However, it was observed that experts who were affiliated with the ACUMEN trial (experts E, J, K, L) provided lower estimates of L, U, and M for all 3 QoIs than the rest of the experts did. Their interquartile ranges of the estimates were also quite narrow compared with the rest of the experts, indicating a high level of agreement and confidence. Sensitivity analyses were conducted separately for the responses of ACUMEN-affiliated and nonaffiliated experts, as outlined below.

During the reflection process, 7 experts revised their estimates for L and U at this point; they were also asked to provide justifications for any changes. Five of them increased the overall range of the interval and reported that this was to reduce their initial overconfidence. However, it was noted that even after adjusting for the credible interval, 3 experts were still not confident of QoI $X_{2}$ and QoI $X_{3}$ , concerned that the true value of X would not fall between the estimated range that they had provided. Most participants took much longer to estimate the median (M). One indicated that he was not entirely confident of estimating the M and would prefer to place it as the midpoint of L and U. It was observed that most experts placed their M closer to their estimated L.

The fitted individual beta distributions for QoI $X_{1}$ , QoI $X_{2}$ , and QoI $X_{3}$ with linear pooling are illustrated in Figure 3a–c.

Figure 3

Individual and aggregated fitted distributions for quantities of interest (QoIs) $X_{1}, X_{2}, and X_{3}$ : (a) fitted distribution of QoI X₁, (b) fitted distribution of QoI X₂, and (c) fitted distribution of QoI X₃.

The probability judgments of L, M, and U were fitted to a beta distribution, and linear pooling was performed. The resulting shape parameters ( $α$ , $β$ ) are presented in Table 1. After performing the method of moments, the expected relative risk reduction of cardiovascular events (CVD) due to regular exercise for women with early-stage endometrial cancer was 36.2%, whereas the expected uptake of an exercise program at a national level was 45.7% for clinicians and 44.6% for eligible participants.

Table 1

Shape Parameters of the Aggregated Beta Distributions

Base Case
Quantities of Interest	Shape 1(α)	Shape 2(β)	Mean (μ)	Standard Deviation ( $σ$ )
$X_{1}$	3.36	5.91	0.362	0.150
$X_{2}$	1.94	2.30	0.457	0.218
$X_{3}$	2.23	2.77	0.446	0.203
Sensitivity analysis 1: ACUMEN-affiliated responses
$X_{1}$	5.23	13.40	0.282	0.101
$X_{2}$	2.10	5.01	0295	0.160
$X_{3}$	4.39	16.60	0.209	0.0861
Sensitivity analysis 2: Non–ACUMEN-affiliated responses
$X_{1}$	5.38	6.26	0.462	0.140
$X_{2}$	4.18	3.01	0.582	0.172
$X_{3}$	4.19	3.01	0.582	0.172
Sensitivity analysis 3: 80% confidence interval
$X_{1}$	1.91	3.18	0.375	0.196
$X_{2}$	2.57	3.15	0.449	0.192
$X_{3}$	1.34	1.64	0.440	0.249

ACUMEN, EnhAnCing treatment oUtcoMes after gynaEcological caNcer.

Results from sensitivity analyses 1 and 2 showed significant variation in the pooled priors for QoI $X_{1}$ , QoI $X_{2}$ , and QoI $X_{3}$ between ACUMEN-affiliated experts and nonaffiliated experts. ACUMEN-affiliated experts consistently provided lower individual prior estimations across all QoIs as compared with nonaffiliated experts, whereas nonaffiliated experts were generally more optimistic in terms of their elicited opinions on the QoIs. When 80% confidence intervals were applied to the elicited estimates, the pooled prior distributions showed minimal variation from the base case. It was also observed that for non–ACUMEN-affiliated experts, their elicited priors for QoIs $X_{2}$ and $X_{3}$ showed negligible differences.

Descriptive Statistics of the Survey Data

Eleven participants completed the survey questionnaire issued on the overall experience of this co-designed expert elicitation. The majority were “confident” or “very confident” (54.6%) about the elicitation estimates they had provided. More than one-third (36.4%) found the co-designed elicitation session slightly difficult to complete, with only 1 reporting it as easy (Table 2). Of the experts, 63.6% had never come across or heard of elicitation before this study. Most (72.7%) found it a credible method of formulating evidence (Table 3).

Table 2

Results of Survey Questions 1 and 2

How confident are you that your answers reflect belief about the true risk reduction of CVD due to exercise?
Statement	Very Confident = 5	Confident = 4	Neutral = 3	Slightly Confident = 2	Not Confident = 1
Frequency	3	3	4	1	0
Percentage (%)	27	27	36	9	0
How easy or difficult do you find the elicitation process to understand?
Statement	Difficult = 5	Slightly Difficult = 4	Neutral = 3	Not Too Easy = 2	Easy = 1
Frequency	0	4	5	1	1
Percentage (%)	0	36	46	9	9

CVD, cardiovascular events.

Table 3

Results of Survey Questions 3 and 4

Had you heard of the concept of structured expert elicitation or taken part in any elicitation session prior to this?	Frequency	Percentage (%)
Yes	4	36
No	7	64
Do you think this is a credible method to determine clinical outcomes that are often not possible to capture in clinical trials?	Frequency	Percentage (%)
Yes	8	73
No	3	27

Thematic Analysis: Perceptions of the Elicitation Process

After transcribing all 11 audio recordings, a simple inductive content analysis was performed. The qualitative part of this study aimed to explore the participating experts’ first-hand experience of SEE and to identify areas for improvement for future SEE practices. Three main themes were identified: experts’ perceptions of SEE, their understanding of the elicitation concept, and their assessments of the credibility of SEE as a form of evidence gathering.

Theme 1: Perceptions of SEE

Most participants indicated that their experiences with the SEE were interesting but somewhat confronting. They found the exercise interesting as it required deep thinking and statistical challenges. One expert noted that the elicitation exercise served as “a good activity to reflect” (expert I). The challenges stemmed from the overall navigation of the SEE questions. The participating experts deemed that the hypothetical scenario provided was comprehensible and reflected the nuances of their clinical practice. Some also noted that if an alternative scenario were provided, or if the scenario had elements that did not align with their daily work, their estimates of likelihood might also change. Hence, it seems that clinical veracity (i.e., scenarios reflecting what is usually encountered in practice) is essential to capturing the most valid estimates from clinical experts in an elicitation exercise. Approximately one-third of participants (36%) reported prior exposure to a form of SEE. While 36.4% of them found the elicitation methodology difficult to understand, more than half were confident in the prior distributions that they estimated with the co-designed SEE. This highlights the value of ensuring an SEE is appropriate to the clinical context it aims to capture.

Theme 2: Understanding elicitation concepts

Despite the concerns of the co-design team that participants would struggle with the terminology and process of the SEE, the participants expressed a good understanding of both, apart from one important element of SEE. The elicitation process that most experts found ambiguous was associated with formulating probabilistic judgments. In particular, they found estimating the 50th percentile (M) confusing. A few experts tended to place their M right in the middle of the L and U or just “smack bang in the middle” (expert A) because they did not truly understand what was asked of them. Many therefore suggested that the explanation of the elicitation concept was best guided by an interactive graphical tool for better visualization, as one expert said it would be “easier if I had something where I could move my margins . . . visually see how it looked” (expert H). This underscored a preference among the experts for elicitation software that enabled clear visualization—a finding that contrasted with the explicit recommendations of the co-design team to the contrary.

According to the experts, the strength of this co-designed SEE was the clarity of the elicitation concept, which they judged was facilitated by providing an accessibly written evidence dossier as prepreparation, the elicitation training provided, and the prompts that were given during the elicitation process. One reported that, “Just doing the pre-reading by myself. I’m like, ‘Oh, I think I know what this is asking’” (expert A). The elicitation training was also well-received by participants. One noted, “I found it very simple from the way you explained it” (expert C). This highlights the clear value of collaboratively developing elicitation materials that target clinicians with clinicians. The general perception was that the co-design process ensured the best possible experience for them, demystified a potentially challenging procedure, and therefore enabled the production of the most valid and reliable elicitation results for this particular health context.

Theme 3: Credibility of SEE as evidence

Many experts believed that, at best, an SEE would play a minor role in producing evidence for practice, particularly compared with the clinical gold standard of RCTs. While one participant did indicate that the process is “a good way of accumulating quick evidence” (expert I), most reported a more typical perception. That is, in relation to “this type of evidence versus a real clinical trial, I would take the clinical trial” (expert H). This might reflect individuals’ limited understanding of the elicitation concept and the intention of the process, but given their statements confirming their general understanding of SEE, it more likely reflects their exposure to discipline-specific research training. Clinicians are rarely, if ever, trained in health economics and its methods. They are usually taught (and therefore most value) RCT methodology, which from a disciplinary perspective produces the best form of evidence to inform practice. A process such as SEE, in which accepted RCT principles such as randomization, blinding, and controlling and comparing groups have no place, is not likely to be valued by those immersed in such principles.

Most experts considered participant selection a key determinant of a SEE’s credibility, which indicates that the careful selection of participants in terms of expertise and previous experiences that was practiced here was warranted. Many emphasized that the quality of an SEE and its outcomes was predicated on the need for the selected experts to be committed to, and knowledgeable about, the larger research problem to ensure that they engage fully with the process and were therefore in a position to make well-informed elicitation decisions. Others emphasized that the number of experts should be large enough “to have a significant number of people with significant experience” (expert H) to ensure credible estimates. This last point is again consistent with the clinical valorization of RCTs, which is driven by the principle that the larger the sample size, the greater natural variation is captured and accounted for and, therefore, the more generalizable and reliable the results.

Discussion

The co-design process described in this study entailed the development of an SEE on 3 uncertain model parameter inputs for a decision-analytic model. The expert elicitation using the collaboratively developed process was subsequently conducted in an Australian clinical setting with 12 experts in the field of exercise oncology. To our knowledge, this is the first SEE conducted in Australia specifically co-designed with clinicians in the field of exercise oncology. Three beta distributions were estimated. These were the probability of risk reduction of CVD of exercise for women treated for endometrial cancer (Mean: 0.362, SD: 0.150), the probability that a clinician would refer these women for exercise therapy (Mean: 0.457, SD: 0.218), and the probability that a woman treated for endometrial cancer would use such a health service upon referral (Mean: 0.446, SD: 0.203).

From the qualitative responses of the participating experts, their overall experience of the co-designed SEE appeared to be positive, and it was generally perceived as a more relevant and comprehensible approach than other SEE methods to which participants had been exposed. Apart from ensuring that the SEE is clinically relevant, the process of administering the co-designed SEE highlighted a number of attributes that could shape the application of SEE in clinical research and might influence future research efforts in this field. First, recruitment of expert participants proved more challenging and time-consuming than expected. Medical personnel (general practitioners and oncologists) exhibited little interest in participating in the elicitation session, predominantly citing lack of time and the inexact nature of the process in a health system context where “hard” evidence (i.e., clinical data derived from RCT) is more valued than the “soft” opinion captured by a SEE. This issue has also been raised in previous elicitation studies.⁴¹ Most importantly, we found that clinicians hesitated to speculate about the potential outcomes of new health care interventions in the absence of the “hard” data that would help them determine the aspect most important to them (i.e., clinical effect).⁹

Second, significant between-expert variation was observed in this co-designed SEE. The prior distributions formulated for all QoIs were much lower from experts who were associated in some way with the ACUMEN trial compared with other clinicians, despite sharing similar professional backgrounds. The qualitative data indicated that this divergence originated in a prevailing sense of pessimism regarding the long-term prospects of exercise adoption among women with gynecological malignancies, given these participants’ hands-on experience in this context. Their skepticism about likely exercise intervention uptake stemmed from the high withdrawal rate and slow recruitment of ACUMEN participants to whom they had been exposed in the course of the trial. Although such disagreements are not an overtly positive outcome, this study highlights how important it is to collect qualitative data after the SEE to account for any dissonance, as this is an effective way to understand the reasons experts rate likelihoods the way they do. The qualitative data enabled the team to attain a rounded understanding of the factors influencing the experts’ overall confidence in the elicited estimates. The qualitative research conducted in this study was aligned with the current recommendations in the literature, since past systematic reviews of expert elicitation have advocated the incorporation of qualitative information to guide the elicitation process and to generate better-quality evidence.^22,28,31,42 Our study is the first to integrate such an approach in exercise oncology to elicit the prior beliefs of experts who were unfamiliar with this research tool, and this was found to be valuable.

Third, linear pooling of individual prior estimations was decided by the co-design team after thorough deliberation. During the co-design meetings, the team decided that assigning differential weights to ACUMEN and non–ACUMEN-affiliated experts could introduce potential bias, as no appropriate “seed” questions could be derived to justify such differentiation. Moreover, assigning greater weight based solely on experts’ affiliation with the ACUMEN trial was considered problematic, given the diverse research backgrounds of the invited experts. Consequently, the decision to adopt an equal weighting aggregation approach was made to ensure robust and unbiased results.

Our study has several limitations. First, given the scope of the study, we did not include an analysis of the impacts of the elicited model parameter inputs on the decision-analytic model. These results will be presented in our forthcoming research. Second, given resource limitations, we were unable to do a direct comparison of our co-design elicitation approach with the established SEE protocols to ascertain its potential for enhancing the elicitation process while addressing practical constraints in clinical research. For future research, if an SEE is conducted virtually, leveraging elicitation software would significantly enhance the appeal of the SEE. In addition, gathering qualitative feedback from the experts after elicitation could further facilitate the implementation of SEE across various research disciplines.

Conclusion

To our knowledge, this is the first co-designed SEE study in exercise oncology. Our study employed a co-design approach to facilitate the practical application of SEE in model-based economic evaluation. The engagement of diverse stakeholders from various disciplines through a series of co-design meetings proved not only effective but also practical in introducing expert elicitation into clinical research. The qualitative findings collected alongside an SEE proved useful in further enhancing its usefulness within clinical research.

Footnotes

Acknowledgements

We would like to acknowledge the ACUMEN team for their invaluable contributions to this study. We would also like to thank Dr Janine Porter-Steele, Dr Kim Edmunds, and Dr Ruvini Hettiarachchi for their advice and feedback on the elicitation materials and, lastly, Dr Angelika M. Stefan for her statistical advice.

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by Medical Research Future Fund APP1199890: EnhAnCing treatment oUtcoMes after gynaEcological caNcer (ACUMEN): Using exercise to promote health after cancer therapy.

Author Contributions

Conceptualization, methodology, and design: YW, AM, HT; data analysis: YW; data interpretation: YW, AM, HT; drafting of manuscript: YW; critical editing of manuscript: YW, AM, HT.

Ethics

Ethical approval was granted from the Royal Brisbane and Women’s Hospital Human Research Ethics Committee (HREC/2020/QRBW/67832): protocol version and date at point of ethical approval v1.2 dated 30.11.2020; protocol version and date at point of submission v1.9 dated 17.07.2023 and ratified by the University of Queensland (2020002937). The study is authorized by the Research Governance Offices of the Royal Brisbane and Women’s Hospital, Mater Hospital and the Wesley Hospital in Brisbane, Australia.

Consent to Participate Statement

All participants gave informed consent to partake in this study, having been fully informed about the objectives and procedures of the research, and voluntarily chose to participate.

Consent for Publication Statement

Participants agreed to the publication of the research results.

ORCID iD

Yufan Wang

References

Ramsey

Willke

Glick

, et al. Cost-effectiveness analysis alongside clinical trials II—an ISPOR good research practices task force report. Value Health. 2015;18(2):161–72.

Briggs

Sculpher

Claxton

. Decision Modelling for Health Economic Evaluation. Oxford (UK): Oxford University Press; 2006.

Ijzerman

Steuten

. Early assessment of medical technologies to inform product development and market access: a review of methods and applications. Appl Health Econ Health Policy. 2011;9:331–47.

Edmunds

Scuffham

Reeves

, et al. Demonstrating the value of early economic evaluation alongside clinical trials: exercise medicine for men with metastatic prostate cancer. Eur J Cancer Care. 2021;30(5):e13479.

Tuffaha

Scuffham

. The Australian managed entry scheme: are we getting it right? Pharmacoeconomics. 2018;36(5):555–65.

Fenwick

Steuten

Knies

, et al. Value of information analysis for research decisions—an introduction: report 1 of the ISPOR value of information analysis emerging good practices task force. Value Health. 2020;23(2):139–50.

Wang

McCarthy

Hayes

, et al. Economic evaluation of exercise interventions for individuals with cancer: a systematic review. Prev Med. 2023;172:107491.

Gordon

Scuffham

Battistutta

Graves

Tweeddale

Newman

. A cost-effectiveness analysis of two rehabilitation support services for women with breast cancer. Breast Cancer Res Treat. 2005;94(2):123–33.

Cao

Postmus

Hillege

Buskens

. Probability elicitation to inform early health economic evaluations of new medical technologies: a case study in heart failure disease management. Value Health. 2013;16(4):529–35.

10.

Iglesias

Thompson

Rogowski

Payne

. Reporting guidelines for the use of expert judgement in model-based economic evaluations. Pharmacoeconomics. 2016;34:1161–72.

11.

Soares

Sharples

Morton

Claxton

Bojke

. Experiences of structured elicitation for model-based cost-effectiveness analyses. Value Health. 2018;21(6):715–23.

12.

Claxton

. Exploring uncertainty in cost-effectiveness analysis. Pharmacoeconomics. 2008;26:781–98.

13.

O’Hagan

. Expert knowledge elicitation: subjective but scientific. Am Stat. 2019;73(suppl 1):69–81.

14.

Oakley

O’Hagan

. SHELF: the Sheffield Elicitation Framework (version 4). 2024. Available from: https://shelf.sites.sheffield.ac.uk/

15.

Gosling

Quigley

Morton

Dias

. SHELF: The Sheffield elicitation framework. In: Elicitation. Vol 261. Springer International Publishing AG; 2018. p. 61–93.

16.

Cooke

Mendel

Thijs

. Calibration and information in expert resolution; a classical approach. Automatica. 1988;24(1):87–93.

17.

Hemming

Burgman

Hanea

McBride

Wintle

. A practical guide to structured expert elicitation using the IDEA protocol. Methods Ecol Evol. 2018;9(1):169–80.

18.

Hanea

Hemming

Nane

. Uncertainty quantification with experts: present status and research needs. Risk Anal. 2022;42(2):254–63.

19.

Williams

Wilson

. A comparison of prior elicitation aggregation using the classical method and SHELF. J R Stat Soc Ser A (Stat Soc). 2021;184(3):920–40.

20.

O’Hagan

Buck

Daneshkhah

, et al. Uncertain Judgements: Eliciting Experts’ Probabilities. London: John Wiley & Sons; 2006.

21.

Butler

Thomas

Pintar

. Systematic review of expert elicitation methods as a tool for source attribution of enteric illness. Foodborne Pathog Dis. 2015;12(5):367–82.

22.

Peel

Jenks

Choudhury

, et al. Use of expert judgement across NICE guidance-making programmes: a review of current processes and suitability of existing tools to support the use of expert elicitation. Appl Health Econ Health Policy. 2018;16(6):819–36.

23.

Clemen

Winkler

. Combining probability distributions from experts in risk analysis. Risk Anal. 1999;19:187–203.

24.

Tan

Baxter

Newell

, et al. Knowledge elicitation for validation of a neonatal ventilation expert system utilising modified Delphi and focus group techniques. Int J Hum Comput Stud. 2010;68(6):344–54.

25.

Grigore

Peters

Hyde

Stein

. Methods to elicit probability distributions from experts: a systematic review of reported practice in health technology assessment. Pharmacoeconomics. 2013;31:991–1003.

26.

Knol

Slottje

van der Sluijs

Lebret

. The use of expert elicitation in environmental health impact assessment: a seven step procedure. Environ Health. 2010;9(1):1–16.

27.

Kuhnert

Martin

Griffiths

. A guide to eliciting and using expert knowledge in Bayesian ecological models. Ecol Lett. 2010;13(7):900–14.

28.

Bojke

Soares

Claxton

, et al. Reference case methods for expert elicitation in health care decision making. Med Decis Making. 2022;42(2):182–93.

29.

Husbands

Jowett

Barton

Coast

. How qualitative methods can be used to inform model development. Pharmacoeconomics. 2017;35(6):607–12.

30.

Campbell

Ezzy

Neil

, et al. A qualitative investigation of the health economic impacts of bariatric surgery for obesity and implications for improved practice in health economics. Health Econ. 2018;27(8):1300–18.

31.

Bojke

Soares

Claxton

, et al. Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study. Health Technol Assess. 2021;25(37):1.

32.

Cooke

. Experts in Uncertainty: Opinion and Subjective Probability in Science. New York; Oxford University Press; 1991.

33.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

34.

Wang

McCarthy

Tuffaha

. Cost-utility analysis of a supervised exercise intervention for women with early-stage endometrial cancer. Support Care Cancer. 2023;31(7):391.

35.

Garthwaite

Kadane

O’Hagan

. Statistical methods for eliciting probability distributions. J Am Stat Assoc. 2005;100(470):680–701.

36.

Soares

Sculpher

Claxton

. Health opportunity costs: assessing the implications of uncertainty using elicitation methods with experts. Med Decis Making. 2020;40(4):448–59.

37.

Fischer

Lewandowski

Janssen

. Estimating unknown parameters in haemophilia using expert judgement elicitation. Haemophilia. 2013;19(5):e282–8.

38.

Hansen

. Large sample properties of generalized method of moments estimators. Econometrica. 1982;50(4):1029–54.

39.

Drummond

. Methods for the Economic Evaluation of Health Care Programmes. Oxford (UK): Oxford University Press; 2015.

40.

Horscroft

Lee

Jankovic

Soares

Bojke

. Structured expert elicitation—example training deck. STEER 2022. 2023. Available from: https://www.york.ac.uk/che/economic-evaluation/steer/

41.

Grigore

Peters

Hyde

Stein

. A comparison of two methods for expert elicitation in health technology assessments. BMC Med Res Methodol. 2016;16:85.

42.

Wilson

ECF

Usher-Smith

Emery

Corrie

Walter

. Expert elicitation of multinomial probabilities for decision-analytic modeling: an application to rates of disease progression in undiagnosed and untreated melanoma. Value Health. 2018;21(6):669–76.

Co-designing a Structured Expert Elicitation with Clinicians to Enhance Health Care Decision Making in Exercise Oncology

Abstract

Background

Objectives

Methods

Results

Conclusions

Highlights

Keywords

Introduction

Method: The Co-design Process

Research Problem: Define the Elicitation Questions

Expert Selection

Elicitation Approach

Aggregation and Sensitivity Analyses

Implementing the Elicitation Exercise

Results

Experts’ Quantitative Estimates

Descriptive Statistics of the Survey Data

Thematic Analysis: Perceptions of the Elicitation Process

Theme 1: Perceptions of SEE

Theme 2: Understanding elicitation concepts

Theme 3: Credibility of SEE as evidence

Discussion

Conclusion

Footnotes

Acknowledgements

Author Contributions

Ethics

Consent to Participate Statement

Consent for Publication Statement

ORCID iD

References