The Voting Scheme Determines the Winner. Considering Possible Options for Designing a Delphi Study

Abstract

The Delphi technique is an important tool for structuring a group communication process among key stakeholders to inform the development of an intervention. However, when designing a Delphi study the scarcity of information on different voting schemes to analyse key stakeholders’ responses became evident, despite the fact that selecting the voting scheme a priori is paramount, since the selected method can influence the outcome. To fill that gap in literature, this article introduces key concepts derived from social choice theory to inform Delphi design. This includes optimising collective decisions by gaining deeper understandings of key stakeholders’ preferences, reducing avoidable unhappiness and minimising possibilities of insincere voting strategies. Four voting schemes (majority, approval, Borda, range) are presented and their implications are discussed. Since the purpose of each Delphi study is unique, we aim to increase awareness of the differential utility of voting schemes and encourage researchers to make an informed choice.

Keywords

mixed methods methods in qualitative inquiry qualitative evaluation qualitative meta-analysis/synthesis community based research

Introduction

Health behaviour interventions can be effective approaches to prevent or manage chronic conditions (Araújo-Soare et al., 2018; Riegel et al., 2017). Such interventions are often complex and influenced by numerous factors, including the context in which the intervention will be implemented (Schloemer & Schröder-Bäck, 2018). To add to this complexity, factors which might be a facilitator in one implementation context could be a barrier in another (May et al., 2016). Even if an intervention has a strong theoretical underpinning, it can still fail to have an impact if contextual factors are disregarded (O’Cathain et al., 2019). To successfully implement a health behaviour intervention, it is crucial to understand the relationship between intervention and context (Waltz et al., 2019).

To address this issue, the Medical Research Council advises including key stakeholders to inform the development and implementation of interventions (Craig et al., 2018). Key stakeholders represent relevant professions or groups, are affiliated to a particular setting or work field and possess clinical and/or academic expertise (Jünger et al., 2017). They can provide valuable insights into the barriers and facilitators of intervention implementation by considering contextual factors based on practical experience (Duggleby & Williams, 2016). The selection of these key stakeholders (=experts) needs to be executed carefully and based on valid criteria. Strategies to identify relevant experts include peer nomination, potential political impact of experts, personal involvement in the topic, external cues such as years of job experience, self-assessment of expertise, past performance of experts and knowledge and psychological assessment of experts (Mauksch et al., 2020). Besides individual expertise, diversity among the selected experts regarding their demography, personal beliefs, values and experiences is considered important (Spickermann et al., 2014).

An effective methodological approach for structuring a group communication process among key stakeholders on a given topic is the Delphi technique (Linstone & Turoff, 2011; Waltz et al., 2019). It is particularly relevant for complex interventions, where there could be a higher level of varying opinions among participating key stakeholders, but a degree of consensus must still be obtained to guide intervention implementation (Revez et al., 2020). Using a questionnaire, stakeholders can express their thoughts on predetermined themes by employing the Delphi technique. The benefit of this approach is that viewpoints from all parties involved are brought together without being influenced by group dynamics like social pressure or group conformity, and regardless of the participants' locations. The Delphi technique is a scientific method consisting of three major phases (i.e. preparing, conducting, and analyzing) used to aggregate opinions of experts through an iterative process, where the experts remain anonymous to each other (Murphy et al., 1998; Beiderbeck et al., 2021). It uses several survey rounds, with each round being based on the responses from the previous round until a predetermined level of agreement is achieved (Jünger et al., 2017; Revez et al., 2020).

The Delphi technique is particularly helpful when developing health behaviour interventions and has been successfully used in prior studies. As it is crucial to understand the relationship between intervention and context when implementing health interventions, it is advised to involve key stakeholders during the design stage of an intervention (Eldredge et al., 2016). Within health research, the Delphi technique has been employed, for example, to establish consensus on barriers on the usage of wearables (Neumann et al., 2022), on barriers and facilitators for interventions in a primary care setting (de Manincor et al., 2015), to determine elements for mental health services from the transition between child- and adulthood (Cleverley et al., 2021) and the viability of workplace health promotion interventions (Perry et al., 2017).

Besides health research, Delphi studies are used for a wide range of purposes such as forecasting, technology, environment and social policy (de Loë et al., 2016; Revez et al., 2020; Schieber et al., 2017; van Lier et al., 2018). Furthermore, the Delphi method is a useful tool for theory development (Brady, 2015). Delphi studies are often referred to as mixed methods design. Their methodological scope ranges from a primarily qualitative scope to a larger focus on the quantitative aspect (Brady, 2015). Early applications of the Delphi technique in the 1950s and 60s go back to the military and technological environment (Gupta & Clarke, 1996). These studies were conducted by the RAND company and first published in 1963 (Dalkey & Helmer, 1963). From then on, the Delphi technique became increasingly popular (Gupta & Clarke, 1996).

Within the context of our study, we applied the Delphi technique following qualitative interviews with key stakeholders for a heart failure (HF) behaviour change intervention (Herber et al., 2021; Whittal et al., 2021). So far, self-care interventions for improving health outcomes for patients with HF have shown mixed results (Jonkman et al., 2016), which might be due to the fact that these interventions were not theory-based or not well described for certain contexts. To replicate an intervention, it is crucial that specific criteria for the intervention description are met. Davidson et al. (2003) identified eight descriptors for this objective (content, interventionist, target population, location, mode of delivery, format, duration, and intensity). With the support of key stakeholders (i.e. patients, clinicians, policy makers and/or potential funders), the purpose of our Delphi exercise was to achieve consensus on these eight descriptors in order to successfully implement a HF intervention in the German healthcare system. First, qualitative semi-structured interviews with the same key stakeholders were conducted. Interviewees’ responses were included in the Delphi questionnaire and subsequently ranked according to stakeholders’ preferences (Herber et al., 2018, 2021; Whittal et al., 2020a, 2020b). In total, we conducted three Delphi rounds and the key stakeholders decided that HF nurses should deliver the intervention (interventionist), the target population should include patients and carers, the intervention should take place in an outpatient HF clinic (location) and should be a mixture of group and individual training sessions (mode of delivery). Furthermore, a training session (format) should last for 30 minutes (duration) and should occur more frequently in the beginning and less often over time (intensity) (Whittal et al., 2021).

Prior to employing the Delphi technique, we performed an extensive literature search for guidance on the method and its application, and found sufficient information in the literature on most aspects, including feedback (Jünger et al., 2017; Niederberger, Renn & Refstyled, 2019), determining consensus (de Loë et al., 2016; Jünger et al., 2017; Niederberger, Renn & Refstyled, 2019), maintaining anonymity (de Loë et al., 2016; Keeney et al., 2011; Niederberger, Renn & Refstyled, 2019), working with expert panels (de Loë et al., 2016; Jünger et al., 2017; Niederberger, Renn & Refstyled, 2019), study design (de Loë et al., 2016; Jünger et al., 2017; Keeney et al., 2011; Niederberger, Renn & Refstyled, 2019; Sossa et al., 2019) and types of Delphi studies (Jünger et al., 2017; Keeney et al., 2011; Niederberger, Renn & Refstyled, 2019; Sossa et al., 2019). There was, however, one important aspect that we were unable to find with regards to employing the Delphi technique: information on the voting scheme, that is, how stakeholders provide their responses and the corresponding analysis technique.

Arriving at a decision at the end of all survey rounds is necessary, but it is not always possible to elicit the desired consensus among stakeholders on complex topics. This means that some method must be applied in which a decision can be made after the final Delphi round, even if a pre-defined threshold of consensus (e.g. agreement of 75% or higher) has not been reached. The chosen voting scheme and analysis approach can influence the results, making it critical to understand and select this approach carefully prior to conducting the study. Moreover, efforts have been made to reduce cognitive biases within future-oriented Delphi studies by adapting specific Delphi design features (Winkler & Moser, 2016). However, to the best of our knowledge, there is no guidance on this in the literature, although there are calls for improving the clarification around the methodological approaches to Delphi studies (Markmann, Spickermann, von der Gracht, & Brem, 2021; Niederberger & Spranger, 2020). Therefore, this methodological insight aims to address this gap by showing how researchers who intend to conduct a Delphi study can apply methods from social choice theory to inform their selection of a voting scheme. The voting scheme and analysis technique form an important bridge between the qualitative input we derived from the interviews and the need to objectively weigh individual preferences where can be only one final option. In our case, that included, for example, to decide in which location the intervention should be delivered. Therefore, our paper aims to contribute to the research of mixed methods design in Delphi studies. To this end, we introduce different voting schemes and considerations for choosing the most appropriate approach for individual Delphi studies.

Discussion

Deciding on the Voting Scheme

Using an arbitrary voting example, Nurmi (2012) demonstrated that the same sample of participants would come to different results each time without changing their preference, depending on the voting scheme and the analysis technique applied.

To understand the reason behind this, some key concepts from social choice theory need to be understood. On the one hand, there is the aim to aggregate participants’ highest preferences towards available options. An appropriate collective choice should thus be based on agreement. On the other hand, there is a concept of reducing “avoidable unhappiness” (Smith, 2005a). This means that besides looking at participants’ preferred options, the disagreement or lower preference around other options should also be considered. To manage the tension between aggregated preferences and reducing avoidable unhappiness, social choice theory aims to reveal the deeper preference structure of participants, to obtain the most accurate and comprehensive insight into participants’ preferences as possible (Faliszewski et al., 2017). Another aspect to consider when finding an appropriate voting scheme is the aim to inhibit strategic voting as much as possible. Strategic voting is insincere voting that does not follow an individual’s genuine preference structure. Instead, participants might disregard certain options that they would otherwise select, in order to give other promising options an advantage.

In the following sections, we provide an overview of four different voting schemes derived from social choice theory that could be used within the scope of a Delphi study: (1) majority voting, (2) approval voting, (3) Borda voting, and (4) range voting. Advantages and disadvantages for each of the four voting schemes are discussed. We follow each description with an example based on our Delphi study, which aimed to determine contextual factors for a heart failure self-care intervention, in order to demonstrate that different voting schemes lead to different outcomes. In the fictitious example provided, ten participants were asked to rank the primary location (from most preferred to least preferred) where the intervention should be delivered (Whittal et al., 2021). Each example uses the same number of participants and preferences for the delivery location, with the only difference being the type of voting scheme applied.

Majority Voting

The most commonly used voting scheme is the majority vote, also known as ‘plurality’ vote. In majority voting, each participant selects their most preferred option out of all given options. By aggregating these votes, the winner option is determined as that which received the majority of the votes (Faliszewski et al., 2017; Szpiro, 2010). That means, however, that researchers do not obtain any further insights into participants’ deeper preference structures, i.e., no understanding is acquired of participants’ preferences besides their most preferred option. As a result, majority voting can be prone to avoidable unhappiness. Strategic voting is also a risk; participants might select an option other than their true preference if they suspect that the selected option might be popular among other participants. Ideally, reaching consensus among stakeholders should not only be based on measuring the highest agreement towards a preferred option, but also by reducing the avoidable unhappiness of participants (Nurmi, 2012). In the case of a majority vote, where participants select one option and the option with the most votes wins, avoidable unhappiness cannot be prevented because only the surface of the participants’ deeper preference structure is visible (Nurmi, 2012; Szpiro, 2010).

In the example below, a case of majority voting is portrayed. A total of ten participants selected their ordered preferences for the primary location where the intervention should take place, i.e., GP practice, hospital, cardiologist, at home and rehabilitation clinic. The green box shows the top choice in the list, and to the left is the number of people who voted for that location as their preferred option. Since four participants selected ‘GP practice’ as their preferred option, the ‘GP practice’ is the selected location where the intervention should be delivered. At the same time, for five participants the ‘GP practice’ is the least favourable outcome. This circumstance is, however, not reflected in the aggregated preferences following the voting scheme majority voting and thus is an example of avoidable unhappiness.

Example of majority voting.

Approval Voting

An alternative voting scheme to be considered in Delphi studies is approval voting, a method which provides a deeper insight into the preference structure of participants (Brams & Fishburn, 1978). Participants can decide whether they ‘approve’ of each available option or not. They can vote for as many alternatives as they want and the option cumulating most approval wins (Faliszewski et al., 2017; Szpiro, 2010).

Hence, we gain a deeper understanding of which option receives the sum of approval of most participants. This can help prevent avoidable unhappiness, because besides selecting only their most preferred option, participants indicate which options are tolerable for them.

Approval voting can be an appropriate voting scheme if the purpose of a Delphi study is to select a previously defined number of options. A further advantage of the method is that it is quite simple and thus user friendly. Like majority voting, this method also faces the risk of strategic voting. This means participants could select preferred options for approval that they think other participants will also select, even though they might have approved of other additional options if they had voted sincerely (Endriss, 2012).

In the example below, a case of approval voting is portrayed. The green letters show the names that were ‘approved’, and to the left is the number of people, who selected those delivery locations as their preference in this order. The names in black were not ‘approved’. For example, in the first sequence of delivery locations, four participants approved of ‘at home’; in the second sequence three participants approved of ‘at home’ and in the third sequence, a further two. Since ‘at home’ received the most approvals (9), ‘at home’ is the selected location. Here, ‘GP practice’ is lacking approval from a total of six participants, while being approved by only four participants who have a strong preference towards this option. However, their strength of preference is not displayed by the approval voting scheme.

Example of approval voting.

Borda Voting

A further voting scheme is Borda voting. With this method, participants must rank all available options according to their preference. Points are assigned based on the inverted values of the given ranks, that is, the number of answer options minus a given rank. Borda voting is an appropriate voting scheme for a ranking-type Delphi study if consensus must be elicited for a single option. It provides even deeper insights into the participants’ preference structures than approval voting, however, it does hold some disadvantages. First, it is not very practical if there are too many options, because participants often struggle to rank them (Azzara, 2010). Second, as all voting schemes, Borda voting still leaves room for strategic voting, although less than approval voting. Strategic voting in this scheme may manifest as participants placing options they perceive as potentially threatening to their most preferred option in a lower rank to better ensure they are disregarded (Ludwin, 1978). However, they cannot disregard all other options because all options need to be ranked. Thus, they might be encouraged to vote more sincerely and place options according to their sincere preference structure.

In the following example Borda voting is shown. Our ten fictitious participants selected their preferences for the five different delivery locations by ranking them. Each option then received a point according to its inverse rank, the “Borda Count" (Szpiro, 2010). For instance, there are five delivery locations to choose from, corresponding to points of 0–4 (4 being most preferred). In the top line, ‘GP practice’ was ranked number 1, so received its inverse points (4). In the second line, it was ranked last, so received its inverse points (0), and so on. Points are then multiplied by the number of participants who selected them and added together to give a sum score. The option with the highest sum score wins. Since ‘hospital’ received the most points (29), ‘hospital’ is the selected location. Despite being the most preferred option of one voter only, ‘hospital’ is ranked relatively high by all other voters, i.e., as their second or third most preferred option. Therefore, based on the Borda voting scheme, ‘hospital’ is considered to best reflect the voters’ aggregated preferences.

Example of Borda voting.

Range Voting

Finally, range voting (Smith, 2000) requires participants to assign each answer option a value on a given scale according to their preference, e.g. 80 out of 100. This approach enables further knowledge to be acquired, that is, the distance between response options. Of the voting schemes discussed, range voting provides the most insight into the deeper preference structure of participants and can best reduce avoidable unhappiness. The advantages of range voting centres on its flexibility and accurate reflection of participants’ opinions. Range voting leaves participants the flexibility to assign any number of points to each option without the restriction of placing them in a rank order. If participants do have a preference structure that follows a ranking scheme, range voting can still accurately represent this. Disadvantages include the complexity of the method and susceptibility to strategic voting resulting from insincere response behaviours to boost or lower certain response options (Smith, 2005b). In terms of complexity, participants could be overwhelmed by the number of choices they are required to make and might find it difficult to reflect on the value they wish to assign to every single option. Furthermore, an overly complex voting scheme might increase dropout rates by discouraging participants. This implies that whenever the number of options is relatively high, range voting might not be the preferred voting scheme. However, if the cognitive capacity and motivation of the participants is sufficient, the complexity might have a less negative impact, so in some cases range voting could still be suitable despite a higher number of options. To decide whether range voting is a suitable voting scheme for their study, researchers should evaluate their participants’ cognitive capacities and motivation. If, for example, the participants include individuals with dementia, the study design might be adapted by providing visual aids to improve their decision-making capacity (Chang & Bourgeois, 2020). Efforts for a user-friendly presentation might mitigate the disadvantages of range voting.

Example of range voting.

Referring to our example, ten participants selected their preferences for the key delivery location, this time by rating them on a scale of 0–100 (100 being most preferred). The numbers selected for each location are then multiplied by the number of participants who selected them and added together to give a sum score. The option with the highest sum score wins. Cardiology practice received the highest range score (460), therefore the cardiology practice is the selected location. In this example, the two participants who assigned a score of 100 to ‘cardiology practice’ expressed a stronger preference towards one option than all other participants, while the four voters in the first line seemed to have equally strong preferences among their three most preferred options. Yet, they do not seem to be completely satisfied with any of these options, based on the fact that they assigned all of them only 50 points.

Reviewing the above examples, it becomes evident that the voting scheme plays a significant role in determining the winner. Every example uses the same delivery locations as possible response items and has the same intended outcome; to identify consensus on the most preferred option. Depending on the voting scheme selected, however, the results change. With the majority vote, the ‘GP practice’ is the selected location where the intervention should be delivered, with the approval vote, it is ‘at home’, with the Borda vote, it is ‘hospital’, and with the range vote, the ‘cardiology practice’ is the selected location.

It is therefore of paramount importance to understand the voting scheme and analysis options, and to choose wisely in the context of the particular Delphi study. Similar to determining a consensus threshold prior to conducting the study, the voting scheme requires the same a priori decisions and reporting to achieve meaningful results and avoid researcher bias.

Conclusion

Of all voting schemes described, range voting appears to be the most comprehensive in taking into account most and least preferred options, and thus has the highest potential to reduce avoidable unhappiness yet tends to become complex when the number of options is high. Majority voting, even though popular, provides only shallow insights into the preference structure and hence is not recommended in most cases. Approval voting allows deeper insights and matches a context, where approval of the most participants possible is considered the most important outcome. Borda voting provides even deeper insights and seems to have a good trade-off between accuracy of reflecting preference structures without becoming overly complex. Also, it is the most effective voting scheme to mitigate effects of strategic voting. However, there are numerous other factors to consider when deciding on the voting scheme for a Delphi study and there is no one best solution that fits every situation. Rather, we encourage researchers to reflect on the purpose and rationale of their study and consider the characteristics of different voting schemes such as the complexity, reduction of avoidable unhappiness and context appropriateness. Researchers should make an informed a priori decision on which voting scheme is most suitable for their individual Delphi study.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Deutsche Forschungsgemeinschaft; HE 7352/1–2. The ACHIEVE study was supported by the German Research Foundation.

ORCID iD

Isabell Ehringfeld

References

Araújo-Soare

Hankonen

Presseau

Rodrigues

Sniehotta

F. F.

(2018). Developing behavior change interventions for self-management in chronic illness. European Psychologist, 24(1), 17–25. https://doi.org/10.1027/1016-9040/a000330

Azzara

C. V.

(2010). Questionnaire design for business research: Beyond linear thinking - an interactive approach. Tate Publishing.

Beiderbeck

Frevel

von der Gracht

H. A.

Schmidt

S. L.

Schweitzer

V. M.

(2021). Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements. MethodsX, 8, 101401. https://doi.org/10.1016/j.mex.2021.101401

Brady

S. R.

(2015). Utilizing and adapting the Delphi method for use in qualitative research. International Journal of Qualitative Methods, 14(5), 1609406915621381. https://doi.org/10.1177/1609406915621381

Brams

S. J.

Fishburn

P. C.

(1978). Approval voting. The American Political Science Review, 72(3), 831–847. https://doi.org/10.2307/1955105

Chang

W. Z. D.

Bourgeois

M. S.

(2020). Effects of visual aids for end-of-life care on decisional capacity of people with dementia. American Journal of speech-language Pathology, 29(1), 185–200. https://doi.org/10.1044/2019_AJSLP-19-0028

Cleverley

McCann

O’Brien

Davies

Bennett

Brennenstuhl

Courey

Henderson

Jeffs

Miller

Pignatiello

Rong

Rowland

Stevens

Szatmari

(2021). Prioritizing core components of successful transitions from child to adult mental health care: A national Delphi survey with youth, caregivers, and health professionals. European Child & Adolescent Psychiatry, 31(11), 1–14. https://doi.org/10.1007/s00787-021-01806-6

Craig

di Ruggiero

Frohlich

K. L.

Mykhalovskiy

White

(2018). Taking account of context in population health intervention research: Guidance for producers, users and funders of research. National Institute for Health Research. https://doi.org/10.3310/CIHR-NIHR-01

Dalkey

Helmer

(1963). An experimental application of the Delphi method to the use of experts. Management Science, 9(3), 458–467. https://doi.org/10.1287/mnsc.9.3.458

10.

Davidson

K. W.

Goldstein

Kaplan

R. M.

Kaufmann

P. G.

Knatterud

G. L.

Orleans

C. T.

Spring

Trudeau

K. J.

Whitlock

E. P.

(2003). Evidence-based behavioral medicine: What is it and how do we achieve it? Annals of Behavioral Medicine, 26(3), 161–171. https://doi.org/10.1207/S15324796ABM2603_01

11.

de Loë

R. C.

Melnychuk

Murray

Plummer

(2016). Advancing the state of policy Delphi practice: A systematic review evaluating methodological evolution, innovation, and opportunities. Technological Forecasting and Social Change, 104(1), 78–88. https://doi.org/10.1016/j.techfore.2015.12.009

12.

de Manincor

Bensoussan

Smith

Fahey

Bourchier

(2015). Establishing key components of yoga interventions for reducing depression and anxiety, and improving well-being: A Delphi method study. BMC Complementary and Alternative Medicine, 15(1), 85. https://doi.org/10.1186/s12906-015-0614-7

13.

Duggleby

Williams

(2016). Methodological and epistemological considerations in utilizing qualitative inquiry to develop interventions. Qualitative Health Research, 26(2), 147–153. https://doi.org/10.1177/1049732315590403

14.

Eldredge

L. K. B.

Markham

C. M.

Ruiter

R. A.

Fernï

M. E.

Kok

Parcel

(2016). Planning health promotion programs: An intervention mapping approach. Wiley.

15.

Endriss

(2012). Sincerity and manipulation under approval voting. Theory and Decision, 74(3), 335–355. https://doi.org/10.1007/s11238-012-9301-z

16.

Faliszewski

Skowron

Slinko

Talmon

(2017). Multiwinner voting: A new challenge for social choice theory. In Endrisse

(Ed.), Trends in computational social choice. COST. European Cooperation in Science & Technology. https://www.mimuw.edu.pl/∼ps219737/papers/chapter2-fal-sko-sli-tal.pdf

17.

Gupta

U. G.

Clarke

R. E.

(1996). Theory and applications of the Delphi technique: A bibliography (1975–1994). Technological Forecasting and Social Change, 53(2), 185–211. https://doi.org/10.1016/S0040-1625(96)00094-7

18.

Herber

O. R.

Atkins

Störk

Wilm

(2018). Enhancing self-care adherence in patients with heart failure: A study protocol for developing a theory-based behaviour change intervention using the COM-B behaviour model (ACHIEVE study). BMJ Open, 8(9), Article e025907. https://doi:10.1136/bmjopen-2018-025907

19.

Herber

O. R.

Ehringfeld

Steinhoff

Whittal

(2021). Identifying relevant factors for successful implementation into routine practice: Expert interviews to inform a heart failure self-care intervention (ACHIEVE study). BMC Health Services Research, 21(585), 585. https://doi.org/10.1186/s12913-021-06596-w

20.

Jonkman

N. H.

Westland

Groenwold

R. H.

Ågren

Atienza

Blue

Bruggink-André de la Porte

P. W. F.

DeWalt

D. A.

Hebert

P. L.

Heisler

Jaarsma

Kempen

G. I. J. M.

Leventhal

M. E.

Lok

D. J. A.

Mårtensson

Muñiz

Otsu

Peters-Klimm

Rich

M. W.

Jaarsma

(2016). Do self-management interventions work in patients with heart failure? An individual patient data meta-analysis. Circulation, 133(12), 1189–1198. https://doi.org/10.1161/CIRCULATIONAHA.115.018006

21.

Jünger

Payne

S. A.

Brine

Radbruch

Brearley

S. G.

(2017). Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliative Medicine, 31(8), 684–706. https://doi.org/10.1177/0269216317690685

22.

Keeney

Hasson

McKenna

(2011). The Delphi technique in nursing and health research. Wiley. https://doi.org/10.1002/9781444392029

23.

Linstone

Turoff

(2011). Delphi: A brief look backward and forward. Technological Forecasting and Social Change, 78(9), 1712–1719. https://doi.org/10.1016/j.techfore.2010.09.011

24.

Ludwin

W. G.

(1978). Strategic voting and the borda method. Public Choice, 33(1), 85–90. https://doi.org/10.1007/BF00123946

25.

Markmann

Spickermann

von der Gracht

H. A.

Brem

(2021). Improving the question formulation in Delphi‐like surveys: Analysis of the effects of abstract language and amount of information on response behavior. Futures & Foresight Science, 3(1), Article e56. https://doi.org/10.1002/ffo2.56

26.

Mauksch

von der Gracht

H. A.

Gordon

T. J.

(2020). Who is an expert for foresight? A review of identification methods. Technological Forecasting and Social Change, 154, 119982. https://doi.org/10.1016/j.techfore.2020.119982

27.

May

C. R.

Johnson

Finch

(2016). Implementation, context and complexity. Implementation Science, 11(1), 141. https://doi.org/10.1186/s13012-016-0506-3

28.

Murphy

Black

Lamping

McKee

Sanderson

Askham

Marteau

(1998). Consensus development methods, and their use in clinical guideline development. Health Technology Assessment (Winchester, England), 2(3), 88. https://doi.org/10.3310/hta2030https://fa.testingtreatments.org/wp-content/uploads/sites/4/2016/11/Murphy-et-al-1998-HTA-Consensus-development-methods.pdf/

29.

Neumann

Tiberius

Biendarra

(2022). Adopting wearables to customize health insurance contributions: A ranking-type Delphi. BMC Medical Informatics and Decision Making, 22(1), 1–7. https://doi.org/10.1186/s12911-022-01851-4

30.

Niederberger

Renn

(Eds.), (2019). Delphi-Verfahren in den Sozial-und Gesundheitswissenschaften: Konzept, Varianten und Anwendungsbeispiele. Springer-Verlag.

31.

Niederberger

Spranger

(2020). Delphi technique in health sciences: A map. Frontiers in Public Health, 8, 457. https://doi.org/10.3389/fpubh.2020.00457

32.

Nurmi

(2012). Comparing voting systems. Springer Science & Business Media.

33.

O’Cathain

Croot

Duncan

Rousseau

Sworn

Turner

K. M.

Yardley

Hoddinott

(2019). Guidance on how to develop complex interventions to improve health and healthcare. BMJ Open, 9(8), Article e029954. https://doi.org/10.1136/bmjopen-2019-029954

34.

Perry

Nicholls

Duffield

Gallagher

(2017). Building expert agreement on the importance and feasibility of workplace health promotion interventions for nurses and mid-wives: A modified Delphi consultation. Journal of Advanced Nursing, 73(11), 2587–2599. https://doi.org/10.1111/jan.13345

35.

Revez

Dunphy

Harris

Mullaly

Lennon

Gaffney

(2020). Beyond forecasting: Using a modified Delphi method to build upon participatory action research in developing principles for a just and inclusive energy transition. International Journal of Qualitative Methods, 19, 1–12. https://doi.org/10.1177/1609406920903218

36.

Riegel

Moser

D. K.

Buck

H. G.

Dickson

V. V.

Dunbar

S. B.

Lee

C. S.

Lennie

T. A.

Lindenfeld

Mitchell

J. E.

Treat‐Jacobson

D. J.

Webber

D. E.

on behalf of the American Heart Association Council on Cardiovascular and Stroke Nursing; Council on Peripheral Vascular Disease; and Council on Quality of Care and Outcomes Research (2017). American heart association Council on cardiovascular and stroke nursing; Council on peripheral vascular disease; and Council on quality of care and outcomes ResearchSelf‐care for the prevention and management of cardiovascular disease and stroke: A scientific statement for healthcare professionals from the American heart association. [Journal of the American Heart Association Electronic Resource], 6(9), Article e006997. https://doi.org/10.1161/JAHA.117.006997

37.

Schieber

A. C.

Kelly-Irving

Génolini

J. P.

Membrado

Tanguy

Fabre

Lang

for the INTERMEDE Group (2017). Integrating multidisciplinary results to produce new knowledge about the physician–patient relationship: A methodology applied to the intermede project. Journal of Mixed Methods Research, 11(2), 174–201. https://doi.org/10.1177/1558689815588643

38.

Schloemer

Schröder-Bäck

(2018). Criteria for evaluating transferability of health interventions: A systematic review and thematic synthesis. Implementation Science, 13(1), 1–17. https://doi.org/10.1186/s13012-018-0751-8

39.

Smith

(2005a). Bayesian regret for dummies. https://www.rangevoting.org/BayRegDum.html (Accessed 19 January 2022).

40.

Smith

W. D.

(2000). Range voting. https://rangevoting.org/WarrenSmithPages/homepage/rangevote.pdf (Accessed 19 January 2022).

41.

Smith

W. D.

(2005b). Strategy in range voting and COAF voting systems. https://rangevoting.org/RVstrat.pdf (Accessed 19 January 2022).

42.

Sossa

J. W. Z.

Halal

Zarta

R. H.

(2019). Delphi method: Analysis of rounds, stakeholder and statistical indicators. Foresight, 21(5), 525–544. https://doi.org/10.1108/fs-11-2018-0095

43.

Spickermann

Zimmermann

von der Gracht

H. A.

(2014). Surface-and deep-level diversity in panel selection—exploring diversity effects on response behaviour in foresight. Technological Forecasting and Social Change, 85, 105–120. https://doi.org/10.1016/j.techfore.2013.04.009

44.

Szpiro

(2010). Numbers rule: The vexing mathematics of democracy, from plato to the present. Princeton University Press.

45.

van Lier

L. I.

Bosmans

J. E.

Van Hout

H. P. J.

Mokkink

L. B.

van den Hout

W. B.

de Wit

G. A.

Dirksen

C. D.

Nies

H. L. G. R.

Hertogh

C. M. P. M.

van der Roest

H. G.

(2018). Consensus-based cross-European recommendations for the identification, measurement and valuation of costs in health economic evaluations: A European Delphi study. European Journal of Health Economics, 19(7), 993–1008. https://doi.org/10.1007/s10198-017-0947-x

46.

Waltz

T. J.

Powell

B. J.

Fernández

M. E.

Abadie

Damschroder

L. J.

(2019). Choosing implementation strategies to address contextual barriers: Diversity in recommendations and future directions. Implementation Science, 14(1), 1–15. https://doi.org/10.1186/s13012-019-0892-4

47.

Whittal

Atkins

Herber

O. R.

(2020a). What the guide does not tell you: Reflections on and lessons learned from applying the COM-B behavior model for designing real life interventions. Translational behavioral medicine. https://doi.org/10.1093/tbm/ibaa116

48.

Whittal

Ehringfeld

Steinhoff

Herber

O. R.

(2021). Determining contextual factors for a heart failure self-care intervention: A consensus Delphi study (achieve). Health Education & Behavior. https://doi.org/10.1177/10901981211043116

49.

Whittal

Störk

Riegel

Herber

O. R.

(2020b). Applying the COM-B behaviour model to overcome barriers to heart failure self-care: A practical application of a conceptual framework for the development of complex interventions (ACHIEVE study). European Journal of Cardiovascular Nursing, 20(3), 261–267. https://doi:10.1177/1474515120957292

50.

Winkler

Moser

(2016). Biases in future-oriented Delphi studies: A cognitive perspective. Technological Forecasting and Social Change, 105, 63–76. https://doi.org/10.1016/j.techfore.2016.01.021