A Descriptive Comparison of Scoring Methods and Criteria for Evaluating Consensus in Modified Delphi Methods

Abstract

Background and Purpose

Modified Delphi methods are increasingly used to develop healthcare pathways with input from people with lived experience (PWLE) and clinicians/others. However, guidance on consensus analysis in this context remains limited. We examined consensus outcomes across different scoring methods and criteria when participants were treated as a single combined group (Objective 1) versus two distinct groups (Objective 2).

Methods

We conducted a secondary analysis of Round 1 data from a project involving PWLE (N = 8) and clinicians/others (N = 51). To assess agreement on 68 Delphi statements, we applied three methods for scoring percentage agreement that differed in how the middle response on a three-point Likert scale (“approve”, “not sure either way”, “do not approve”) was treated. Method 1 excluded the middle response, methods 2 and 3 grouped the middle response with “do not approve”, and “approve”, respectively. We compared consensus rates (% of items reaching consensus) using percentage agreement cutoffs of ≥70%, ≥80%, and ≥90% of participants.

Results

Consensus results varied by participants grouping, treatment of middle response categories, and cutoff criteria. Results from the combined group of PWLE and clinicians/others provided a simplified overview consensus outcome. Treating the participants into as separate groups provide nuanced results.

Conclusion

The analysis of data can change the results from which to draw conclusions and inform practice. Investigators should consider the alignment of each approach with the goals of their Delphi study.

Keywords

Modified Delphi evaluating consensus pathway

Get full access to this article

View all access options for this article.

References

Barrington

Young

Williamson

P. R.

(2021). Patient participation in Delphi surveys to develop core outcome sets: Systematic review. BMJ Open, 11(9), e051066. https://doi.org/10.1136/bmjopen-2021-051066

Beiderbeck

Frevel

von der Gracht

H. A.

Schmidt

S. L.

Schweitzer

V. M.

(2021). Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements. MethodsX, 8(101401), 1–20. https://doi.org/10.1016/j.mex.2021.101401

Best

R. W.

Campbell

J. T.

Edwards

M. C.

Cline

L. L.

(2025). An Overview of the Delphi Method’s Origin, Modifications, and Use to Augment Instrument Development and Data Collection: A Research Note. Journal of International Agricultural and Extension Education, 32(1). https://doi.org/10.4148/2831-5960.1497

Boulkedid

Abdoul

Loustau

Sibony

Alberti

(2011). Using and reporting the delphi method for selecting healthcare quality indicators: A systematic review. PLoS ONE, 6(6), e20476–e20476. https://doi.org/10.1371/journal.pone.0020476

Brookes

S. T.

Macefield

R. C.

Williamson

P. R.

McNair

A. G.

Potter

Blencowe

N. S.

Strong

Blazeby

J. M.

(2016). Three nested randomized controlled trials of peer-only or multiple stakeholder group feedback within Delphi surveys during core outcome and information set development. Trials, 17(1), 409. https://doi.org/10.1186/s13063-016-1479-x

Campbell

Hotchkiss

Bradshaw

Porteous

(1998). Integrated care pathways. British Medical Journal, 316(7125), 133–137. https://doi.org/10.1136/bmj.316.7125.133

Clibbens

Walters

Baird

(2012). Delphi research: Issues raised by a pilot study. Nurse Research, 19(2), 37–44. https://pubmed.ncbi.nlm.nih.gov/22338807/ https://doi.org/10.7748/nr2012.01.19.2.37.c8907

de Loë

R. C.

Melnychuk

Murray

Plummer

(2016). Advancing the state of policy Delphi practice: A systematic review evaluating methodological evolution, innovation, and opportunities. Technological Forecasting and Social Change, 104, 78–88. https://doi.org/10.1016/j.techfore.2015.12.009

Diamond

I. R.

Grant

R. C.

Feldman

B. M.

Moore

A. M.

Wales

P. W.

Pencharz

P. B.

Ling

S. C.

(2014). Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of Clinical Epidemiology, 67(4), 401–409. https://doi.org/10.1016/j.jclinepi.2013.12.002

10.

Eubank

B. H. F.

Lackey

S. W.

Slomp

Werle

J. R.

Kuntze

Sheps

D. M.

(2021). Consensus for a primary care clinical decision-making tool for assessing, diagnosing, and managing shoulder pain in Alberta, Canada. BMC Family Practice, 22(1), 1–20. https://doi.org/10.1186/s12875-021-01544-3

11.

Everink

I. H. J.

van Haastregt

J. C. M.

Kempen

G. I. J. M.

Schols

J. M. G. A.

(2020). Building consensus on an integrated care pathway in geriatric rehabilitation: A modified Delphi study among professional experts. Journal of Applied Gerontology, 39(4), 423–434. https://doi.org/10.1177/0733464818774629

12.

Fernandez

Thompson

Berendonk

Schick-Makaroff

(2022). Mental health care for adults treated with dialysis in Canada: A scoping review. Canadian Journal of Kidney Health and Disease, 9, 1–16. https://doi.org/10.1177/20543581221086328

13.

Gattrell

W. T.

Hungin

A. P.

Price

Winchester

C. C.

Tovey

Hughes

E. L.

van Zuuren

E. J.

Goldman

Logullo

Matheis

Harrison

(2022). ACCORD Guideline for reporting consensus-based methods in biomedical research and clinical practice: A study protocol. Research Integrity and Peer Review, 7(1), 1–10. https://doi.org/10.1186/s41073-022-00122-0

14.

Hasson

Keeney

(2011). Enhancing rigour in the Delphi technique research. Technological Forecasting and Social Change, 78(9), 1695–1704. https://doi.org/10.1016/j.techfore.2011.04.005

15.

Hasson

Keeney

McKenna

(2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008–1015. https://doi.org/10.1046/j.1365-2648.2000.t01-1-01567.x

16.

I. S. S.

Azcoaga-Lorenzo

Akbari

Davies

Khunti

Kadam

U. T.

Lyons

R. A.

McCowan

Mercer

S. W.

Nirantharakumar

Staniszewska

Guthrie

(2022). Measuring multimorbidity in research: Delphi consensus study. BMJ Medicine, 1(1), e000247. https://doi.org/10.1136/bmjmed-2022-000247

17.

Hock Ong

M. E.

Shin

S. D.

Sung

S. S.

Tanaka

Huei-Ming

Song

K. J.

Nishiuchi

Leong

B. S.-H.

Karim

S. A.

Lin

C.-H.

(2013). Recommendations on ambulance cardiopulmonary resuscitation in basic life support systems. Prehospital Emergency Care, 17(4), 491–500. https://doi.org/10.3109/10903127.2013.818176

18.

Hsu

C.-C.

Sandford

B. A.

(2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research, and Evaluation, 12(1), 1–8. https://doi.org/10.7275/pdz9-th90

19.

Keeney

Hasson

McKenna

H. P.

(2001). A critical review of the Delphi technique as a research methodology for nursing. International Journal of Nursing Studies, 38(2), 195–200. https://doi.org/10.1016/S0020-7489(00)00044-4

20.

Keeney

Hasson

McKenna

H. P.

(2011). The Delphi technique in nursing and health research. Wiley-Blackwell. https://onlinelibrary.wiley.com/doi/book/10.1002/9781444392029

21.

Khodyakov

Grant

Denger

Kinnett

Martin

Peay

Coulter

(2020). Practical considerations in using online modified-Delphi approaches to engage patients and other stakeholders in clinical practice guideline development. The Patient - Patient-Centered Outcomes Research, 13(1), 11–21. https://doi.org/10.1007/s40271-019-00389-4

22.

Kroenke

Spitzer

R. L.

Williams

J. B. W.

(2001). The PHQ-9. Journal of General Internal Medicine, 16(9), 606–613. https://doi.org/10.1046/j.1525-1497.2001.016009606.x

23.

Lange

Kopkow

Lützner

Günther

K.-P.

Gravius

Scharf

H.-P.

Stöve

Wagner

Schmitt

(2020). Comparison of different rating scales for the use in Delphi studies: Different scales lead to different consensus and show different test-retest reliability. BMC Medical Research Methodology, 20(1), 28. https://doi.org/10.1186/s12874-020-0912-8

24.

Aroojis

Mulpuri

Shea

K. G.

Schaeffer

E. K.

(2021). Development of a DDH care pathway for India: A study methodology to guide similar efforts in other countries and for other conditions. Indian Journal of Orthopaedics, 55(6), 1549–1558. https://doi.org/10.1007/s43465-021-00534-y

25.

Nasa

Jain

Juneja

(2021). Delphi methodology in healthcare research: How to decide its appropriateness. World Journal of Methodology, 11(4), 116–129. https://doi.org/10.5662/wjm.v11.i4.116

26.

NHS England. (2022). 12-Item Short Form Survey (SF-12). https://www.england.nhs.uk/uploads/2022/12.

27.

Niederberger

Homberg

(2023). Argument-based QUalitative analysis strategy (AQUA) for analyzing free-text responses in health sciences delphi studies. MethodsX, 10(10256), 1–14. https://doi.org/10.1016/j.mex.2023.102156

28.

Niederberger

Spranger

(2020). Delphi technique in health sciences: A map. Frontiers in Public Health, 8. https://doi.org/10.3389/fpubh.2020.00457

29.

Palliative care Outcome Scale. (2012). IPOS-Renal and translations. https://pos-pal.org/maix/ipos-renal.php#renglish .

30.

Panella

Marchisio

Di Stanislao

(2003). Reducing clinical variations with clinical pathways: Do pathways work? International Journal for Quality in Health Care, 15(6), 509–521. https://doi.org/10.1093/intqhc/mzg057

31.

Pedrosa Carrasco

A. J.

Berlin

Betker

Riera-Knorrenschild

von Blanckenburg

Seifart

(2022). Developing a care pathway for hospital-based advance care planning for cancer patients: A modified Delphi study. European Journal for Cancer Care, e13756. https://doi.org/10.1111/ecc.13756

32.

Powell

(2003). The Delphi technique: Myths and realities. Journal of Advanced Nursing, 41(4), 376–382. http://www.itsuandi.org/itsui/downloads/Itsui_Materials/The_Delphi_Technique_Myths_and_Realities.pdf https://doi.org/10.1046/j.1365-2648.2003.02537.x

33.

Schick-Makaroff

Berendonk

Overwater

Streith

Lee

Escoto

Cukor

Klarenbach

Sawatzky

(2022). How are Albertans “adjusting to and coping with” dialysis? A cross-sectional survey. Canadian Journal of Kidney Health and Disease, 9, 20543581221118436. https://doi.org/10.1177/20543581221118436

34.

Schick-Makaroff

Berendonk

Salum

Yoeun

Wichart

Armstrong

Thompson

Elliott

Lee

Smith

Reintjes

Fillier

Klarenbach

Sawatzky

(2025). Developing and tailoring a person-centred pathway for mental health care for people receiving dialysis. Nephron, 149(7), 392–410. https://doi.org/10.1159/000544058

35.

Shang

(2023). Use of Delphi in health sciences research: A narrative review. Medicine (Baltimore), 102(7), e32829. https://doi.org/10.1097/md.0000000000032829

36.

Shaw

Price

Clayton

Grimison

Shaw

Rankin

Butow

Shaw

J. M.

Price

M. A.

Clayton

J. M.

Butow

P. N.

(2016). Developing a clinical pathway for the identification and management of anxiety and depression in adult cancer patients: An online Delphi consensus process. Supportive Care in Cancer, 24(1), 33–41. https://doi.org/10.1007/s00520-015-2742-5

37.

Sossa

J. W. Z.

Halal

Raul Hernandez

(2019). Delphi method: Analysis of rounds, stakeholder and statistical indicators. Foresight: The Journal of Futures Studies, Strategic Thinking and Policy, 21(5), 525–544. https://doi.org/10.1108/FS-11-2018-0095

38.

Spitzer

R. L.

Kroenke

Williams

J. B.

Löwe

(2006). A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine, 166(10), 1092–1097. https://doi.org/10.1001/archinte.166.10.1092

39.

Stewart

Gibson-Smith

MacLure

Mair

Alonso

McIntosh

Codina

Cittadini

Fernandez-Llimos

Fleming

Gennimata

Gillespie

Kempen

Harrison

Junius-Walker

Wiese

Kardas

Lewek

Kinnear

…Scullin

(2017). A modified delphi study to determine the level of consensus across the European Union on the structures, processes and desired outcomes of the management of polypharmacy in older people. PLoS ONE, 12(11), e0188348. https://doi.org/10.1371/journal.pone.0188348

40.

Thompson

Hill

Lecky

Shaw

(2021). Defining major trauma: A Delphi study. Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine, 29(1), 63. https://doi.org/10.1186/s13049-021-00870-w

41.

von der Gracht

H. A.

(2012). Consensus measurement in Delphi studies. Review and implications for future quality assurance. Technological Forecasting and Social Change, 79(8), 1525–1536. https://doi.org/10.1016/j.techfore.2012.04.013

42.

Watanabe

S. M.

Nekolaichuk

Beaumont

Johnson

Myers

Strasser

(2011). A multicenter study comparing two numerical versions of the Edmonton symptom assessment system in palliative care patients. Journal of Pain and Symptom Management, 41(2), 456–468. https://doi.org/10.1016/j.jpainsymman.2010.04.020

43.

Woodcock

Adeleke

Goeschel

Pronovost

Dixon-Woods

(2020). A modified Delphi study to identify the features of high quality measurement plans for healthcare improvement projects. BMC Medical Research Methodology, 20(1), 8. https://doi.org/10.1186/s12874-019-0886-6

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.08 MB