Leveraging the Delphi Technique to Enrich Knowledge and Engage Educational Policy Problems

Abstract

The Delphi technique is a research method that involves experts and stakeholders in structured deliberation about topics through multiple, iterative rounds of data collection such as surveys or in-person meetings. Although it has been utilized extensively in other fields for researching problems, when there is disagreement among key stakeholders or as issues are just emerging, it has been less common in educational research. In this article, the authors outline the Delphi technique, review the scope of its recent applications in educational research, and offer recommendations to advance efforts to support changes in policies and practices through Delphi approaches to research.

Keywords

educational policy policymakers educational reform qualitative research survey research

Get full access to this article

View all access options for this article.

References

Adler

Ziglio

(1996). Gazing into the Oracle: The Delphi method and its application to social policy and public health. London, England: Jessica Kingsley.

Beamish

Bryer

(1999). Practitioners and parents have their say about best practice: Early intervention in Queensland. International Journal of Disability, Development and Education, 46, 261-278.

Birnbaum

Shushok

(2001). The “crisis” crisis in higher education: Is that a wolf or a pussycat at the academy’s door? In Altbach

P. G.

Gumport

P. J.

Johnstone

D. B.

(Eds.), In defense of American higher education (pp. 59-84). Baltimore, MD: Johns Hopkins University Press.

Brown

(1992). The definition of a profession: The authority of metaphor in the history of intelligence testing, 1890-1930. Princeton, NJ: Princeton University Press.

Brydon-Miller

Kral

Maguire

Noffke

Sabhlok

(2011). Jazz and the banyan tree: Roots and riffs on participatory action research. In Denzin

N. K.

Lincoln

Y. S.

(Eds.), The SAGE handbook of qualitative research (pp. 387-400). Los Angeles, CA: Sage.

Bulkley

Fisler

(2002). A review of the research on charter schools. Philadelphia: Consortium for Policy Research in Education, University of Pennsylvania.

Burkhardt

Schoenfeld

A. H.

(2003). Improving educational research: Toward a more useful, more influential, and better-funded enterprise. Educational Researcher, 32(9), 3-14.

Cancian

Armstead

(1990). Participatory research: An introduction. Unpublished manuscript.

Clark

R. W.

Farmer

E. I.

Welch

S. M.

(2010). An examination of leadership issues facing Pennsylvania career and technical administrators. Career & Technical Education Research, 35, 47-62.

10.

Clayton

M. J.

(1997). Delphi: A technique to harness expert opinion for critical decision-making tasks in education. Educational Psychology, 17, 373-386.

11.

Coady

M. H.

Galea

Blaney

Ompad

D. C.

Glidden

Vlahov

(2007). Increased interest in vaccination following a community-based intervention in NYC [PowerPoint slides]. Retrieved from https://apha.confex.com/apha/135am/recordingredirect.cgi/id/17434

12.

Cohen

Barnes

C. A.

(1999). Research and the purposes of education. In Lagemann

Shulman

L. S.

(Eds.), Issues in Education Research (pp. 17-41). San Francisco, CA: Jossey-Bass.

13.

Cohen

Garet

(1991). Reforming education policy with applied social research. In Anderson

Biddle

(Eds.), Knowledge for policy: Improving education through research (pp. 123-140). London, England: Falmer Press.

14.

Colbeck

C. L.

(2000). Reshaping the forces that perpetuate the research–practice gap: Focus on new faculty. New Directions for Higher Education, 2000(110), 35-47.

15.

Conrad

C. F.

(1989). Meditations on the ideology of inquiry in higher education: Exposition, critique, and conjecture. Review of Higher Education, 12, 199-220.

16.

Cross

K. P.

(2000). The educational role of researchers. New Directions for Higher Education, 2000(110), 63-74.

17.

Cuban

(1990). Reforming again, again, and again. Educational Researcher, 19(1), 3-13.

18.

Dalkey

Helmer

(1963). An experimental application of the Delphi method to the use of experts. Management Science, 9, 458-467.

19.

Dee

(2006). Institutional autonomy and state-level accountability: Loosely coupled governance and the public good. In Tierney

W. G.

(Ed.), Governance and the public good (pp. 133-155). Albany: SUNY Press.

20.

Delbecq

A. L.

Van de Ven

A. H.

Gustafson

D. H.

(1975). Group techniques for program planning: A guide to nominal group and Delphi processes. Glenview, IL: Scott Foresman.

21.

Dutta

Kundu

Chan

(2010). The conduct of socially valid investigation by culturally diverse researchers: A Delphi study. Rehabilitation Education, 24, 113-121.

22.

Ellett

C. D.

Teddlie

(2003). Teacher evaluation, teacher effectiveness, and school effectiveness: Perspectives from the USA. Journal of Personnel Evaluation in Education, 17, 101-128.

23.

Floden

R. E.

(2011). Speaking truth to policy and practice. In Conrad

C. F.

Serlin

R. C.

(Eds.), The SAGE handbook for research in education: Pursuing ideas as the keystone of exemplary inquiry (pp. 29-42). Thousand Oaks, CA: Sage.

24.

Fox

M. F.

(1985). Publication, performance, and reward in science and scholarship. In Smart

J. C.

Tierney

W. G.

(Eds.), Higher education: Handbook of theory and research (pp. 255-282). Bronx, NY: Agathon Press.

25.

Franklin

K. K.

Hart

J. K.

(2007). Idea generation and exploration: Benefits and limitations of the Policy Delphi research method. Innovative Higher Education, 31, 237-246.

26.

Gates

(2013, April 3). A fairer way to evaluate teachers. The Washington Post. Retrieved from http://www.washingtonpost.com/opinions/bill-gates-a-fairer-way-to-evaluate-teachers/2013/04/03/c99fd1bc-98c2-11e2-814b-063623d80a60_story.html

27.

Glass

J. H.

Scott

A. J.

Price

M. F.

(2013). The power of the process: Co-producing a sustainability assessment toolkit for upland estate management in Scotland. Land Use Policy, 30, 254-265.

28.

Goldhaber

D. D.

Brewer

D. J.

(2008). What gets studied and why: Examining the incentives that drive education research. In Hess

F. M.

(Ed.), When research matters: How scholarship influences education policy (pp. 197-217). Cambridge, MA: Harvard Education Press.

29.

Graham

(2010). Hearing the voices of general staff: A Delphi study of the contributions of general staff to student outcomes. Journal of Higher Education Policy and Management, 32, 213-223.

30.

Hargreaves

D. H.

(1999). Revitalising educational research: Lessons from the past and proposals for the future. Cambridge Journal of Education, 29, 239-249.

31.

Hasson

Keeney

McKenna

(2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008-1015.

32.

Helmer

(1977). Problems in futures research: Delphi and causal cross-impact analysis. Futures, February, 1977, 17-31.

33.

Hendrickson

R. M.

Lane

J. E.

Harris

J. T.

Dorman

R. H.

(2013). Academic leadership and governance of higher education: A guide for trustees, leaders, and aspiring leaders of two- and four-year institutions. Sterling, VA: Stylus.

34.

Henig

J. R.

(2008). Spin cycle. How research is used in policy debates: The case of charter schools. New York, NY: Russell Sage Foundation and Century Foundation.

35.

Herrington

C. C.

Gadsden

V. L.

(2013). Inaugural editorial. Educational Researcher, 42(7), 7-8.

36.

Hill

(1996). Education: How research affects policy and how both affect education. In Galster

(Ed.), Reality and research: Social science and U.S. urban policy since 1960 (pp. 131-155). Washington, DC: Urban Institute Press.

37.

Hirsch

(2000). Practitioners as researchers: Bridging theory and practice. New Directions for Higher Education, 2000(110), 99-106.

38.

Holden

M. C.

Wedman

J. F.

(1993). Future issues of computer mediated communications: The results of a Delphi study. Educational Technology Research and Development, 41(4), 5-24.

39.

Hsu

Sandford

B. A.

(2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), 1-8.

40.

Hussler

Muller

Ronde

(2011). Is diversity in Delphi panelist groups useful? Evidence from a French forecasting exercise on the future of nuclear energy. Technological Forecasting & Social Change, 78(9), 1642-1653.

41.

Judd

R. C.

(1972). Use of Delphi methods in higher education. Technological Forecasting and Social Change, 4, 173-186.

42.

Kaestle

(1993). The awful reputation of education research. Educational Researcher, 22(1), 23-31.

43.

Keeney

Hasson

McKenna

H. P.

(2001). A critical review of the Delphi technique as a research methodology for nursing. International Journal of Nursing Studies, 38, 195-200.

44.

Keller

(1985). Trees without fruit: The problem with research about higher education. Change: The Magazine of Higher Learning, 17(1), 7-10.

45.

Kennedy

(1999). Infusing educational decision making with research. In Cizek

G. J.

(Ed.), Handbook of Educational Policy (pp. 54-76). New York: Academic Press.

46.

Kezar

(2000a). Still trees without fruit? Higher education research at the millennium. The Review of Higher Education, 23, 443-468.

47.

Kezar

(2000b). Understanding the research-to-practice gap: A national study of researchers’ and practitioners’ perspectives. New Directions for Higher Education, 2000(110), 9-19.

48.

Kezar

Eckel

(Eds.) (2000). Moving beyond the gap between research and practice in higher education. San Francisco: Jossey-Bass.

49.

Kezar

Maxey

(2014). The Delphi technique: An untapped approach of participatory research. International Journal of Social Science Research Methods, 2014, 1-18.

50.

Koster

Brekelmans

Korthagen

Wubbels

(2005). Quality requirements for teacher educators. Teaching and Teacher Education, 21, 157-176.

51.

Labaree

D. F.

(1998). Educational researchers: Living with a lesser form of knowledge. Educational Researcher, 27(8), 4-12.

52.

Labaree

D. F.

(2003). The peculiar problems of preparing educational researchers. Educational Researcher, 32(4), 13-22.

53.

Labaree

D. F.

(2004). The trouble with ed schools. New Haven, CT: Yale University Press.

54.

Lagemann

E. C.

(1999). An auspicious moment for education research. In Lagemann

Shulman

L. S.

(Eds.), Issues in education research (pp. 17-41). San Francisco, CA: Jossey-Bass.

55.

Landeta

(2006). Current validity of the Delphi method in social sciences. Technological Forecasting & Social Change, 73, 467-482.

56.

Levin

(2004). Making research matter more. Educational Policy Analysis Archives, 12(6), 1-20.

57.

Lincoln

Y. S.

Guba

E. G.

(2000). Paradigmatic controversies, contradictions, and emerging influences. In Denzin

N. K.

Lincoln

Y. S.

(Eds.), Handbook of qualitative research (2nd ed., pp. 163-186). Thousand Oaks, CA: Sage.

58.

Linstone

H. A.

Turoff

(1975). The Delphi method: Techniques and applications. Reading, MA: Addison-Wesley.

59.

F. J.

Hsu

Chan

Cheen

Kao

(2012). Assessing college student-athletes’ life stress: Initial measurement development and validation. Measurement in Physical Education and Exercise Science, 16, 254-267.

60.

Makridakis

S. G.

Wheelright

S. C.

(1978). Interactive Forcasting: Univariate and Multivariate Methods. Holden-Day, Incorporated.

61.

Manley

R. A.

(2011). The Decentralization of Perkins: History, impact, and recommendations for future CTE legislation. Career and Technical Education Research, 36, 119-152.

62.

Marshall

(2014). Exploring the ethical implications of MOOCs. Distance Education, 35, 250-262.

63.

McNiff

(2013). Action research: Principles and practice. New York, NY: Routledge.

64.

Millar

Thorstensen

Tomkins

Mepham

Kaiser

(2006). Developing the ethical Delphi. Journal of Agricultural and Environmental Ethics, 20, 53-63.

65.

Nworie

(2011). Using the Delphi technique in educational research. TechTrends, 55(5), 24-30.

66.

Pill

(1971). The Delphi method: Substance, context, a critique, and an annotated bibliography. Socio-Economic Planning Sciences, 5(1), 57-71.

67.

Plank

D. N.

(2011). Minding the gap between research and policy making. In Conrad

C. F.

Serlin

R. C.

(Eds.), The SAGE handbook for research in education: Pursuing ideas as the keystone of exemplary inquiry (pp. 43-58). Los Angeles, CA: Sage.

68.

Plinske

Packard

W. J.

(2010). Trustees’ perceptions of the desired qualifications for the next generation of community college presidents. Community College Review, 37, 291-312.

69.

Pollard

(2004). Research priorities in educational technology: A Delphi study. Journal of Research on Technology in Education, 37, 145-160.

70.

Powell

(2003). The Delphi technique: myths and realities. Journal of Advanced Nursing, 41(4), 376-382.

71.

Ramaley

J. A.

(2006). Governance in a time of transition. In Tierney

W. G.

(Ed.), Governance and the public good (pp. 157-177). Albany: SUNY Press.

72.

Reason

(1994). Three approaches to participative inquiry. In Denzin

N. K.

Lincoln

Y. S.

(Eds.), Handbook of qualitative research (pp. 324-339). Thousand Oaks, CA: Sage.

73.

Reeves

Jauch

L. R.

(1978). Curriculum development through Delphi. Research in Higher Education, 8, 157-168.

74.

Robinson

V. M.

(1998). Methodology and the research–practice gap. Educational Researcher, 27(1), 17-26.

75.

Rossouw

Hacker

de Vries

M. J.

(2011). Concepts and contexts in engineering and technology education: An international and interdisciplinary Delphi study. International Journal of Technology and Design Education, 21, 409-424.

76.

Rowe

(1994). Enhancing judgement and decision making: A critical and empirical investigation of the Delphi technique (Unpublished PhD thesis). University of Western England, Bristol, UK.

77.

Rowe

Wright

(1999). The Delphi technique as a forecasting tool: Issues and analysis. International Journal of Forecasting, 15(4), 353-375.

78.

Sackman

(1975). Delphi critique: Expert opinion, forecasting, and group process. Boston, MA: Lexington.

79.

Sadhasivam

(2014). Educational paradigm shift: Are we ready to adopt MOOC? International Journal of Emerging Technologies in Learning, 9(4), 50-55.

80.

Salkind

N. J.

(2010). Delphi technique. In Salkind

N. J.

(Ed.), Encyclopedia of research design (pp. 344-347). Thousand Oaks, CA: Sage.

81.

Sangrà

Vlachopoulos

Cabrera Lanzo

(2012). Building an inclusive definition of e-learning: An approach to the conceptual framework. The International Review of Research in Open and Distributed Learning, 13(2), 145-159.

82.

Schmidtke

(2010). Math and science instructor perceptions of their American Indian students at a sub-baccalaureate technical college: A Delphi study. Journal of Career and Technical Education, 25(2), 8-23.

83.

Shelton

(2010). A quality scorecard for the administration of online education programs: A Delphi study (Doctoral dissertation). University of Nebraska, Lincoln, NE.

84.

Shriberg

Bonner

Sarr

B. J.

Walker

A. M.

Hyland

Chester

(2008). Social justice through a school psychology lens: Definition and applications. School Psychology Review, 37, 453-468.

85.

Sroufe

G. E.

(1997). Improving the “awful reputation” of education research. Educational Researcher, 26(7), 26-28.

86.

Strauss

H. J.

Zeigler

L. H.

(1975). The Delphi technique and its uses in social science research. The Journal of Creative Behavior, 9, 253-259.

87.

Tierney

Rhoads

(1994). Enhancing promotions, tenure, and beyond (ASHE-ERIC Higher Education Report Series). San Francisco, CA: Jossey Bass.

88.

Tyack

Cuban

(1995). Tinkering toward utopia: A century of public school reform. Cambridge, MA: Harvard University Press.

89.

Udoh

I. A.

Mantell

J. E.

Sandfort

Eighmy

M. A.

(2009). Potential pathways to HIV/AIDS transmission in the Niger Delta of Nigeria: Poverty, migration, and commercial sex. AIDS Care, 21, 567-574.

90.

Vergari

(2007). The politics of charter schools. Educational Policy, 21, 15-39.

91.

Westerheijden

D. F.

Huisman

de Boer

(2004). Ground force does the Dutch higher education gardens. Three scenarios revisited. Policy Futures in Education, 2, 374-387.

92.

Wright

T. S. A.

(2007). Developing research priorities with a cohort of higher education for sustainability experts. Journal of Sustainability in Higher Education, 8, 34-43.

93.

Zemsky

(2009). Making reform work: The case for transforming American higher education. New Brunswick, NJ: Rutgers University Press.