Reflection/Commentary on a Past Article: “Using the Delphi Method for Qualitative Research in Health Leadership”

In 2014, we published our application of a modified Delphi method for qualitative, participatory action research (PAR) on health leadership. The lead author (Fletcher) was, at the time, a postdoctoral research fellow working in the area of research methodology. This article was one of her first peer-reviewed journal publications. The second author (Marchildon) was Canada Research Chair in public policy and economic history specializing in health policy research.

The article reported on a pan-Canadian research project about leadership during health system redesign, which sought to identify best practices in leadership during major health system change. Our project was one of the five regional studies across Canada, each examining a case of health system restructuring. The goal of our case study was to examine leadership practices during the onset of Shared Services—a restructuring initiative to consolidate service delivery and “back-office” functions across Saskatchewan’s 12 health regions. At the time, Shared Services was promoted as an alternative to full consolidation of the health regions. However, some of our participants wondered if Shared Services was a first step toward centralization, a suspicion confirmed years later when the province’s health regions were amalgamated into a single provincial health authority.

The project methodology needed to address a policy problem (the requisite leadership capacity needed to achieve health reform/restructuring, although this could be applied to many policy management arenas outside health care), a research problem (the difficulty of defining and measuring health leadership and the contested concept of leadership), and the research design (PAR), which harnesses the knowledge and expertise of participants by integrating them as project collaborators. We faced several challenges in our methodological design. The first pertained to power and confidentiality in a study of distributed leadership. In order to best understand the experience of leading change at different levels of authority, our sample needed to consist of health system leaders at three levels: “front line” units, senior leadership below CEO level, and senior executive leadership in the ministry and health regions. To evaluate effective leadership in practice, participants needed to feel safe commenting on the leadership of those above them, which necessitated careful confidentiality provisions. The second challenge was to find an effective mechanism for sharing and validating results with both our participants and our participant collaborators (i.e., our PAR partners) while preserving confidentiality. The leaders’ busy schedules added an additional layer of difficulty; for many, the change initiative—not to mention the research project examining it—was largely being completed “off the side of their desks,” a commonly heard refrain in our findings.

Our underlying challenge, therefore, was to find a method that ensured participant confidentiality while still facilitating dialogue and constant involvement of our participant collaborators. Few of the standard qualitative methods would serve this purpose alone: while in-depth interviews would provide the necessary confidentiality, interview transcripts could not be shared with the participant collaborators and would not allow for the kind of dialogue provided through group methods like focus groups or nominal group technique. Considering the power differentials and high levels of tension and uncertainty participants were experiencing during the Shared Services transition, the lack of confidentiality inherent in group activities made them a non-option.

Delphi technique, in contrast, provided an opportunity for iterative dialogue on the interview themes while guarding confidentiality. The two rounds allowed us to validate and analyze the themes more deeply with participants by creating an interface between researcher and knowledgeable participant as well as a two-way flow, giving the participants an opportunity to “speak back” and interpret the results. This approach also allowed the interviewees to come to a more thoughtful and informed judgment (i.e., a type of public judgment) by seeing the responses of their peers, their bosses, and their employees. The goal of establishing a judgment on the issue most resembles the original aims of a Delphi (Adler & Ziglio, 1995); however, being mindful of critiques that consensus-seeking may simply reflect normative influence or conformity (Goodman, 1987), we chose a primarily qualitative approach and did not seek statistical consensus. Although our interviews did involve some quantitative, Likert-type questions, the overarching approach was qualitative and required a dialogic as opposed to consensus-building approach. Fortunately, the Delphi literature showed a strong tradition of modification, which inspired our qualitative adaptation.

Our confidentiality and consent process also required some modification because of the power dynamics within the study. It was possible that some participants could be recognized through anecdotes or other details provided in the interviews. Therefore, in addition to the usual pre-interview consent form required by the Research Ethics Board, we asked participants to sign a post-interview confidentiality form adapted from Kaiser (2009). The form allowed participants to remove or modify certain details they felt might inadvertently identify them. We believe this added a layer of comfort and protection for participants by providing them more agency over their information.

Delphi also provided the rapid feedback desired by our health leader partners, who hoped the research would assist them in leading a major change. Although this made Delphi a good fit with PAR, the process required a faster turnaround time for coding and analysis than might normally be desired by researchers. However, the process ensured that participants were kept up-to-date on our findings, which may have helped prevent attrition and kept findings relevant during a real-time transition.

Impact and New Directions

According to the International Journal of Qualitative Methods website, our article has been downloaded 2,605 times in the past year. Based on the 25 citing articles to date, its primary use appears to have been in the medical and health sciences, where it has informed studies in surgical endoscopy, oncology, diabetes, and psychology. However, the article has also been used in such diverse research topics as teamwork rubrics (Parratt et al., 2016), philosophical hermeneutics (Guzys, Dickson-Swift, Kenny, & Threlkeld, 2015), and management of protected natural areas (Fefer, De-Urioste Stone, Daigle, & Silka, 2016).

Observing the particular points cited most often over the past 4 years has helped us identify our article’s most relevant contributions to the field of Delphi research. Many studies seem to have relied on the article for general guidance on applying the approach (Adamina, Buchs, Penna, & Hompes, 2018; Matz, Conley, & Johanneson, 2017; Paquette-Warren, Tyler, Fournie, & Harris, 2017) and for justifying its suitability for exploratory research (Schaller et al., 2017; Tan et al., 2017). Some studies have modified the Delphi method in a similar way as our study did; for example, Eremenco, Pease, Mann, and Berry (2017) employed a modified Delphi with no final voting process for consensus. Castro, Dahlin-Ivanoff, and Mårtensson (2016) also used a qualitative modification of Delphi in their study of cultural awareness among occupational therapy students.

At the time of writing our article, we could find very few articles on the use of Delphi method for PAR. Since then, participatory approaches to Delphi have been applied in higher education (Kezar & Maxey, 2016), learning priorities for Danish health-care providers (Kjaer, Vedsted, & Høpner, 2017), and cerebral palsy (CP) management (Morgan, Williams, Tracy, & McDonald, 2016). In the latter study, Morgan, Williams, Tracy, and McDonald (2016) cited our threefold combination of PAR, qualitative interviews, and Delphi approach, which informed their own development of a CP clinical decision-making tool.

A key theme in the citing literature is the use of Delphi method for constructing tools, rubrics, and techniques. In one particularly interesting study, Adamina, Buchs, Penna, and Hompes (2018) used the method to establish international expert support for a rectal cancer procedure. Their study involved 37 surgeons from 20 countries and produced a strong consensus on guidelines for the procedure’s safe use. While writing our article in 2013, we noticed a significant amount of Delphi research on competency frameworks, nursing leadership and practice, and in management and information fields. Over the past few years, the method seems to be increasingly accepted as a means of gathering expertise on medical treatment protocols and tools. Overall, the specific utility of Delphi for exploratory research continues to be strongly supported. Delphi remains a useful tool for gathering and validating expert opinion—whether consensus-based or not—in areas where knowledge is limited. We are pleased to have made a contribution to this useful methodology.