Abstract
In this essay, we explore the tension between research using continuous improvement (CI) paradigms, such as improvement science, and conventional research, and the role and regulation of Institutional Review Board (IRB) oversight. We argue that the current regulatory structure privileges traditional research and hinders collaborative inquiry that centers the voice and agency of those traditionally marginalized. We also question whether IRBs should limit CI efforts required of educational leaders as part of their jobs. We offer recommendations for how IRBs and scholar-practitioners can together support CI efforts.
Schools, historically and contemporarily, have not served all children well. A growing acknowledgment of this fact has created a greater focus on school leaders and their ability to respond to and change that reality, yet these leaders might not always have the tools to do so. Continuous improvement (CI) methodologies are gaining recognition, with the increasing network of institutions as part of the Carnegie Project on the Education Doctorate, as systematic approaches that serve and guide collaborative teams of principals, teachers, community members, students, and families to identify problems, test change ideas, measure their impact, and subsequently adopt, adapt, or abandon the change ideas (Bryk et al., 2015). CI research designs are distinguished as being accountable to “people in schools and communities,” as opposed to traditional research, which is accountable to scholarly peers and funding agencies (Russell & Penuel, 2022, p. 4). Common principles of this kind of research include focusing on solving a problem in a local context, engaging collaborative inquiry that involves diverse actors, using evidence and inquiry as a necessity for improvement, and recognizing the complexity in which true educational transformation takes place (Rutledge et al., 2022, p. 272). CI methodologies include a wide array of approaches, such as (but not limited to) improvement science (IS; Bryk et al., 2015; Hinnant-Crawford, 2020), design-based implementation research (Fishman et al., 2013), community-based design research (Bang et al., 2016), and lesson study (Lewis, 2009). As CI becomes an increasingly common approach to improving the quality of schools, university faculty who engage in these research methodologies with their scholar-practitioners and the policy offices that oversee research are forced to reckon with the evolving conceptions of the meaning of research. In particular, the line between what is generalizable research, requiring the approval of Institutional Review Boards (IRBs), and what is job-embedded quality improvement research has been blurred with the introduction of CI methodologies.
The purpose of this essay is to explore the tension between CI research methodologies and the regulatory oversight of the IRB, particularly as it relates to educational leadership preparation and educational doctorate programs. Specifically, we seek to (a) describe the unique position of educational doctoral programs within this tension, as they are designed to cultivate scholar-practitioners and require students to use these methodologies; (b) illustrate the current landscape of CI research and IRB and the policies governing this landscape; (c) argue that the current regulatory structure privileges traditional research and hinders collaborative inquiry that centers the voice and agency of those traditionally marginalized; and (d) offer recommendations for how IRB offices, scholar-practitioners, and doctor of education (EdD) faculty can together support CI efforts.
To begin, we contextualize the history of the IRB and why it was created, how the common rule defines research versus continuous or quality improvement in terms of the subject of the research, and what it means for oversight in CI. We then offer two vignettes to tell the story of the changing landscape of research in the field of education, which has shifted from a bifurcated field of experts and practitioners to the intentional cultivation of scholar-practitioners, who conduct collaborative, job-embedded research to ensure CI in our schools. The two scholar-practitioners highlighted in the vignettes are enrolled in EdD programs that center CI research methodologies in program pedagogy and in the dissertation-in-practice experience. As part of their doctoral studies, both scholar-practitioners are expected to collaboratively identify, inquire, and work to address a problem of practice within their own professional working environments with the communities and actors who are directly and indirectly affected by the problem.
The Institutional Review Board and Continuous Improvement
The IRB was created in 1974 as a response to unethical research conducted on marginalized and minoritized populations (Lahman, 2017) and has since been used as a safeguard to protect these vulnerable populations. The human-subject abuses in the government-funded Tuskegee Syphilis Study, which saw 300 Black men left untreated for syphilis even after treatment was found to be effective, are examples of why we need to protect vulnerable populations. The Belmont Report published by the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research was part of efforts to significantly revise federal regulations regarding the conduct of research. The report provided three guiding principles for ethical human-subject research—beneficence, respect, and justice—to avoid and to minimize risks, especially for vulnerable groups, while maximizing benefits to society and humanity. We agree that we still live in a society where marginalized populations continue to need protection. However, the framework in the Belmont Report and the Federal Policy for the Protection of Human Subjects (aka the Common Rule) primarily assumes that research would happen only “on” subjects and not “with” subjects. This perception occurs because many forms of research methodologies, including CI, blur the lines between subject and researcher. In CI studies, “subjects” and “researchers” co-design, co-collect, and co-analyze data and then co-implement interventions in their settings, such that the former are involved in setting the research agenda. In the following section, we discuss how research and human subjects are defined per the Common Rule to explain why CI methodologies do not constitute research in the traditional sense.
What Is Research?
Although IRBs across the United States are governed by the guidance of the U.S. Department of Human Health and Services Office of Human Research Protection (OHRP), the guidance can be interpreted differently by each institution. An example is Dyrbye et al.’s (2007) study, which revealed how the same study submitted to different IRBs received different and inconsistent responses, with some accepting the study without any changes and others requiring wholesale changes. Social scientists and action research scholars cautioned practitioners engaging in such inquiry to explore their local IRBs to get a sense of how comfortable and familiar they are with the planned method (Herr & Anderson, 2014).
According to the OHRP, the revised Common Rule, which guides human-subject research, defines research as “systematic investigation . . . designed to develop or contribute to generalizable knowledge” (2018, Sub-part A, Sec. 46.102). According to the OHRP, if the project is intended to test a hypothesis, answer a question, and contribute to generalizable knowledge beyond the research setting, IRB approval is required. Penuel and associates (2020, p. 650) explained, “[W]hereas traditional [research] models seek to identify what things work, ceteris paribus,” CI research is context-specific, and “these models never see ‘all things as equal’ and, instead, focus on what works where, when, and for whom.” Instead of being generalizable, such studies may be better characterized as informative—that is, CI studies may contribute to the “degree of belief” (Langley et al., 2009) of another context’s theory of improvement, but they typically do not make claims of generalizability. Unlike research practice partnerships with an expressed commitment to contribute to generalized knowledge (Henrick et al., 2017), CI within educational leadership programs is designed to support practitioners who are collaborating with their communities to improve or change programs, policies, and processes in their organizational context.
Who Is a Human Subject?
The Common Rule defines a human subject as an “individual
Who Has Oversight in Continuous Improvement?
If the goal of CI is the development of more equitable opportunities to learn in a particular context—not generalizable theory—and it is deemed “not research,” then where is the ethical oversight if investigators want to learn from subjects in their context about how a program, policy, or process functions? We ask, What if protection were to go beyond consent to do research on individuals and instead were reframed to focus on collaborative approaches where marginalized persons not only participate in but also define the research agenda, the inquiry questions, the data collection methods, findings dissemination, and overall impact? We argue that in collaborative inquiry, the oversight of protecting participants shifts away from being an external process (IRB) to being a part of the process of inquiry, where diverse research participants are equally involved in the research process and have a direct say over the data that are shared or included, ensuring equity in the research process. CI research has the very goal of being collaborative in these ways, where the researcher is often a key actor and works closely with other actors and communities to define and address the issue. Although action research (particularly critical or participatory models) often has the goal of improvement, the required collaborative nature of CI is distinct in that it should involve those who are affected by the problem (Hinnant-Crawford et al., 2021). University IRB officers and faculty often view such collaboration with the lens of “doing research on” and impose traditional research definitions for human-subject protection. As a result, IRB often stifles collaborative research, which we believe “reinforce[s] hierarchical power structures and modes of knowledge production that run counter to the goals” of collaborative scholarship (Bradley, 2007, p. 340) and maintains marginalized persons on the outside of the research process. To better understand collaborative research in CI, we look to Penuel and associates (2020, p. 627), who argued that there is a “central role to ‘non-researchers’ in evaluating the quality” of the research. CI must prioritize equity in the research process as well as seek equitable outcomes by involving communities affected by the problem (Bowman et al., 2021; Hinnant-Crawford, 2020; Peterson & Carlile, 2021; Valdez et al., 2020). Equitable research processes consider who is involved in and who is affected by the research process, such that oversight is provided not just by the IRB but by the people involved (Hinnant-Crawford, 2020). When research is truly democratized with authentic, reciprocal partnerships and shared power, the research team provides the necessary oversight over the research. However,
by controlling the models of research, who gets to speak and how subjects get to represent themselves, IRBs . . . can, and often do, silence the voices of the marginalized and perpetuate an academic political economy and a traditional top-down research and professional model that quantify and objectify human lives by keeping them nameless, faceless, and voiceless. (Bradley, 2007, p. 341)
Instead of controlling who is involved in the research process, who is listed on the protocol, who has completed Collaborative Institutional Training Initiative training, and who has access to the data, we believe that ethical oversight of CI inquiries should ask, How are power and decision-making being shared among the wide array of actors and communities involved? Furthermore, in recognition of the fact that CI is designed for organizational learning and that failure is an integral part of the process, safeguards to job-embedded improvement research might include assurances that no data collected can be used for the evaluation of individual performances.
We present the following vignettes based on real scenarios and interactions with IRBs and practitioners enrolled in EdD programs—scholar-practitioners—conducting CI in their context. The vignettes highlight the current challenges CI is posing to some university IRBs.
Vignette #1: Superintendent Salka
Aaliyah Salka (she/her), a superintendent who is enrolled in Southwest Virginia University’s EdD program, is concerned that principals in her district are not providing feedback to teachers about their practices to effect positive changes. In her informal conversations with her principals, they have shared their need for intentional and direct modeling of how to critically provide feedback to their teachers. Salka’s dissertation-in-practice is framed around learning from her principals the challenges they face in providing feedback and designing professional development opportunities that would help them build the needed capacities. She proposes measuring how a professional development program that she designed would affect their feedback practices.
Although Salka supervises the principals, she plans to interview each of them to inquire into the details of the problem, and then she will implement the professional development program. Upon review, the IRB chair, however, indicates that her method is unacceptable due to the possibility of coercion stemming from her supervisory role. The board suggests that someone else conduct the research interviews, transcribe the recordings, and de-identify the information prior to sharing the data with Salka. When the dissertation chair explains that this approach is inconsistent with the EdD program design and its use of CI as a method in collaborative inquiry and implementation, and that it is inauthentic to how Salka engages with principals in their improvement work together, the IRB chair suggests conducting the study in a school district other than her own. After much back-and-forth, the IRB chair finally writes a letter to Salka and the dissertation committee, stating that the project does not constitute human-subject research and, as such, that the IRB does not have oversight. The letter then outlines ethical concerns about coercion, suggests possible alternate oversight processes, and recommends that the EdD program change its dissertation-in-practice design, framed in CI, to a more traditional dissertation.
Vignette #2: Principal Jones
Gabriel Jones (he/him) is the principal of Highest Potential Elementary School in Mountainville, Tennessee, and a doctoral student at High Peaks Regional Comprehensive University. Jones’s dissertation-in-practice focuses on the problem of poor instructional quality as evidenced by the underachievement of minoritized students. Jones has designed, in conjunction with key actors, an instructional intervention that teaches his staff how to employ an historically and culturally responsive literacy pedagogical framework (Muhammad, 2020). His EdD program at High Peaks Regional Comprehensive University has taught him how to employ IS research to identify potential change ideas and assess the effectiveness of these change ideas on improving the achievement of his minoritized students through improved instruction. Jones plans to use the Culturally Responsive Instruction Observation Protocol to observe culturally responsive instructional behaviors in the classroom before, during, and after the multipart professional development intervention he has designed.
As part of his project, Jones has completed multiple revisions of his dissertation study for the IRB. Similar to Salka’s situation, the IRB chair expressed concern about Jones’s position of authority and the possibility of coercion in asking teachers to participate. The IRB chair also questioned the appropriateness of conducting research in his own school, objecting to the idea of a design team, data sharing, and data collected at one point in time informing additional collection cycles. Subsequent reviews rejected protocols where Jones proposed observing teachers, even though his job requires him to observe and evaluate them regularly. To address IRB concerns, several changes were made, including having instructional coaches conduct all observations and paying an assistant to analyze each teacher’s data. The new plan diminishes Jones’s involvement and is in opposition to his role as an instructional leader. Jones is convinced that the IRB does not understand what he is trying to do or what he is required to do in his job. Should the IRB require Jones to abdicate his responsibility to supervise colleagues as they engage in CI or to assume responsibility for improvement in another school or district?
The Duality of Scholar and Supervisor in Continuous Improvement
Scholar-practitioners live in two worlds. Superintendent Salka and Principal Jones are enrolled in doctoral programs that use CI as a signature methodology and that require scholar-practitioners to work with actors and communities within their professional work context to address a problem of practice. Their dissertations-in-practice focus on problems found in their workplace, and interventions that are part of the CI process are designed to ameliorate those problems and to ultimately improve their organizations’ function. Yet to engage in this work, scholar-practitioners have the dual responsibility of following university policy regarding requirements for the dissertation-in-practice, including those of the IRB, and of following contractual expectations for improvement outcomes in their school districts. These responsibilities, however, are sometimes at odds with each other. 1
Today’s school leaders are hired with the expectation that they have the skills to work with actors and communities to improve their school systems in an ever-changing environment. The idea of job-embedded research to improve learning outcomes is a key skill expected of leaders of every school district. CI is an appropriate approach for such leaders because it emphasizes improvement, conducted in collaboration with actors and communities in their own context. CI research specifically asks, What works in a particular context, and why? Data are collected and analyzed with the goal of learning for improvement within that context. The Professional Standards for Educational Leaders (formerly Interstate School Leaders Licensure Consortium), written by the National Policy Board for Educational Administration (2015, p. 18), also expect school leaders to lead job-embedded research. Standard 10, part B, on School Improvement, states that effective school leaders “use methods of continuous improvement to achieve the vision, fulfill the mission, and promote the core values of the school.” Part G of standard 10 further elaborates that effective leaders “develop technically appropriate systems of data collection, management, analysis and use . . . in planning, implementation, monitoring, feedback and evaluation” (p. 18). A national study found that only 60% of principals have the opportunity in their preparation programs to lead inquiry and to collect and analyze data to solve problems (Darling-Hammond et al., 2022). The job-embedded inquiry expected of educational leaders in EdD programs is distinct from the analytical and theoretical inquiry orientation of traditional students in doctor of philosophy (PhD) programs, yet universities and IRB offices expect both groups of students to engage in research in identical ways during their doctoral preparation and to be held to the same IRB regulations. This expectation may be the case because, as Foster et al. (2023) showed, there are many similarities between what is required for a PhD and EdD when it comes to admission requirements and research work.
Universities that acknowledge and support the job-embedded research of their students are supporting their preparation to become scholar-practitioners, enhancing their impact, increasing outcomes for K–12 students, and cultivating the skills educational leaders need. We argue that offering this type of preparation for those who want to remain in practice is the purpose of university schools and colleges of education. To find a way forward in this dilemma, in the subsequent paragraphs, we discuss IRB concerns about power and consent, share an example from the University of Pittsburgh’s (Pitt’s) IRB process for CI-focused dissertations, and offer recommendations for IRB reviewers, scholar-practitioners, and EdD faculty.
IRB Concerns About Power and Consent
As exemplified in our vignettes, the principal investigator of a CI project is likely to involve work colleagues in improvement initiatives within their professional setting. In response to this dynamic, IRB reviewers have typically had concerns about the power present in this environment and implications related to the voluntary nature of project participants. Osterman and associates (2014, p. 98) found in educational leadership programs where scholar-practitioners employed action research, they “received push back from IRB to conduct studies in a work setting where the doctoral student is a leader in that setting.” In addition, Whicher et al. (2015) discovered that, for CI practitioners, IRB guidance was confusing in terms of distinguishing what counted as research from what was normal activity encountered in their everyday practice. Educational leaders submitting IRB applications routinely receive requests for further clarification as to the nature of the principal investigator in relation to the potential for the use of coercion or undue influence to affect the voluntary nature of those who are participating in the study.
Scrutiny by IRB reviewers has historically focused on the nature of voluntary participation in the face of a set of unique power dynamics associated with every insider research project. The concerns most expressed with “voluntariness” are the concepts of coercion and undue influence over study participants. The Belmont Report (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979, p. 7) described instances of coercion as an “overt threat intentionally presented by one person to another person in order to obtain compliance.” Coercion diminishes what Belmont described as the principle of autonomy. Undue influence also occurs, according to the Belmont Report, “through an offer of an excessive, unwarranted, inappropriate or improper reward or other overture in order to obtain compliance” (p. 7). These definitions of coercion and undue influence, involving an overt threat of harm or improper reward for electing nonparticipation, do not describe the typical case we are presented when scholar-practitioners seek IRB approval for CI research. Yet scholar-practitioners may occupy positions of authority where sanctions, hiring, and firing decisions are made, and so IRBs are right to be concerned about power in CI research. Nonetheless, dismissing or rejecting the study would be unethical because it would not help improve or address the problem or inequities in the context for vulnerable and marginalized groups, such as students.
In their professional contexts, scholar-practitioners outside degree programs engage in CI as a normal part of their jobs, with their peers, supervisors, and those they supervise. IRBs designed for traditional research approaches that view research as happening on “subjects” raise ethical concerns about the voluntary participation of these subjects and their ability to opt out. For scholar-practitioners, there may be inequities in programs, policies, and processes in their professional context that could continue to adversely affect their communities if they do not collaboratively work with staff, the community, and other key actors to address problems. Largent et al. (2013, p. 506) posed the question “Why are IRB members so likely to have misconceptions about coercion and undue influence?” We wonder the same thing. Briggs (2022, para. 17) suggested that it is because “many IRBs in the U.S. continue to unreasonably scrutinize low-risk research.” We argue that IRBs in CI research should be more concerned with how stakeholders are involved in investigating the problem, defining the problem, designing the intervention, and implementing improvement efforts than with voluntary participation. When an IRB wonders about coercion and undue influence in a CI study, it should ask more about how the scholar-practitioner who may occupy a position of authority is involving staff, community, and key actors who are directly and indirectly affected by the problem. We suggest that an IRB should be more concerned if a CI study does not include people directly affected by the problem when making decisions about the study.
Nonetheless, although an IRB may require applications to include language on the voluntary nature of participation, employees or colleagues (participants) in the scholar-practitioner’s workplace may be required by their jobs to collectively join improvement efforts in their organization. If the CI study is considered research by the local IRB, consent forms may say that participation in the initiative is required but that use of individuals’ data for the dissertation-in-practice is something individuals can opt out of. For scholar-practitioners to fulfill their job-performance requirements, doctoral programs and IRB offices must find a balance for their students to work with their colleagues and employees to sustain improvement efforts in their organizations and to be able to gather data to determine whether the improvement efforts are working. The required language of voluntariness around CI efforts in the IRB application might suggest that employees can opt out of the duties of their job when, in reality, they cannot and are crucial to inquiring into problems of practice and implementing change ideas.
Finding a Balance: A Way Forward
Many of the challenges with IRBs that we have heard are not uncommon across education doctoral programs. By design, these programs require new admits to be in positions of leadership and to conduct CI research in their organizations. Faculty members who serve on IRBs are not necessarily aware of the nuances of the Common Rule, the day-to-day work of educational leaders in terms of CI, the dissertation-in-practice (or other required capstones), and the job expectations of educational leaders. As Briggs (2022, para. 19) explained, “IRBs will nearly always lack the expertise to genuinely promote ethical research across diverse disciplines.” Pitt offers one way that might help other university IRBs in understanding IS research. The Pitt Human Research Protection Office (HRPO) oversees the IRB for all of the university, including the medical school. The Pitt IRB adheres to the 2018 Common Rule, which stresses that human-subject research is about the person, not about their experience of an external event or the evaluation of a program, policy, or process. In other words, human-subject research defines the scenario where the researcher is obtaining information or biospecimens directly from the individual or manipulating their environment for research purposes. Therefore, asking participants about their perceptions of whether a policy or program in their work environment is working, for example, does not constitute human-subject research. This Pitt IRB office has a full-time staff who reviews applications and a chair who is in charge of monitoring regulations and compliance. Below is a list of steps the Pitt IRB undergoes when reviewing CI dissertations-in-practice for the Pitt EdD program.
Step 1: After their proposal overview with their faculty/practitioner committees, all EdD students provide a brief description of their improvement intervention to the IRB via the question portal.
Step 2: The IRB staff reviews the description and determines whether the project includes research, as defined by the IRB (noted above), or whether the project is a CI effort. If it is determined to be CI, then no IRB is needed, and students receive an email from an agent that will serve as proof of IRB review for journals and future publications.
It is important to note that because CI is embedded into the Pitt EdD program, students work toward their dissertation-in-practice starting in the first semester of the program and continue to build their proposals through four inquiry courses. Early in their coursework, students conduct empathy interviews with key actors and communities in the organization and review organizational data to “understand the problem” and “see the system.” The HRPO was consulted to determine whether these job-embedded activities required IRB approval; the response was that any data gathered before the actual dissertation work (i.e., before the intervention was implemented) were considered preparatory and did not need IRB approval.
Overall, the majority of EdD students at Pitt engaging in a CI dissertation-in-practice do not need IRB approval according to the federal regulations or the Pitt HRPO. Of course, there are exceptions, especially if students are working with vulnerable populations, such as youth, prisoners, or pregnant people. For EdD students working in K–12 settings, local IRB review is often sought as well, although this varies by institution. The example of the Pitt EdD program’s interaction with the IRB offers one model of how CI research conducted by scholar-practitioners might be managed.
Building on Pitt’s example, we want to help expand understanding of CI research and the IRB. To support the work of practitioners employing CI in EdD programs, we offer recommendations in the following section for how to move forward.
Recommendations
Our recommendations (see Table 1) address (a) the requisite knowledge for writing and reviewing IRB applications that have CI as the methodology, (b) the necessity for clarity, (c) the nature of collaboration for equity in CI research designs, and (d) new ethical considerations not currently accounted for in federal research guidance.
Recommendations for Scholar-Practitioners, EdD Faculty, and IRB Reviewers
Note. CI = continuous improvement; EdD = doctor of education; IRB = Institutional Review Board; IS = improvement science; OHRP = Office of Human Research Protection.
Conclusion
In response to horrific abuses of human rights during scientific experiments, the Belmont Report (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979) sought to ensure that human subjects were treated as autonomous (respect for persons), that researchers were intentional about ensuring human subjects’ well-being (beneficence), and that human subjects were treated fairly—ensuring that benefits entitled and minimized burden (justice). Now, we ask, What happens when individuals involved in research are no longer subjects or participants but collaborators? What happens when justice is not simply about how people are treated but about how they are included? What happens when part of the responsibility of the research is embedded in someone’s job? As the landscape of research has changed, the ethical standards must reflect new ethical dilemmas and rectify the incongruences between old standards and new paradigms.
As demonstrated in the opening vignettes, an IRB is positioned to either promote justice and serve as a support for increasing the dignity and well-being of human subjects or to erect and defend barriers to implementing collaborative CI designs, especially in practitioner-oriented doctoral programs. Although the IRB is often seen as an insignificant hurdle that doctoral students must jump, some scholar-practitioners are feeling its full power as it privileges traditional forms of research while potentially hindering others. CI models do not seek to erode respect for persons, beneficence, or justice; to the contrary, this work elevates justice in new and exciting ways (Lewis, 2015). New designs that privilege minoritized voices and allow those in the margins to lead have the potential to produce transformative results in education. With greater knowledge, intentional clarity, recognized collaboration, and new ethical considerations, we believe that the resistance to CI currently faced can be mitigated.
