Advancing the Impact of Critical Qualitative Research on Policy,Practice,and Science

Abstract

Discourses of research impact shape the ways in which critical qualitative research and researchers are evaluated in contemporary academic environments. Mainstream conceptualizations of research impact arise from a positivist perspective that challenges the aims and approaches of critical qualitative research. In this paper, we propose a framework for conceptualizing the impact of critical qualitative research on policy, practice, and science. After critiquing literature that presents mainstream views on research impact, we summarize a recent framework for conceptualizing the impact of critical research specifically. We then add to the Machen framework by highlighting the impacts of critical qualitative research on the institutions and practices of science. We provide examples of ways in which researchers at the Centre for Critical Qualitative Health Research at the University of Toronto have made contributions to the impact of critical qualitative research on science, and conclude by addressing implications of this framework for the ways in which critical qualitative researchers can plan and evidence the impact of their work.

Keywords

research impact knowledge translation critical theory philosophy of science qualitative evaluation social justice

The discourses of research impact and their key performance indicators are defining features of contemporary research governance and have found their way into many aspects of research practice in health and social science. Research impact is used as a metric to indicate the quality of researchers (Webster et al., 2019), to determine a proportion of funding to some university departments (Hill, 2016), and to make decisions about the awarding of new research funds to research teams (MacDonald, 2017). Although the academy has always been allied to certain applied interests (Pestre, 2004), the ways in which research impact has become an accepted logic of the academic research ecosystem appears to be of a unique and increasingly aggressive contemporary neoliberal character (Colley, 2014). This trend poses unique challenges and opportunities for qualitative researchers, and particularly for those who identify with critical, emancipatory, and decolonizing epistemologies (Cann & DeMeulenaere, 2020; Machen, 2020). The sorts of changes sought after by critical qualitative research (CQR) could frequently clash with the neoliberal impulses of the research impact agenda that seeks to foster commercialization and incremental changes in policy or practice without disrupting the status quo (Berg & Seeber, 2018; MacDonald, 2017).

This paper is written from our perspective as critical qualitative researchers and educators in the health and social sciences. Trained across a variety of disciplines, including, sociology, nursing, physiotherapy, rehabilitation science, education, and public health, we came together over a shared commitment to produce theoretically informed qualitative methodologies and develop critical approaches to health as a social phenomenon. While the examples used throughout this paper stem from our collective work in the health sciences context, we believe the paper will be of interest to a wide range of audiences, across professional and disciplinary boundaries, who are interested in the impact of CQR on practice, policy, and science.

Specifically, in this paper we present a framework for thinking about the research impact of CQR, and comment on the ways in which critical qualitative researchers can both evidence the impact of their work and advance their critical agendas. We begin by outlining our theoretical approach to this issue, and then provide a brief history of dominant discourses of research impact in the health and social sciences. We then present the impact framework developed by Machen for critically-oriented research (Machen, 2020), and describe an addition to the framework to illustrate how CQR can exert impact on the institution of science itself. We provide examples from our experience at the Centre for Critical Qualitative Health Research (CCQHR) at the University of Toronto, and comment on the implications of this framework for the ways in which critical qualitative researchers can plan and evidence the impact of their work.

Theoretical Approach

Our approach to this paper is inspired by an explicitly critical lens. We define the concept of “critical” in relation to health-related aims as

the capacity to inquire “against the grain”: to question the conceptual and theoretical bases of knowledge and method, to ask questions that go beyond prevailing assumptions and understandings, and to acknowledge the role of power and social position in health-related phenomena. The notion includes self-critique, a critical posture vis a vis qualitative inquiry itself. (Centre for Critical Qualitative Health Research, 2019)

Any satisfying history and description of critical theory is beyond the scope of this paper, but a few brief comments are warranted to clarify our approach. Critical scholarship takes its inspiration from many sources, but is widely regarded as having its original roots in the writings of Karl Marx as taken up by the group of scholars in Germany known as “the Frankfurt School” in the first half of the 20^th Century (Feenberg, 1981; Horkheimer, 1972a, 1972b). As members of the Frankfurt School fled to North America during the Second World War, this school of thought came to influence philosophy, social science, and social activism in the United States (Antonio, 1981). Although the range of contemporary thinking allied to critical theory is rooted in a variety of ontological and epistemological positions (Sayer, 2009), one common theme that underpins many critical theories is the effort to more accurately understand the ways in which power becomes manifest in society and the effects it produces in so doing (Antonio, 1981; J. Eakin et al., 1996). However, we note that there is a wide diversity of views that fit within the domain of critical theory.

In our shared view, taking a critical approach to understanding research and research impact means acknowledging the role of a broader neoliberal discourse that is acting on the research enterprise, leading to growing emphases on number of publications, prestigious awards, research dollars, and the apparently bottomless commitment of researchers to devote their time to the machinations of their research institutions (Berg & Seeber, 2018; Pinel, 2021). We draw on the definition of neoliberalism as summarized by Thompson (2005), where neoliberalism is understood as a shift in policy at several levels away from historical modes of state action in governance and service provision toward the centrality of markets and capital. Thompson (2005) explains that, beyond such policy shifts, neoliberalism “is the elevation of capitalism, as a mode of production, into an ethic, a set of political imperatives, and a cultural logic.” (pp. 22–23). Neoliberalism creates a particular logic of institutional activity that comes to characterize universities’ varied decision-making processes, including the ways in which researchers’ careers are assessed (Webster et al., 2019). The neoliberal logic influences the ways in which science and research are understood by researchers and the broader public, shaping how “good science” and “good scientists” are conceptualized (Machen, 2020; Pain et al., 2011).

In previous work (Webster et al., 2019), several of us have highlighted the ways in which the neoliberal influence on the research enterprise is complemented by the epistemological and normative position of “scientism” (i.e., the view that positivist assumptions of science are epistemologically and normatively superior to others). We state that, “scientism is shaping our worldview, including the way we view the health sciences, because it frames how we interpret common concepts such as evidence and rationality, refusing to consider human experience, politics, subjectivity, and values, in knowledge production” (p. 2). Although a detailed discussion of the quality of qualitative research is beyond the scope of our paper, we acknowledge the influence of a neo-liberal logic of science on conceptualizations of quality in qualitative research. Research-intensive environments are frequently characterized by a scientistic understanding of the quality and rigor of research, and neglect the foundational knowledge required to better understand the ways in which theory, quality and rigor work together in CQR.

This view of scientism is not just dominant within university environments, but also pervades the ways in which other social institutions make decisions (e.g., government policymaking) (Hoppe, 2005). The narrow focus on a scientific version of knowledge that neglects to consider the ways in which values and power shape the research enterprise limits the possibilities for academic research to contribute to a better world for all. The pervasiveness of scientism, and its intersection with neoliberalism, poses particular challenges for critical qualitative researchers in the context of the research impact agenda, which we explore in this paper.

Methodology

Our effort in this paper is to critique a narrow understanding of research impact and to propose a more sophisticated framework of mechanisms through which CQR might impact policy, practice, and science. To do so, we have employed a process methodology of “thinking with theory”, as described by Jackson and Mazzei (2018). In their work on the practice of thinking with theory as an analytic methodology, Jackson & Mazzei (2013, 2018) draw on posthuman and poststructural worldviews to complicate conventional notions of the ways in which theory is applied in qualitative analysis (Jackson & Mazzei, 2018). They propose that, against conventional notions of analysis as a linear application of method, “thinking with theory does not come at the end of anything but is emergent and immanent to that which is becoming” (p. 1232). In this way, thinking with theory is a process that informs an analytic endeavor from the beginning, shaping the ways in which the world is experienced, data are generated, and sense is made (Guenther & Falk, 2021).

Jackson and Mazzei (2018) outline the value of drawing on concepts from theories of various kinds to pursue “eruptions”, which are analytic moves that push thinking beyond taken-for-granted conceptualizations of the topic of a given study. Such eruptions enable new thinking about the dimensions of a topic and occur throughout an entire research process. In the analytic process we employed in preparing this paper, we focused on a small set of theoretical concepts that we carried through our review of literature, critique of existing frameworks, and ultimately the establishment of the new framework we propose.

Specifically, we drew on two concepts from the definition of “critical” presented earlier to guide our critique and re-construction of research impact. First, we used the notion of “inquiring against the grain” to resist the pull toward linear notions of impact moving only from science to society, and sought instead to understand how the efforts of CQR might act back on science itself. In order to do so, we had to establish a view of science not only as a set of methods, but rather as a collection of people and institutions investing resources to achieve particular ends.

Second, we used the notion of positional power to build our framework, understood as the power of individuals and institutions that is derived from their location in intersecting social systems. The positional power of critical qualitative researchers vis a vis other researchers, students, academic administrators, research funders, policymakers, and other stakeholders is consequential for their capacity to achieve research impact. In this way, the concept of positional power was inspirational for our framework in important ways.

We drew on these concepts to articulate mechanisms by which critical qualitative researchers can facilitate the impact of CQR. In this sense, mechanisms refer to strategies by which a CQR-related output can be made to influence some social process or structure, thereby achieving impact. Importantly, science is also cast as including both social process (i.e., practices) and social structures (i.e., institutions) in our analysis. Where the work of a researcher drawing on CQR facilitates impact that more directly aligns with local, individual, or small group shifts in practices related to conducting or applying research, we grouped mechanisms as being most relevant at the “practice level”. Conversely, where such work produces shifts among large groups or institutional processes that influence large groups, we grouped mechanisms as being most relevant at the “policy level” or “institutional level”. Although we acknowledge this is an imperfect classificatory scheme, we believe it provides ample structure for the framework we generate in our paper. Specific definitions of each these levels are elaborated later in our paper.

Carrying forward the notion of “thinking with theory” as described by Jackson and Mazzei (2018), we reviewed literature to critique conventional understandings of research impact across select health and social science disciplines. We then focused on expanding a promising framework by Machen (2020), adding an explicit focus of the impact of CQR on science. These additional mechanisms of impact for CQR were generated through dialogue between the author team, informed by the concepts of “inquiring against the grain” and “positional power”, and in relation to examples of actual critical qualitative research projects taking place at the Centre for Critical Qualitative Health Research at University of Toronto.

Brief History of Research Impact

The notion of research impact has a number of relevant histories (Hill, 2016), but we situate our discussion in the emergence of evidence-based medicine and its extension to evidence-based policy and practice more generally (Greenhalgh et al., 2016; Mykhalovskiy & Weir, 2004; Webster, 2020). As evidence-based medicine grew in popularity, the practices of knowledge translation, research dissemination, and implementation science arose ostensibly in service of achieving a vision of more evidence-based health and social services (Straus et al., 2013). The ideas and methods of these fields have continued to spread into a wide range of domains, and created a foundation for understanding the particular ways in which research across disciplines might have influence in applied domains of relevance to the public (Fowler, 1997; Oakley, 2002). The last few decades have seen the formalization of these concepts and practices into a systematic emphasis on the effort to ensure research can achieve impact to represent its value to society (Bastow et al., 2014; Hill, 2016).

Debates about research impact in the social sciences and humanities rose to prominence in the academic literature surrounding the development and implementation of the UK’s Research Assessment Exercise (RAE) and subsequently the Research Excellence Framework (REF). The REF assessment, which was intended to measure the “real world” impact of universities’ research output, was first carried out in 2014 (Greenhalgh & Fahy, 2015; Machen, 2020). Its results bring prestige and funding to universities that score highly, but critics argue that the “impact agenda” and “audit culture” associated with the REF and similar frameworks’ narrow concept of impact are harmful to academic freedom (MacDonald, 2017). These critics assert that true impact is not measurable by such a-theoretical instruments (Colley, 2014). Supporters, on the other hand, argue that the REF’s definition of impact is uniquely broad among other audits, as it measures economic, societal and cultural impact as well (Cruz Rivera et al., 2017; Pain et al., 2011). Even some critical scholars have shown support for the REF, reflecting on its capacity to be repositioned in service of social justice and the meaningful engagement of communities throughout the research process (Pain et al., 2011).

The impact agenda has different implications for scholars in different disciplines. In the humanities, researchers report that the impact of their work is non-linear, and that the ways in which their research has the most impact—through public engagement, educational value, and influencing policy and advocacy—are incremental and not easily measured (Machen, 2020; Oancea, 2013). In the social sciences, impact measurements depend on whether the research is “theoretical” or “applied”, which we as authors are quick to identify as a spurious and unhelpful dichotomy (Boswell & Smith, 2017; Oancea, 2013). For example, Oancea (2013) suggested that the impact of theoretical research might be understood as a “slow diffusion of conceptual and critical insight” (p. 245), whereas for more applied work, impact comes from producing strong evidence that can be used as the basis for policies and practice (Oancea, 2013). The point made by Oancea (2013) is that researchers in the social sciences fear that frameworks like the REF force a narrowing of the concept of impact. The consequences of this narrow concept of impact include the closing of departments focused on social science and humanities of health in the United Kingdom as a result of their misalignment with the expectations of the REF exercise.

In the health sciences, the impact discussion has focused largely on the potential of evidence produced by research to improve healthcare policy, systems, delivery, efficiency, and outcomes (Alla et al., 2017). Such perspectives tend to overlook the point that improvements are mostly incremental and not the result of a single study or series of studies no matter what approach is used. Few changes are the result of “breakthroughs”. Impact is seen as an essential outcome of research so as to maximize benefits to patients through the production of evidence, and reduce “research waste” (Cruz Rivera et al., 2017). A review of methods of measuring impact in the health sciences found that most measurements were based on quantitative assessments such as citation rates in high-impact journals. The authors of the review questioned the value of this method, as it measures the passive dissemination of the findings rather than their impact (Cruz Rivera et al., 2017). Based on their literature review, Alla and colleagues (2017) produced a definition of research impact with a particular focus on mental health; we present a slightly modified definition here as a representation of the conventional view of research impact:

Traditionally, research impact is seen as the direct or indirect contribution of research processes or outputs that have informed (or resulted over time in) the development of new health policy/practices, or revisions of existing health policy/practices, at various levels of governance (international, national, state, local, organizational, health unit).

Based on our brief review of the highly cited literature on research impact reported in this section, it is clear to us that health sciences literature has been much less critical of the impact agenda than that in the broader social sciences and humanities.

These various treatments of the research impact agenda represent different disciplinary approaches to understanding science and the ways in which it influences society at large. Although impact assessment exercises like the REF have been assessed in both positive and negative light, we suggest that there has not been enough discussion regarding the place of critically-oriented qualitative work in the research impact conversation. We now move on to present a recent framework for conceptualizing research impact from a critical perspective, and propose modifications that give due weight to the potential impacts of CQR in particular on the institution of science itself. Inspired by critical reflexivity, we seek to broaden the consideration of “research impact” to include not only institutions conventionally seen as “outside” of science (e.g., institutions of policy and practice), but to include the very institutions through which CQR is produced and governed.

Critical Social Science Contributions for Re-thinking Impact

In order to advance the dialogue on the nature of the fit between critical research and the impact agenda, Machen (2020) reviewed literature on research impact and analyzed impact case studies arising from the REF assessment in the UK in 2014. As context for her work, Machen (2020) acknowledged a number of clear challenges with systematic efforts to assess research impact specifically from a critical perspective. First, impact assessment rewards “safe” research that is acceptable to people who make policy decisions (i.e., research that does not fundamentally question the status quo). Second, impact assessment favors so-called star scientists who already enjoy positive reputations and are sought after by those in positions of decision-making power. Finally, impact assessments operate on relatively short cycles (6 years in the case of the UK REF), which puts the more fundamental changes sought after by critical research at a clear disadvantage. However, despite these important points of context, Machen (2020) builds on Pain et al. (2011) suggestion that critical scholars must “reclaim and rethink research impact” (p. 4).

To that end, Machen assessed 16 reports in one subsection of the REF impact case studies database (focused on the discipline of geography) that explicitly stated their alignment with a critical orientation. Building an inductive analysis of the strategies of impact identified in the database, Machen (2020) constructed a conceptual model of ways of thinking about research impact arising from critical work. These were presented through five “modes” of research impact, which are represented visually in Figure 1. Crucially, these modes of impact are not natural occurrences. Rather, they are the product of the coordinated actions of research teams and community members to achieve particular kinds of changes in the world. One point emphasized in Machen’s work that we carry forward is that research impact does not happen automatically or without investment from research teams. Impact is a product of a collection of practices oriented toward achieving a variety of aims. These aims are detailed in Machen’s framework, and we outline them next.

Figure 1.

Framework for impact of critical qualitative research.

Machen (2020) describes Mode 1 in the impact model as “challenging mainstream policy”, involving in-depth analysis of existing policy in terms of its underlying assumptions. This mode is focused on creating the foundations for policy change, and also included more explicit efforts to create changes within existing policy frameworks. Such an approach is a clear move away from “safer” approaches to impact assessment outlined in our brief review of the literature.

Mode 2 is termed “empowering resistance”, and included the development of research findings and related narratives that enable groups and organizations to successfully advocate for an agenda allied to critical research. The prime example given in this case was providing Non-Governmental Organizations (NGOs) with research findings that could be used to advocate for particular policy changes or for a generally stronger position in advancing a particular agenda.

Mode 3 is referred to as “platforming voices”, and was described by Machen (2020) as “listening to, supporting, representing, or mobilizing marginalized voices” (p. 9). Creating the link to participatory action research (PAR), a methodology long used by critical researchers, this mode of research impact was focused on building the skills and knowledge necessary to encourage conventionally under-represented groups to advocate for their own agendas.

Mode 4 is titled “nurturing new critical publics”, and was focused on the foundations that enable particular groups to engage in political processes (whether governmental or not). Such foundations include ensuring particular kinds of relevant knowledge, boosting confidence, and building the networks that enable groups to find a voice and mobilize a message in service of social change.

Mode 5, the final mode, is referred to as “envisioning alternatives”, and included any kind of work that could enable groups to develop imaginaries that were alternative to the status quo. Simply generating understandings that might clarify where social orders need to change is the focus of impact in this category.

The five modes of impact outlined by Machen are extremely useful for critical qualitative researchers seeking to illustrate the different ways in which their work has led or can lead to changes in the world beyond conventional scientistic notions of research impact over time. Machen’s original model refers to the outputs of a research project, in the sense that its focus remains on the connections between research on the one hand, and policy or practice on the other. However, we propose that critical research exerts an additional kind of influence on the world through its impact on science itself, influencing how institutions and knowledge producers think about science. The modes of impact outlined by Machen are oriented to critical research in general, regardless of methodologies employed. We orient our discussion here specifically to CQR.

The Impact of Critical Qualitative Research on Science

In the same way that concerted effort and coordinated action can facilitate the impact of critically-informed research on policy and practice, they can also facilitate the impact of CQR on the scientific enterprise more generally. We are using the term “science” here to refer both to (a) the collection of institutions that support and govern the generation of knowledge, and (b) the collection of practices carried out by a variety of people, including researchers, educators, students, practitioners, policymakers, and others, that constitute the work of generating knowledge (Bloor, 1991; Chalmers, 2013; Latour, 1987). We refer to these two aspects of science, respectively, as the institutional level and the practice level of science, and propose that CQR can exert important impacts on each. Although we acknowledge the shift away from perspectives that privilege the human actor or a simple binary between structure and agency in critical studies of science (Haraway, 2006; Jasanoff, 2004; Latour, 1987; Mol, 2003), we limit our discussion to these two levels to enhance interpretability and practicality of our discussion for a broadly interdisciplinary audience (including university administrators).

We use the institutional level of science to refer to the academic, political, economic and organizational structures that govern scientific output (Au & da Silva, 2021; Herbst, 2013). These include historical and current institutional realities such as systems for the allocation of research funding and resources, access to publication sites and opportunities, networks and links with the innovation economy and evidence-based policy, scientific legitimation and reward mechanisms, and academic career processes such as hiring, tenure, and promotion (Johnson, 2020; Webster et al., 2019). These institutional systems are mechanisms employed by governments and organizations to govern the research enterprise, thereby shaping which actors play which roles in contemporary academic knowledge production (Fuller, 1999; Shapin, 1995, 2009).

We use the practice level of science to refer to the actual doing of scientific research, the everyday practices in which people engage as they participate in processes of producing knowledge (Bloor, 1991; Latour, 1987). These practices include the methods used, the theoretical foundations and disciplinary knowledge banks drawn upon, the ethical and professional social relations of research work, the relationship between researchers and the communities with which they are engaged, and the mobilization of research knowledge within and beyond the academy (Latour, 1987; Papoutsi et al., 2021; Pinel, 2021). These constitute the typical “work” of scientists and others as they contribute to knowledge production.

Through different mechanisms, we suggest that engagement with CQR can have impact on both the institutional and practice levels of science. Impact on these levels of science is not something that comes naturally or without great effort from critical qualitative researchers themselves, but represents work to produce research environments in which CQR researchers can survive and hopefully thrive. The overarching framework for impact, including both Machen’s (2020) work and our contributions here, is visualized in Figure 1. In this framework, we include mechanisms of impact on policy and practice, as well as impact on science at the institutional level and practice level. In the next section, we introduce an overview of the types of impact that can be achieved through the work of CQR researchers at the institutional and practice levels.

The Impact of Critical Qualitative Research on Science

At the CCQHR, over a 20-year period, we have moved from being critical qualitative researchers studying health issues in isolation to establishing a Centre for visibility and legitimacy within a research-intensive university and well-established faculty (Eakin & Gastaldo, 2020). Over this time, we have engaged in several initiatives oriented toward firmly establishing CQR education and research in the university environment, and building a network of scholars who can work together to advance their work and the CCQHR mission more generally. Here we describe ways in which the work of CCQHR leaders and members has impacted the institutional and practice levels of science, through its influence both within and outside of the university environment. We contend that each of these approaches is a representation of the impact of CQR and CQR researchers on science.

Impact on Science at the Institutional Level

The first approach to impact at the institutional level is by challenging the hegemony of scientism in the policies, procedures, and routines of the university environment. Members of CCQHR have engaged in this important work in several ways, ranging from direct interventions in formal procedures to less formal, interpersonal work on graduate student advisory committees. One notable example is a publication by Webster al (2019), which surveyed the academic promotion materials of health sciences faculties in Canada and identified practices that implicitly or explicitly exclude qualitative researchers from career advancement (Webster et al., 2019). That work culminated in the proposal of an alternative framework for the evaluation of qualitative researchers related to hiring, tenure, and promotion decisions, which has been widely disseminated as a tool that CQR researchers can build upon in their promotion processes. By intervening upon the operating procedures of various aspects of conventional university practices, this strategy holds potential to impact the institutions that govern science at the local organization and beyond.

The second avenue for impact at the institutional level of science is through investments in the education of future generations of critical qualitative researchers. Educating the next generation of critical qualitative researchers represents an important element of practice for experienced critical qualitative researchers, but we propose that the mechanism of change it represents belongs at the institutional level. This is because the implementation of a sophisticated CQR curriculum over a number of years has the effect of nurturing a generation of researchers to proceed and become independent drivers of institutional change dispersed throughout the academic ecosystem. Members of CCQHR built a qualitative research curriculum that was made available to students across health-related faculties that encouraged progression toward advanced expertise in qualitative methodologies (Centre for Critical Qualitative Health Research(CCQHR), 2020; J. M. Eakin, 2016). The curriculum was not focused on providing a basic introductory education to qualitative research, but rather on educating graduate students to establish expertise in the methodological and theoretical domains that would enable them to push the boundaries of CQR methodology into the future. This approach to the CQR curriculum is a strategy to grow the cadre of highly talented CQR researchers, thereby producing future opportunities to influence the institutions of science with the cutting edge of critical qualitative approaches (CCQHR, 2020). Importantly, it is only possible to teach advanced and evolving approaches to CQR when educators are themselves deeply engaged in the practice of CQR (J. M. Eakin, 2016), illustrating the crucial link between CQR research and teaching.

The third avenue for impact at the institutional level of science relates to marshalling resources for critical qualitative research. Members of CCQHR have sought out opportunities to influence policy and funding decisions made by important research funders in the country. This has included requesting meetings to explain the nature of interdisciplinary research that crosses the health and social sciences when the social science research funder (The Social Sciences and Humanities Research Council) stated they would not fund health-focused research in 2009. Although the effort was shared across the health-related social sciences and humanities research community, CCQHR contributed to the eventual reversal of this decision (Government of Canada, S. S. and H. R. C. 2017). The commitment of members of the CCQHR community to participate on funding council advisory boards, lobby for greater funding for CQR, and contribute to research grant review committees are other important avenues whereby CCQHR researchers have had impact on the resources available for CQR.

The fourth and final avenue for impact at the institutional level conceptualized here is activist organizing in the scholarly community. Although it might seem to be better described as a background to or foundation for the other activities just summarized, community organizing can itself impact the institutions of science (Cann & DeMeulenaere, 2020). Bringing researchers committed to CQR together enables the exchange of ideas, development of a common agenda, and presentation of a shared voice to those in positions of power (Webster et al., 2019). In this way, organizing the scholarly community of those committed to CQR is a crucial avenue to identifying the issues that demand attention and supporting coordinated action in response. In the exploration of teaching CQR in the health sciences summarized earlier, Eakin (2016) explicitly drew attention to the importance of a strong “… organizational base, strategic curricular content, and a strong community of practice…” to making the curriculum sustainable and successful (J. M. Eakin, 2016). The organizational base and community of practice that have been established by CCQHR offer important opportunities for the coordinated effort required to achieve changes at the institutional level of science.

These four avenues of impact represent mechanisms by which the CQR community can identify and document their impact on the institutional level of science. The framework presented here offers a way to organize the narratives of impact of CQR, building a more comprehensive approach to communicating the impacts of CQR and CQR researchers on the institutions of science. We now turn to describing mechanisms through which CQR researchers can impact the practice level of science.

Examples of Impact on Science at the Practice Level

The practice level of science is the level of individual learning and action. The avenues to impact on the practice level of science that we emphasize here relate to the possibilities opened up by CQR researchers as we promote new ways of engaging with the scientific process among colleagues both closer and more distant in our professional networks. Impact at the practice level arises from generating deeper understandings among members of the research community about the role of power in knowledge production and the subject matters of research. This requires identifying entry points to the promotion of reflexivity. As impact at the practice level spreads and accumulates, institutional change becomes possible; this is an important way in which the two levels are linked. Although we acknowledge the challenge in maintaining a divide between them, we believe it is constructive for the purposes of understanding the impact of CQR. We outline four avenues by which CQR researchers can have impact at the practice level of science.

The first strategy is by informing new beginnings for academic research, wherein CQR illustrates the positive impacts of establishing a starting point for research based on a deeper knowledge of the flows of power in the world (J. Eakin et al., 1996; J. M. Eakin, 2016; Gastaldo, 1997; Gastaldo et al., 2012; Holmes & Gastaldo, 2002). Such an impact on science has the effect of broadening the community of researchers who actively work on topics that more directly address social injustice, thereby growing the possibilities of impact. Although often rooted in the tensions of interdisciplinary collaboration (Albert et al., 2015), such approaches can make space for more critical messaging to appear in the products of otherwise positivist work (Kontos & Grigorovich, 2018). Members of CCQHR collaborate widely with researchers in disciplines not oriented toward the motivations of critical theory, and have shared their experiences to grow the capacity of the CQR community to promote new beginnings in the work of researchers especially in other health-related disciplines (Kontos & Grigorovich, 2018). For example, in her work critiquing the notion of independence as a goal of rehabilitation practice, Gibson (2016) outlines the value of exploring multiple, constructive inter-dependences as a more appropriate goal of rehabilitation for disabled youth. She presents the goal of achieving constructive inter-dependences as an entirely different starting point for rehabilitation research and practice, thereby proposing alternative orientations toward research in this domain (Gibson, 2016).

The second strategy is generating improved communicative strategies, wherein researchers develop approaches to the generation and re/presentation of data that better communicate the complex insights arising from much critical qualitative research. Better communicative methods are various but may include the use of media that is not solely text-dependent, such as body maps, social maps in Institutional Ethnography, photography exhibitions arising from photo voice research projects, and theatrical performances that re-create the emotional and epistemic realities captured in qualitative research (J. M. Eakin & Endicott, 2006; Gastaldo et al., 2018; Rossiter et al., 2008). By enabling better communication between participants and researchers, and between researchers and other “knowledge users”, such methods can enhance the impact of critical research both on science and on practice. Members of CCQHR have advanced several strategies to alternatively convey the meaning of health-related phenomena, often contributing to advanced understandings of such methods. For example, Gastaldo and Magalhaes have done important work to advance and refine body map storytelling as a methodology for the health sciences that is rooted in an effort to support participants’ expression of their embodied and social experiences of health through visual and narrative approaches that are accessible for general audiences (Gastaldo et al., 2018).

A third strategy is through the development of alternative logics for study design. By moving away from the conventional scientistic approach to intervention and experimentation, critical qualitative research opens up alternative avenues to discovery. The open mentality that characterizes inductive research using critical qualitative data generation and analysis methods is capable of establishing insights that conventional science is unable to achieve (e.g., participatory design, intervention, and evaluation that creates conditions for a project to become a sustainable program in a community organization). One strong example is the continued impact of community-based participatory research as a strategy to both systematically learn about the world and also intervene in collaboration with a particular community (Israel et al., 1998; Jull et al., 2017; Poland & Cohen, 2020; Sakamoto et al., 2009; Sullivan et al., 2005). Advancing CQR methodology as alternatives to conventional positivist approaches to health-related research is central to the mission of CCQHR, and many examples exist of members aiming to do so (J. Eakin, 2010; J. Eakin et al., 1996; Shaw, 2016).

A fourth strategy is by advancing the role of critical reflexivity in the practice of science. Critical reflexivity refers to the practice of actively questioning and challenging one’s own taken-for-granted assumptions about knowledge and the world in order to gain deeper understandings of how and why we come to know in particular ways, and incorporates such reflection throughout the research process (J. M. Eakin & Gladstone, 2020; Finlay, 2002; Lunn Brownlee et al., 2017; Shaw, 2016). Reflexivity is crucial, in that it enables the continued critical consciousness to examine assumptions and travel in new directions in research. Reflexivity is never finished, and therefore plays a central role in the evolution of science (Archer, 2009). As reflexivity is encouraged across the practice of science, research strategies stand to grow in their capability to promote self-critique, a critical posture toward science and its outputs, and therefore also to social justice. Several activities of the CCQHR community, including engaging in research collaborations, disseminating academic publications, and teaching, embed the effort to promote reflexivity in the practice level of science. For example, a course offered on qualitative data analysis explicitly builds skills in locating oneself in the social world and explicitly articulating one’s assumptions and worldview. In so doing, the course builds skills in critical reflexivity that is carried through careers and into future collaborations.

Conclusion

As a domain of scientific practice, CQR has historically and continues to face immense challenges from the institutions of science (Kontos & Grigorovich, 2018; Webster et al., 2019). Conventional science is increasingly linked with a corporate agenda that supports the commercialization of science and incremental improvements to existing neoliberal economic models (Cann & DeMeulenaere, 2020; Shapin, 2009), pushing to the margins any approaches that question the status quo. This means that CQR focused on naming and addressing white supremacy, colonial practices, ableist tendencies, and structural oppression are often pushed to the side and structurally silenced. In addition to the institutional structures that systematically push such approaches to the side are the overt epistemic challenges, derogatory comments, and general lack of interest from researchers who lack a basic understanding of the history of science and the variety of approaches to CQR.

The framework presented here illustrates one conceptualization of the ways in which CQR researchers can have impact on practice, policy and science. Implicit in the framework is the assumption that CQR researchers must engage with the institutions of science if they are to have sustainable careers that advance the position of CQR within their local organizations and scientific institutions writ large (Webster et al., 2019). We acknowledge the tensions associated with such engagements with hegemonic institutions, and invite critiques of our approach that deepen insight into the relationship between CQR and the research impact agenda. Ultimately, we call for the entire CQR community, including researchers, university administrators, and research funders, to more comprehensively frame and encourage critical engagement with the research impact agenda.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

James Shaw

Brenda Gladstone

Fiona Webster

References

Albert

Paradis

, & Kuper

(2015). Interdisciplinary promises versus practices in medicine: The decoupled experiences of social sciences and humanities scholars. Social Science & Medicine, 126, 17-25. https://doi.org/10.1016/j.socscimed.2014.12.004.

Alla

Hall

W. D.

Whiteford

H. A.

Head

B. W.

Meurk

C. S.

(2017). How do we define the policy impact of public health research? A systematic review. Health Research Policy and Systems, 15(1), 84–12. https://doi.org/10.1186/s12961-017-0247-z

Antonio

R. J.

(1981). Immanent critique as the core of critical theory: Its origins and developments in hegel, marx and contemporary thought. British Journal of Sociology, 32(3), 330–345. https://doi.org/10.2307/589281

Archer

M. S.

(2009). Conversations about reflexivity. Routledge.

da Silva

R. G. L.

(2021). Globalizing the scientific Bandwagon: Trajectories of precision medicine in China and Brazil. Science, Technology, & Human Values, 46(1), 192–225. https://doi.org/10.1177/0162243920930282

Bastow

Dunleavy

Tinkler

(2014). The impact of the social sciences: How academics and their research make a difference. Sage.

Berg

Seeber

B. K.

(2018). The slow professor. University of Toronto Press.

Bloor

(1991). Knowledge and social imagery. University of Chicago Press.

Boswell

Smith

(2017). Rethinking policy ‘impact’: Four models of research-policy relations. Palgrave Communications, 3(1), 1–10. https://doi.org/10.1057/s41599-017-0042-z

10.

Cann

DeMeulenaere

(2020). The activist academic: Engaged scholarship for resistance, hope and social change. Stylus Publishing, LLC.

11.

Centre for Critical Qualitative Health Research . (2019). What is ‘Critical’? https://ccqhr.utoronto.ca/about-cq/what-is-critical/.

12.

Centre for Critical Qualitative Health Research . (2020). Core curriculum. https://ccqhr.utoronto.ca/education/about-course-series/core-curriculum/.

13.

Chalmers

A. F.

(2013). What is this thing called science? Hackett Publishing.

14.

Colley

(2014). What (a) to do about ‘impact’: A bourdieusian critique. British Educational Research Journal, 40(4), 660–681. https://doi.org/10.1002/berj.3112

15.

Cruz Rivera

Kyte

D. G.

Aiyegbusi

O. L.

Keeley

T. J.

Calvert

M. J.

(2017). Assessing the impact of healthcare research: A systematic review of methodological frameworks. PLoS Medicine, 14(8), Article e1002370. https://doi.org/10.1371/journal.pmed.1002370

16.

Eakin

(2010). Towards a ‘standpoint’perspective: Health and safety in small workplaces from the perspective of the workers. Policy and Practice in Health and Safety, 8(2), 113–127. https://doi.org/10.1080/14774003.2010.11667751

17.

Eakin

J. M.

(2016). Educating critical qualitative health researchers in the land of the randomized controlled trial. Qualitative Inquiry, 22(2), 107–118. https://doi.org/10.1177/1077800415617207

18.

Eakin

J. M.

Endicott

(2006). Knowledge translation through research-based theatre. Healthcare Policy, 2(2), 54–59. https://doi.org/10.12927/hcpol.2007.18528

19.

Eakin

J. M.

Gladstone

(2020). “Value-adding” analysis: Doing more with qualitative data. International Journal of Qualitative Methods, 19, 1609406920949333. https://doi.org/10.1177/1609406920949333

20.

Eakin

Robertson

Poland

Coburn

Edwards

(1996). Towards a critical social science perspective on health promotion research. Health Promotion International, 11(2), 157–165. https://doi.org/10.1093/heapro/11.2.157

21.

Feenberg

(1981). Lukács, Marx and the sources of critical theory. Oxford University Press

22.

Finlay

(2002). Negotiating the swamp: The opportunity and challenge of reflexivity in research practice. Qualitative Research, 2(2), 209–230. https://doi.org/10.1177/146879410200200205

23.

Fowler

P. B.

(1997). Evidence-based everything. Journal of Evaluation in Clinical Practice, 3(3), 239–243. https://doi.org/10.1046/j.1365-2753.1997.00010.x

24.

Fuller

(1999). Governance of science: Ideology and the future of the open society. McGraw-Hill Education.

25.

Gastaldo

(1997). Is health education good for you? Re-thinking health education through the concept of bio-power. Foucault, Health and Medicine, 113(3), 21.

26.

Gastaldo

Carrasco

Magalhães

(2012). Entangled in a web of exploitation and solidarity: Latin American undocumented workers in the greater Toronto area. http://www.migrationhealth.ca/undocumented-workers-ontario/summary-findings

27.

Gastaldo

Rivas-Quarneti

, & Magalhães

(2018). Body-map storytelling as a health research methodology: Blurred lines creating clear pictures. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 19(2), 1-17. https://doi.org/10.17169/fqs-19.2.2858.

28.

Gibson

(2016). Rehabilitation: A post-critical approach. CRC Press.

29.

Government of Canada, S. S. and H. R. C. (2017, May 11). Social Sciences and Humanities Research Council. https://www.sshrc-crsh.gc.ca/funding-financement/apply-demande/background-renseignements/selecting_agency-choisir_organisme_subventionnaire-eng.aspx

30.

Greenhalgh

Fahy

(2015). Research impact in the community-based health sciences: An analysis of 162 case studies from the 2014 UK research excellence framework. BMC Medicine, 13(1), 232–312. https://doi.org/10.1186/s12916-015-0467-4

31.

Greenhalgh

Raftery

Hanney

Glover

(2016). Research impact: A narrative review. BMC Medicine, 14(1), 78–16. https://doi.org/10.1186/s12916-016-0620-8

32.

Guenther

Falk

I. H.

(2021). Generalising from qualitative evaluation. Evaluation Journal of Australasia, 21(1), 7–23. https://doi.org/10.1177/1035719x21993938

33.

Haraway

(2006). A cyborg manifesto: Science, technology, and socialist-feminism in the late 20th century. In The international handbook of virtual learning environments (pp. 117–158). Springer.

34.

Herbst

(2013) The institution of science and the science of institutions: The legacy of Joseph Ben-David (Vol. 302). Springer Science & Business Media.

35.

Hill

(2016). Assessing (for) impact: Future assessment of the societal impact of research. Palgrave Communications, 2(1), 1–7. https://doi.org/10.1057/palcomms.2016.73

36.

Holmes

Gastaldo

(2002). Nursing as means of governmentality. Journal of Advanced Nursing, 38(6), 557–565. https://doi.org/10.1046/j.1365-2648.2002.02222.x

37.

Hoppe

(2005). Rethinking the science-policy nexus: From knowledge utilization and science technology studies to types of boundary arrangements. Poiesis & Praxis, 3(3), 199–215. https://doi.org/10.1007/s10202-005-0074-0

38.

Horkheimer

(1972a) Critical theory: Selected essays (Vol. 1). A&C Black.

39.

Horkheimer

(1972b). Traditional and critical theory. Critical Theory: Selected Essays, 188, 201.

40.

Israel

B. A.

Schulz

A. J.

Parker

E. A.

Becker

A. B.

(1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health, 19(1), 173–202. https://doi.org/10.1146/annurev.publhealth.19.1.173

41.

Jackson

A. Y.

, & Mazzei

L. A.

(2013). Plugging one text into another: Thinking with theory in qualitative research. Qualitative Inquiry, 19(4), 261-271.

42.

Jackson

A. Y.

Mazzei

L. A.

(2018). Thinking with theory. In A new analytic for qualitative inquiry (pp. 717–737). The SAGE Handbook of Qualitative Research.

43.

Jasanoff

(2004). States of knowledge: The co-production of science and the social order. Routledge.

44.

Johnson

D. R.

(2020). A differential association theory of socialization to commercialist career paths in science. Science, Technology, & Human Values, 45(3), 381–404. https://doi.org/10.1177/0162243919854514

45.

Jull

Giles

Graham

I. D.

(2017). Community-based participatory research and integrated knowledge translation: Advancing the co-creation of knowledge. Implementation Science, 12(1), 150–159. https://doi.org/10.1186/s13012-017-0696-3

46.

Kontos

, & Grigorovich

(2018). “ Sleight of hand” or“ selling our soul”? Surviving and thriving as critical qualitative health researchers in a positivist world. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 19(2), 1-16. https://doi.org/10.17169/fqs-19.2.2990.

47.

Latour

(1987). Science in action: How to follow scientists and engineers through society. Harvard university press.

48.

Lunn Brownlee

Ferguson

L. E.

Ryan

(2017). Changing teachers’ epistemic cognition: A new conceptual framework for epistemic reflexivity. Educational Psychologist, 52(4), 242–252. https://doi.org/10.1080/00461520.2017.1333430

49.

MacDonald

(2017). Impact, research and slaying zombies: The pressures and possibilities of the REF. International Journal of Sociology and Social Policy, 37(11/12), 1-15. https://doi.org/10.1108/IJSSP-04-2016-0047

50.

Machen

(2020). Critical research impact: On making space for alternatives. Area, 52(2), 329–341. https://doi.org/10.1111/area.12574

51.

Mol

(2003). The body multiple. Duke University Press.

52.

Mykhalovskiy

Weir

(2004). The problem of evidence-based medicine: Directions for social science. Social Science & Medicine, 59(5), 1059–1069. https://doi.org/10.1016/j.socscimed.2003.12.002

53.

Oakley

(2002). Social science and evidence-based everything: The case of education. Educational Review, 54(3), 277–286. https://doi.org/10.1080/0013191022000016329

54.

Oancea

(2013). Interpretations of research impact in seven disciplines. European Educational Research Journal, 12(2), 242–250. https://doi.org/10.2304/eerj.2013.12.2.242

55.

Pain

Kesby

Askins

(2011). Geographies of impact: Power, participation and potential. Area, 43(2), 183–188. https://doi.org/10.1111/j.1475-4762.2010.00978.x

56.

Papoutsi

Shaw

Paparini

Shaw

(2021). We need to talk about complexity in health research: Findings from a focused ethnography. Qualitative Health Research, 31(2), 338–348. https://doi.org/10.1177/1049732320968779

57.

Pestre

(2004). Thirty years of science studies: Knowledge, society and the political. History and Technology, 20(4), 351–369. https://doi.org/10.1080/0734151042000304330

58.

Pinel

(2021). Renting valuable assets: Knowledge and value production in Academic science. Science, Technology, & Human Values, 46(2), 275–297. https://doi.org/10.1177/0162243920911974

59.

Poland

B. D.

Cohen

(2020). Adaptation of a structured story-dialogue method for action research with social movement activists. Action Research, 18(3), 353–371. https://doi.org/10.1177/1476750317745955

60.

Rossiter

Kontos

Colantonio

Gilbert

Gray

Keightley

(2008). Staging data: Theatre as a tool for analysis and knowledge transfer in health research. Social Science & Medicine, 66(1), 130–146. https://doi.org/10.1016/j.socscimed.2007.07.021

61.

Sakamoto

Chin

Chapra

Ricciardi

(2009). A’NORMATIVE’HOMELESS WOMAN?: MARGINALISATION, EMOTIONAL INJURY AND SOCIAL SUPPORT OF TRANSWOMEN EXPERIENCING HOMELESSNESS. Gay & Lesbian Issues & Psychology Review, 5(1), 2–19

62.

Sayer

(2009). Who’s afraid of critical social science? Current Sociology, 57(6), 767–786. https://doi.org/10.1177/0011392109342205

63.

Shapin

(1995). Here and everywhere: Sociology of scientific knowledge. Annual Review of Sociology, 21(1), 289–321. https://doi.org/10.1146/annurev.so.21.080195.001445

64.

Shapin

(2009). The scientific life: A moral history of a late modern vocation. University of Chicago Press.

65.

Shaw

J. A.

(2016). Reflexivity and the “acting subject” conceptualizing the unit of analysis in qualitative health research. Qualitative Health Research, 26(13), 1735–1744. https://doi.org/10.1177/1049732316657813

66.

Straus

Tetroe

Graham

I. D.

(2013). Knowledge translation in health care: Moving from evidence to practice. John Wiley & Sons.

67.

Sullivan

Bhuyan

Senturia

Shiu-Thornton

Ciske

(2005). Participatory action research in practice: A case study in addressing domestic violence in nine cultural communities. Journal of Interpersonal Violence, 20(8), 977–995. https://doi.org/10.1177/0886260505277680

68.

Thompson

(2005). Review: A brief history of neoliberalism. Retrieved from http://rebels-library.org/files/d3Thompson-1.pdf

69.

Webster

(2020). The social organization of best practice: An institutional Ethnography of physicians’ work. Springer.

70.

Webster

Gastaldo

Durant

Eakin

Gladstone

Parsons

Peter

, & Shaw

(2019). Doing science differently: A framework for assessing the careers of qualitative scholars in the health sciences. International Journal of Qualitative Methods, 18, 1609406919838676. https://doi.org/10.1177/1609406919838676.