Abstract
Hiring and promotion of qualitative researchers in the health sciences, in Canada and internationally, is impacted by the prestige of quantification as the ultimate measure of scientific quality in current academic and health-care settings. This is further exacerbated by neoliberal notions of productivity, which offer very limited forms of assessment for different ways of producing knowledge or doing science differently. While qualitative researchers share the effects of the politics of productivity and corporate university policies with other academics, we argue that they are disadvantaged by the combination of the latent biomedical conservatism that characterizes the health sciences in Canada with the lack of frameworks to acknowledge and properly assess alternative forms of interdisciplinary scholarship. In our experience, it is challenging for qualitative researchers to advance in Canadian health sciences faculties. In light of this, we propose a framework for evaluating their scholarly work. We have structured this article in three sections: (a) to characterize the academic landscape in which qualitative health scholars find themselves when housed in Canadian faculties of medicine and their schools of health sciences, (b) to report on an organizational scan we undertook in order to understand current practices of evaluating scholarly productivity at these institutions, and (c) to propose a set of criteria that could more appropriately evaluate the contributions made by qualitative researchers working in the health sciences.
Introduction
Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.
In our experience, it is challenging for qualitative researchers to advance in Canadian health sciences faculties. Hence, we propose a framework for evaluating their scholarly work. We have structured this article in three sections: (a) to characterize the academic landscape in which qualitative health scholars find themselves when housed in Canadian faculties of medicine and their schools/departments of health sciences, (b) to report on an organizational scan we undertook in order to understand current practices of evaluating scholarly productivity at these institutions, and (c) to propose a set of criteria that could more appropriately evaluate the contributions made by qualitative researchers working in the health sciences.
While we use examples of assessment employed in the health sciences and in Canada, we believe qualitative researchers in other disciplines and other countries could potentially adapt our framework. There are considerable discrepancies in assessment among universities in low-, medium-, and high-income countries and across academic disciplines. Yet shared trends in research evaluation have led to a lack of regard for quality, rushed measurement of bibliometric indicators and the valuing of socially decontextualized articles published in “international” (Anglo-American) journals, leading to the assumption of a “productivity inferiority” (Bosi, 2012, p. 582) of qualitative researchers (Bosi, 2012; Camargo, 2013; Musiige & Maassen, 2015) who do not work in English.
Context: Neoliberalism and Scientism
Two global forces have been shaping academia and the measures of scholarly output for several decades now, namely neoliberalism (and its call for increased productivity) and scientism. The career trajectories of qualitative researchers must be understood in this context. Neoliberalism, as put forth by Harvey (2005) and described by Thompson (2005), describes the gradual shift toward economic and social policies that have given an increased liberality and centrality to markets, market processes, and to the interests of capital…. Neoliberalism is the intensification of the influence and dominance of capital; it is the elevation of capitalism, as a mode of production, into an ethic, a set of political imperatives, and a cultural logic. (pp. 22–23)
Internationally, there has been an application of neoliberal thought to the management of higher education, creating what today is described as the corporate university (Donoghue, 2008; Washburn, 2006). Berg and Seeber (2016) argue that “[c]orporatization has compromised academic life and sped up the clock.” Presently, the corporate university submits pedagogical and intellectual concerns to an economic and managerial logic (Berg & Seeber, 2016), where promotion, tenure, and merit increases, and thus academics’ livelihood, are subordinated to a simplistic form of evaluation mainly based on counting outputs and measurement using generic standards.
Accountability, productivity, and efficiency are the hallmarks of the corporate university (Berg & Seeber, 2016; Busch, 2017). For instance, under the neoliberal logic, PhD programs across disciplines are being reduced to 3 years in some European and Latin American countries and scholars in small, interdisciplinary fields are deemed to be lacking impact because their papers have few citations (Busch, 2017). Busch (2017) notes that “Given that new and interdisciplinary areas are likely to be small, this implies a clear bias against innovation in research” (p. 69).
The concurrent force of scientism is also having a profound impact. “Scientism can be understood as the view that science, and only science, reveals the truth, such that all true claims are part of a true scientific theory, or are reducible to claims of this sort” (Loughlin, Lewith, & Falkenberg, 2013, p. 131). It represents an exaggerated trust in the efficacy of the methods of natural science (e.g., physics, biology) and proposes they should be the standard for knowledge production in all areas (e.g., social sciences and the humanities). This is not a critique against science, but rather against a narrow definition of science that excludes diverse ways of producing scientific knowledge.
Scientism is shaping our worldview, including the way we view the health sciences, because it frames how we interpret common concepts such as evidence and rationality, refusing to consider human experience, politics, subjectivity, and values, in knowledge production (Loughlin et al., 2013). Methods from natural science are being used to study human action in a way that is marginalizing other approaches that may be qualitative and explicitly theory-driven (Fitzpatrick, 2015). Many health scientists frequently face institutional barriers for hiring and promotion because they do science differently, which means engaging with perceptions, narratives, intersubjectivity, cultural practices, social structures, and power relations when studying health and illness. However, doing science differently in order to engage with such issues is supported on occasion by prestigious authors in traditional journals.
Given that scientism is so deeply entrenched and embraced in biomedicine, it may be difficult to recognize that it is only one particular worldview among many. Viewing the nature of evidence (and its evaluation) in such a narrow fashion has negative implications for health research.
An “Uphill” Landscape: Challenges Faced by Qualitative Researchers
For qualitative researchers, their interdisciplinary and nontraditional scientific work cannot be appropriately evaluated using the norms/metrics typically used in biomedicine or the natural sciences. Unfortunately, this is exactly what happens in many instances when qualitative researchers apply for positions in or are embedded in faculties of medicine or health sciences where they are evaluated primarily in terms of productivity and consideration for promotion. In using these mismatched metrics, qualitative health researchers may be viewed as “inferior” as opposed to being simply “different” and in need of assessment that meets the standards of other ways of producing knowledge or doing science.
As has been noted by several scholars (Hart, Poole, Facey, & Parsons, 2017; Kontos & Grigorovich, 2018; Kreiman & Maunsell, 2011), the current emphasis on “extreme productivity” in research compromises thoughtful inquiry. This is unfortunately a widespread discourse in the health sciences and is not restricted to only those conducting qualitative research. However, this is specifically problematic for the assessment of qualitative researchers for several reasons. We focus on four assessment elements that pose challenges for qualitative health researchers, which are features of qualitative research, bench science models of productivity, publication, and funding.
Features of Qualitative Research
High-quality qualitative research is time-consuming as this approach itself requires the ongoing presence of the researcher, thus placing particularly high demands, sometimes hundreds of fieldwork hours on data generation and on the emergent (rather than a priori) theorization required of qualitative data analysis and interpretation (Eakin, 2016). Many qualitative methodologies require long, individual researcher engagement (e.g., ethnographic research or archival work on the history of public health), but researchers using these methodologies have been devalued for not producing journal articles at a desirable speed and disregarded because books and book chapters have been downgraded in the health sciences (Busch, 2017; Donoghue, 2008). Further, the common thread of many qualitative programs of research is the concepts and approaches adopted by a scientist rather than focus on a sole content area. For example, when studying health inequities in Canada, a researcher may work on topics that are apparently unrelated, such us immigrants, homeless youth, and low-income single parents. Yet, building conceptual, methodological, or theoretical expertise is often not acknowledged by promotion committees. A third element that is common is students under the supervision of a researcher do not study the same substantive area as the supervisor; rather, they share a theoretical or methodological interest, creating an impression of incoherent scholarship for the supervisor. This can have deleterious effects on the promotion of qualitative researchers embedded within health sciences faculties, where a specific substantive area is considered the norm.
Productivity Based on Bench-Science Model
When a uniform or simplistic understanding of the diverse forms of knowledge production occur in the health sciences, bench science or lab models tend to be used as the norm. This view assumes one leader working with several students from undergraduate students to postdoctoral fellows, with senior students supervising junior ones, working simultaneously on several projects, and publishing papers from 1,000 to 3,000 words as a team. This model has created what is known today as the hyperprolific researcher (Lawrence, 2003). These assumptions are radically different from the experiences of qualitative researchers. Given the expertise required for data collection and analysis in qualitative research, principal investigators undertake most of the study phases themselves, devoting considerable time to train and supervise each student working in the project and commonly devoting months to lead, if not solely conduct, data analysis.
Publication
Low impact factor is a characteristic of niche areas of expertise (e.g., bioethics) and of most social sciences and nursing journals, in which most qualitative research is published; some medical journals (i.e., British Medical Journal [BMJ]) recently started rejecting qualitative studies as a matter of mandate (Greenhalgh et al., 2016; Webster, 2016), and many do not have reviewers for qualitative inquiry. Therefore, it is inevitable that even the most productive and world-class qualitative researchers are more likely to publish in journals with low impact factors and/or have low “h” scores, despite being considered leaders in their own fields (Burrows, 2012). Richardson and Liddle’s (2017) study on where to publish good quality qualitative studies in the health sciences (journals where six or more qualitative health papers listed in the UK’s 2014 Research Excellence Framework) reveals that the average impact factor (over 5 years) was only 2.0425, excluding the BMJ, which is low relative to many leading clinical journals. These were a combination of clinical, sociology of health, and qualitative methodology journals.
The lack of preparation of editors and reviewers in several clinical and public health journals constitutes a barrier for publications (Gastaldo, 2016). Among the authors of this article, many have received comments such as “this is not a scientific study,” “this is a biased study,” or “nine participants seems like a low number” (in a 10-month study with over 200 hours of fieldwork and hundreds of pages of raw data), “I failed to understand what it [medicalization] means,” and “What does the poststructural perspective refer to?” (as feedback for a poststructuralist analysis of medicalization). Such lack of theoretical and methodological preparation systemically create barriers for qualitative researchers who attempt to publish in clinical journals with higher impact factors. Conversely social sciences journals are often not recognized or given as much stature by promotions committees, which favor empirical multiauthored papers and give less credit for solo empirical or theoretical work.
A final limitation is the disregard for the number of pages/words written for a publication by a single author or a small number of authors and simple counting of number of papers published. Most clinical journals have a word count of 2,500 words on average (ranging up to 4,000 words in some instances), whereas journals that publish qualitative research have more than double the higher range (Social Science & Medicine has 9,000 words) and some have no word limit.
Funding
Qualitative scholars face disadvantages in relation to the funding of their work. Within Canada, Albert, Paradis, and Kuper’s (2015) and Albert and Laberge’s (2018) work has been instrumental in explicating the absence of social scientists or humanities scholars on the Governing Council of the Canadian Institutes of Health Research (CIHR), the largest national funding body for health research. This can lead to what Lamont (in Albert & Laberge, 2018) has termed “cognitive contextualization,” when panelists in interdisciplinary panels tend to favor their own methodological preferences rather than evaluate proposals according to the epistemological and methodological standards that prevail in the applicant’s discipline. Reviewers with low scientific literacy beyond quantification and dominant discourses that perpetuate qualitative research as representing “a low level of evidence,” “soft science,” “context-specific,” and “inherently biased” all shape the disadvantages faced by qualitative researchers when competing for funding from health sciences funding agencies.
In addition, qualitative research studies are often considered less expensive to conduct than clinical trials (even though, when trials have large teams, the amount that accrues to any one individual researcher might sometimes be modest). For this reason, the total grant capture by qualitative researchers will often be lower than that of some of their quantitative colleagues, even when the costs of undertaking this research are not appreciably lower than many other designs, such as survey research. As an example, among the authors of this article, several have had 2- to 3-year grants funded for less than CAD 100,000 or as small as CAD 10,000. In addition, one author has been told to reduce their overall budget and training opportunities within a qualitative project submitted to CIHR, as they were considered too high for a qualitative study (postdoctoral fellowship).
One group of researchers conducted a systematic review identifying 24 existing methodological research impact frameworks (Cruz Rivera, Kyte, Aiyegbusi, Keeley, & Calvert, 2017). In total, these studies recommended 80 different metrics aimed at measuring impact. The authors of this review note that “there is a need for researchers to understand existing methodological impact frameworks and the means by which impact may be quantified.” This emphasis on quantification across the many frameworks reflects the prevailing institutionalized evaluative environment in which qualitative researchers are assessed. In our own review of CIHR's definitions of research impact we noted that many of the categories of impact have features that might disadvantage qualitative scholarship. For example, specific criteria are often largely related to number of publications, journal impact factors, citation counts, and source and dollar amounts of funding. A close reading of the explicit and implicit evaluative criteria in these categories excludes or diminishes the value of much of contributions of qualitative health.
Finally, when these elements are combined, an additional factor that plays a role in disadvantaging qualitative researchers is the lack of qualitative researchers involved in hiring and promotion committees. When there are qualitative researchers on committees, they are often the lone representative with expertise in this form of knowledge production and thus lack the benefit of validation or support from other colleagues. The fact that exclusively clinical and epidemiological researchers often compose the promotions committee has been taken for granted and never problematized. The “natural” explanation for lack of hiring or promotion has been that these qualitative researchers were not as productive as their colleagues. A lack of understanding of cognitive contextualization, power dynamics, and dominant discourses leads to blaming individuals for their so-called failure rather than acknowledging structural forces that lead to discrimination and unfair treatment.
Do Formal, Broad Criteria Mask Exclusionary Practices?
In order to locate the problems noted above in a broader social context, we undertook an organizational scan pertaining to the evaluation of the impact of qualitative research in the context of academic promotion. The purpose of this scan was to identify and describe the formal institutional processes at faculties of health sciences and medical schools across Canada in order to help us develop a framework to support hiring and promotion of qualitative health researchers. Our focus was on Canadian schools who specialize in these fields within faculties of medicine, not to exclude other faculties of health sciences but as a proxy for them. We were curious to understand the extent and nature of policies and studies pertaining to academic promotion among qualitative research specialists. For a detailed description of the scan terms and findings, see Online Appendix B.
Our extensive search identified few relevant results and no detailed peer-reviewed studies pertaining directly to our topic or issues of potential relevance. Our review suggests that, in their general guidelines for promotion, Canadian faculties of medicine and their schools/departments of health sciences do not explicitly establish quantitative criteria for assessment (e.g., number of papers published in 1 year). Moreover, most place a strong emphasis on teaching (and often administration) alongside research. Those which specify how research should be evaluated generally include statements about “creativity” and “quality,” in some cases implying that any consideration of quantity must be balanced with these criteria.
Discourses of creativity and quality, however, were rarely explicitly described. This would seem to leave room for de facto standards for promotion with built-in preferences for certain types of scholarship, and indeed most policies would not actually preclude explicit bibliometric criteria at the departmental level. While Canadian universities do not appear to impose specific bibliometric criteria in their official policies and guidelines for promotion, the extent to which processes actually accommodate qualitative research and creative communications appears to be largely at the discretion of departmental and faculty committees, with a wide variety of guidelines tending toward broad definitions of scholarship. While these policies offer guidelines, it cannot be said with complete confidence that they do not reinforce the dynamics suggested by commentators on issues affecting qualitative researchers. The breadth of the language, the frequent references to boards of review and departmental authorizations, and the diversity in the purpose and style of the documents located in our search all suggest that discretionary decisions play a role in determining how various works and accomplishments are valued in the promotions process.
A New Framework for Evaluating Qualitative Researchers’ Careers for Hiring and Promotion
In the absence of other formal or written assessment criteria, we offer the following guide to help committees evaluate the careers of qualitative researchers. We understand this is a preliminary attempt and we hope to see this framework further developed for national and international relevance. Our framework focuses on solutions to some of the key challenges presented in this article and provides suggestions to qualitative health researchers when presenting their body of work to review committees for both hiring and promotion in faculties of health.
Table 1 has been organized into several key domains common to all research (paradigmatic location, complexity of the research, leadership and originality, longitudinal impact, methodological innovation, scope and multiplicity of innovations, publication record, and grant applications and international collaboration). The second column describes the aspects of these domains that are unique to qualitative research, and the third column offers suggestions for new criteria that could be used to evaluate these unique features.
Criteria to Appraise the Impact of Qualitative Health Researchers.
Limitations and Future Research
Our framework provides an important guideline that can be used by qualitative researchers pursuing their careers in health sciences faculties and possibly to qualitative researchers more broadly, nationally, and internationally. Our goal is to promote fairer assessment of qualitative research and scholarship in the process of hiring and promotion. However, we realize there are many areas for future development that are beyond the scope of our current article. We propose as future lines of inquiry: investigation of departmental and internal faculty policies in the health sciences through requests for information and surveys; investigation that connects the gender gap in science given that the majority of qualitative health researchers are women (Baker, 2010; Holman et al., 2018); studies of promotions, by comparing career trajectories of different types of researchers or examining faculty composition or both (Albert & Laberge, 2018); and key informant or survey research among candidates, decision makers, and/or policy makers to explore further how criteria are interpreted and applied as well as general issues related to the topic.
Conclusion
In this article, we have outlined how qualitative researchers are doing science differently and therefore require appropriate metrics for their knowledge production and scholarly output. We conducted an environmental scan that highlighted how faculties of health do not offer explicit criteria for evaluation for promotion and/or hire. We have offered here a preliminary framework that can be used for further discussion and guide practice, with the goal of developing a useful and comprehensive guide for judging qualitative science that can be disseminated in university and research communities. In doing so, we join a large community of scholars internationally who are working toward countering the politics of productivity and scientism that exert a strong influence on the careers of qualitative researchers working in the health sciences.
Supplemental Material
Supplemental Material, Online_Appendix - Doing Science Differently: A Framework for Assessing the Careers of Qualitative Scholars in the Health Sciences
Supplemental Material, Online_Appendix for Doing Science Differently: A Framework for Assessing the Careers of Qualitative Scholars in the Health Sciences by Fiona Webster, Denise Gastaldo, Steve Durant, Joan Eakin, Brenda Gladstone, Janet Parsons, Elizabeth Peter and James Shaw in International Journal of Qualitative Methods
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
