Post-Normal Science,Post-COVID: Multi-Method Approaches to Actionable Research

Abstract

Researching complex public health issues, particularly in rural settings, requires pragmatic approaches that emphasize local perspectives, actionable findings, and timely knowledge mobilization. This paper presents a multi-phased, place-based methodology employed by a practitioner-researcher to conduct qualitative research, share findings in real time, and co-create practical recommendations in a large rural geography. This paper describes the study’s use of post-normal science and outlines the methodology from conceptualization to conclusion. Next, it comments on both scientific and political rigour of the research, using contributions from the lead author’s reflective journal. With empirical findings from the study reported elsewhere, this paper presents an assessment of the research approach itself. Its contribution to the field is as a case study of an iterative, multi-method qualitative strategy bridging academic quality with practical, on-the-ground relevance.

Introduction

We are living in a global poly-crisis characterized by interconnected health, social, and ecological issues (Lawrence et al., 2024), and risks that transcend national borders (Beck, 2016). While many of the forces shaping these complex issues operate on a global scale, they can be locally influenced and locally felt, and it is in local places that they influence human health and wellbeing (Burris & Lin, 2021). At the local level, responses to such challenges are shaped by local governance - an assemblage of people and systems (Burris & Lin, 2021; de Leeuw, 2022). Conducting research to understand and engage the complex system of local governance during a global crisis is challenging and demands approaches beyond traditional positivist methods.

Purpose

It has been suggested that learning about research is best done by studying actual research (Hunter & Brewer, 2016). The purpose of this paper is to provide an example of pragmatic, qualitative multi-method research that allowed critical exploration of a complex public health issue and led to actionable findings. Multiple-methods studies are often accompanied by tensions rooted in philosophy, theory, and methodology (Bazeley, 2016); we explore these below.

In the case study below, we describe the lead author (AM)’s doctoral research project. Following a brief introduction to the topic of study and the researcher’s positionality, we describe a sequential, qualitative multi-method approach that aimed to provide timely, actionable findings for local communities and sectors involved in rural governance during times of disruption. Next, we apply frameworks to assess the scientific and political rigour of this research, integrating entries from AM’s research journal for insight.

Theoretical Considerations: Pluralism, Post-normal Science, Systems Thinking and Transdisciplinarity

Pluralism is a central theme in this research. Kim et al. (2022), in their discussion of paradigms for urban health (a close counterpart to rural health, which forms the context for this case) highlight the importance of focusing more on coherence than consensus among disciplines to address the complex and unpredictable dynamics of urban health and to foster societal change “by creating and applying knowledge about real-world problems and solutions” (Kim et al., 2022, p. 3). This insight provides a rationale for the efforts to embrace multiple perspectives among research participants.

The context of this research is marked by high levels of uncertainty and, often, conflicting purposes among actors. The complex challenges align with what Rittel and Webber (1973) described as wicked problems: issues that cannot be solved by traditional or simplistic approaches. Science has traditionally sought to eliminate uncertainty by focusing on how systems work. However, this traditional approach often falls short when addressing complex social and public health challenges, where certainty is elusive, and solutions are difficult to define.

Post-normal science (PNS) offers an alternative paradigm that embraces uncertainty, pluralistic thinking, and complexity, making it well-suited to addressing complex social issues. It does so by making values explicit and seeking to manage uncertainties rather than eliminate them. Post-normal science recognizes the plurality of perspectives inherent to complex issues and widens and broadens the network of actors involved in research, intentionally incorporating diverse bodies of knowledge to foster more effective problem-solving (Funtowicz & Ravetz, 1993). Funtowicz and Ravetz (1993) present PNS as a complex approach that encapsulates both applied science and professional consultancy, useful when neither of these is sufficient on its own to address the issue at hand.

PNS can be supported by two complementary approaches: systems thinking and transdisciplinarity. Systems thinking involves studying patterns within systems to anticipate and design for change (Arnold & Wade, 2015; Beck, 2006; van der Leeuw, 2018). Transdisciplinarity emphasizes collaboration across disciplines, the creation of new knowledge, and the integration of insights from both academic and non-academic actors (Choi & Pak, 2006). Together, these approaches enable researchers to expand both the scope of their inquiry and the range of knowledge domains drawn upon, fostering deeper understanding and more comprehensive solutions.

Multi-Method Research and Pragmatism

Exploring complex phenomena often requires more than a simple or straightforward study design. Qualitative multi-method research provides a means to gain insight into multiple aspects of a phenomenon (Hesse-Biber et al., 2016). This approach borrows from yet another paradigm, pragmatism, which embraces the plurality of perspectives often associated with complex social problems (Biesta, 2015). Rather than delivering a single depiction of reality, pragmatism seeks useful points of connection, offering a deeper understanding of relationships, actions, and their consequences (Mertens, 2020).

Using the paradigms of post-normal science and pragmatism, this case study embraces systems thinking and transdisciplinarity to explore local governance of a complex local public health issue in rural Northern Ontario, Canada.

Assessing Rigour: Scientific and Political

Ways to measure quality in qualitative research vary and are continuously evolving (Ritchie & Ormston, 2014). Fabregues et al. (2023) documented 16 relevant frameworks for assessing quality of multi-method studies. Among these is the Quality Assessment with Diverse Studies Appraisal (QuADS), which includes thirteen criteria related to research design, data collection, analysis, and reporting (Harrison et al., 2021) (Figure 1). However, societal implications lie at the heart of PNS and Temper et al. (2019) highlight criteria specifically designed to assess a study’s utility to society and efficacy in addressing complex problems. In other words, for transdisciplinary research to be truly useful, it must complement scientific rigour with political rigour. In their discussion of political rigour, Temper et al. (2019) describe seven profiles of activist-scholars developed through an arts-based, exploratory process with groups of scholars in four countries. From this work, they propose ten values relevant to political research (Table 1) and suggest that researchers may also benefit from co-designing their own values when engaging in transgressive, transdisciplinary work.

Figure 1.

Quality Assessment for Diverse Studies (QuADS) Criteria (Harrison et al., 2021)

Table 1.

Principles of Political Rigour (Temper et al., 2019)

Principle	Description
1. Accessibility	Research can be understood broadly and a means for social learning
2. Reflexivity	Critical examination of our own practices, presumptions and assumptions and the power relationships in our work
3. Relevance	Co-defining matter of concern with all involved. Research must be useful to emancipatory efforts of groups we work with
4. Transparency	Clarity of structure, processes and outcome
5. Care-fullness	Relations of care with oneself, loved ones, communities of scholar and participants
6. Respectfulness	How are other forms of knowledge and worldviews valorized, recognized and integrated into the research process
7. Relationality	Research should be grounded and context dependent
8. Reciprocity	Co-design of research question, methods, analysis and outputs works as one method to help ensure reciprocity
9. Fallibility	Possibility to fail and learn from failure
10. Transformativity /Transgression	How is the research transforming power relations and transgressing practice as usual to open up new emancipatory possibilities

Case Study: Identifying Ways to Reduce Impact of Disruption in Rural Northern Ontario

This research project was undertaken in the large geographic area of Northern Ontario, Canada, and aimed to balance local insight, actionable findings, and timely knowledge mobilization, while assuring scientific and political rigour. Empirical findings from the study are reported elsewhere (Mongeon et al., 2024a, 2024b). The qualitative research aimed to facilitate learning and change in real time in a region with a distinct history and health governance structure. It also aimed to contribute critical knowledge to strengthening related systems and community governance efforts. The research took place in the wake of the COVID-19 pandemic, a phenomenon to which researchers have been called to apply PNS (Waltner-Toews et al., 2020), yet PNS is a relatively new concept, and examples of its application are limited. We aim to contribute to the literature by sharing an example, with detailed methodology, of using a PNS approach.

Project Initiation and Literature Review

Research can be catalyzed in many ways, and for AM, a practitioner-scholar, it was from a call from the Canadian Institutes of Health Research for projects that addressed the areas of focus within the 2021 Chief Public Health Officer report, A Vision to Transform Public Health (Public Health Agency of Canada, 2021). AM was at that time emerging from the most acute response phase of the COVID-19 pandemic in her work. She had learned through her COVID experience that in emergencies, the public health system contains many actors, and transforming public health requires including the entire system. AM had just seen the rural Northern Ontario communities in which she works struggle through the response, and an absence of system recovery, despite its importance as part of a cycle of emergency management. Her question became: how can we, broadly as a system, reduce the negative impacts of future disruption in our rural communities?

AM carried out a literature review to provide conceptual clarity to the research, whereby she explored published literature related to rurality, governance, and health, developing a conceptual framework to represent their convergence (Mongeon et al., 2023). This conceptual framework acted as a guide and touchpoint throughout the study.

Methodology

Case Study

This study used a case study design, with Northern Ontario as a macro case, and rural municipalities with populations <10,000 as individual cases. Its aim was to explore the experiences of these municipalities and some of the key organizations that supported them during COVID to identify lessons learned and opportunities to reduce the impact of future disruption. This study used a post-normal approach, aligning with Waltner-Toews et al.’s (2020) call for greater recognition of complexity and for broadening the peer community in relation to the COVID-19 pandemic. As presented in Thomas’ typology of case studies, the approach taken was interpretive, where we sought to understand multiple perspectives (Thomas, 2016). This can further be characterized as a parallel multiple case (several municipalities being studied concurrently), retrospective study (exploring an experience that is in the past).

Relevance to Emergency/Disaster Studies

Hunter and Brewer (2016) describe the field of disaster research as one particularly suited to pragmatic, multimethod approaches to successfully address the multiple audiences and potential users of research. In 2001, researchers Ben Wisner and Maureen Fordham created a collaborative which they dubbed RADIX, for Radical Interpretations of Disaster and Radical Solutions (RADIX, n.d.) This group developed a globally co-created Disaster Studies Manifesto and accompanying Disaster Studies Accord articulating values and principles to guide research in this field, such as valuing local researchers and local research and local epistemologies, recognizing the political nature of disaster research, and addressing the root causes of vulnerability.

Researcher Positionality

Researcher positionality is integral to identifying the ways in which a researcher’s experience, areas of knowledge, values, and beliefs may influence their interpretation of research findings and other decisions within the research process (Holmes, 2020). Following Holmes’ (2020) suggestions for articulating positionality, we take time here to share AM’s positionality.

AM is a recent doctoral graduate who also works in the field as a local public health practitioner, at mid-life. She has deep contextual knowledge, of both the rural region being studied and the work of local public health in this area; she also works as a colleague to many of the participants in the research. Although she chose a pragmatic research approach, she also personally subscribes to a relational worldview, believing that all beings live in relation with one another (Chilisa, 2019) and seeing this connectedness translate into the phenomena studied below. This deep connection between researcher, research subject and research participants led AM to seek a research methodology that accommodated or even capitalized on her positionality.

Methods

AM led the project, under the advisement of LD and KM. The project used multiple qualitative methods, implemented sequentially and each informed by the one before (Figure 2). AM refined the research idea by discussing with colleagues in public health, municipal associations, as well as rural and northern research institutes, and in this process gathered letters of support for the project. The project was successful in securing funding from a national research body; we immediately began the process of seeking ethics approval from the University of Guelph Research Ethics Board and formed a project advisory committee. All organizations that had submitted letters of support for the project, as well as one additional organization with a key role in local emergency management, were invited to join the advisory committee. The first advisory committee meeting took place in October 2022 to orient members to the project and discuss their role: as local advisors, to support quality and local relevance of the project, and support knowledge mobilization. A shared drive was set up to provide advisory committee members access to key project related documents. A research assistant was hired.

Figure 2.

Steps in Multi-Method Sequential Qualitative Study and Assessments of Quality/Rigour

In February of 2023, AM began recruitment. For the purposes of public health service delivery, Northern Ontario at the time was divided into seven regions (Northwestern, Thunder Bay, Algoma, Porcupine, Timiskaming, North Bay Parry Sound, and Sudbury and Districts) ¹ . Although the cases in this study were municipalities, the study was grounded in public health, so the objective was to recruit two municipalities in each of the seven public health regions across Northern Ontario.

Phase 1: Document Review

The first phase of research used document review to identify how municipalities experienced the COVID-19 pandemic (Bowen, 2009). AM and a research assistant gathered municipal agenda packages from January 2020-June 2022, representing the first six waves of the pandemic (Canadian Institute for Health Information, 2022). They were unsuccessful in accessing historical packages from one municipality, therefore this phase included content from six of the seven participating municipalities (Supplemental Table 1). This elicited thousands of pages of documents, more than could be feasibly read in the time available. Informed by the Canadian Institute for Health Information (CIHI) COVID-19 timeline (Canadian Institute for Health Information, 2022), AM and a research assistant chose one package from each participating municipality corresponding with the six first pandemic waves. They reviewed these six packages to identify an initial list of pandemic-related terms that could be used to search remaining documents. They did so independently, then compared and discussed lists to develop a master list. They stopped when no additional keywords were being identified. All municipal documents were uploaded to NVivo 14 and searched by AM using the keywords, then highlighted passages scanned to identify those related to the COVID-19 pandemic. AM then conducted a thematic analysis (Braun & Clarke, 2022), identifying 10 themes.

Results, and a draft interview script informed by these results, were shared with the advisory committee for discussion and reviewed in a second meeting in May 2023.

Phase 2: Semi-Structured Interviews

Phase 2 was carried out from June 2023-April 2024, aiming to explore successes and challenges experienced by small and rural communities in responding to the COVID-19 pandemic. Semi-structured interviews were chosen for this phase for their ability to bridge the specific lines of inquiry identified in the document review with latitude for the interviewer to pursue additional inquiry that may seem relevant as the interview unfolds (Brinkmann, 2023). Participating municipalities were asked to identify one current elected official and one senior municipal staff, who had also been in their role during the COVID-19 pandemic. Due to limited staff capacity and high workload, one municipality identified only an elected representative and no staff. Interviews lasted approximately 90 minutes. At the start of each interview, the interviewer (AM) reviewed the consent to participate, and participants provided verbal consent before the interview began, recorded on paper by the interviewer and a copy later sent by email to the participant. Most interviews were audio and video recorded in MS Teams, with the MS Teams-generated transcript used as the primary data for analysis. Each interview participant was invited to review their transcript and to share feedback, including whether the transcript accurately reflected what the participant meant to say, any new information they would like to add, or anything they would like to change.

Analysis used Braun & Clarke’s reflexive thematic analysis (TA) for its compatibility with the lead author’s positionality (Braun & Clarke, 2022). Reflexive TA facilitates the development of meaning and supports efforts to generate situated knowledge within qualitative research. It also allows the use of both inductive and deductive learning from the data. This research took a more deductive approach, shaped by our conceptual framework. Furthermore, the research applied a critical and constructionist lens, attempting to “unpack the realities that are expressed within the dataset” (Braun & Clarke, 2022, p. 52).

AM completed the six phases of reflexive TA: familiarization with the data set; coding; generating, developing, reviewing, refining, defining, and naming themes; and writing up the findings, all described in Supplemental Tables 2 and 3. AM developed themes from the municipal interviews (presented in Mongeon et al., 2024a). These were then presented to the advisory committee in December 2023. Also presented at this time were proposed interview questions for the organizational interviews, and a list of proposed organizations to invite to interview. Both were revised with advisory committee input. Semi-structured interviews were then carried out with representatives from nine organizations and again data were analyzed using reflexive thematic analysis, again in NVivo 14 (Braun & Clarke, 2022).

Findings from the key informant interviews identified four opportunities for action, to be addressed during the next phase of research. These were presented to the advisory committee for discussion in April 2024, along with an outline of the format for the participatory workshop, described below.

Phase 3: Structured Interview Matrix

The objective of Phase 3 was to identify supports and structures that will help small and rural communities more effectively respond to disruption; this took place from April-July 2024. This final data collection phase involved an in-person Structured Interview Matrix (SIM) process: a participatory, multi-round methodology useful for gathering diverse perspectives and through a deliberative process moving towards, in this case, a synthesized set of recommendations (Mongeon et al., 2024a, 2024b). Originally developed for use in education and further developed by O’Sullivan et al. (2015) as a research method, the Structured Interview Matrix described by Chartier (2002) is a tool with which to facilitate dialogue.

Every municipality or organization that participated in the research was invited to send one individual to the in-person workshop taking place in Sudbury, Ontario, in May 2024. Thirteen participants attended, and in four small groups worked to address the four areas of action identified above, as presented in Supplemental Table 4. The SIM event concluded with three recommendations for each of the areas for action.

AM used Nvivo 14 to develop a final set of project recommendations, based on those made during the SIM workshop and semi-structured interviews, and informed by the literature review and conceptual framework. These were reviewed by the project advisory committee, then further organized and condensed using NVivo 14 to four recommendations with 19 sub-recommendations (Supplemental Table 5) (Mongeon et al., 2024a, 2024b).

Phase 4: Knowledge Mobilization

The final phase of research involved compiling and sharing research findings to inform policy and practice. Due to the topic’s relevance and timeliness, research findings were shared as they were developed, throughout the duration of the project, at conferences and online learning events across a diversity of settings and disciplines. A public-facing report was published alongside a policy brief, both shared with all project collaborators, knowledge users, and participants (Mongeon et al., 2024a, 2024b). In addition, an early draft of the policy brief was submitted to inform a provincial policy consultation. Research findings have also been reported and submitted for peer-reviewed publication as they were developed.

Challenges

Research with multiple components and groups of participants can be inherently vulnerable to challenges and complications; learning from these may help others better anticipate and mitigate them in the future. We encountered two noteworthy challenges in implementing this project. First, was a delay in timelines. The initial project design proposed a twelve-month study. Ethics approval and participant recruitment each took longer than initially anticipated which led to a doubling in the project timeline.

The second notable challenge was in recruitment and retention of research assistance. Midway through the first phase of research, the project’s research assistant left the project for personal reasons. To maintain project momentum as recruitment for another research assistant resumed, AM took on research assistant tasks. A second research assistant was hired to complete the initial one-year term of the research assistant contract. Overall project delays meant that by the end of one year, the project was not yet done. However, the choice of analytical method was compatible with a single researcher approach, and the team proceeded without a research assistant for the second year of the study.

Results and Discussion

Our results and discussion focus on evaluating our study’s scientific and political rigour, using the quality criteria described in Figure 1 and Table 1. To assess scientific rigour, we applied the QuADS critical appraisal tool developed by Harrison et al. (2021), developed to assess the quality of mixed- and multi-method research studies being used in systematic reviews. The results of this quality rating are detailed in Table 2. With possible scores for each criterion from 1-3, we have rated our research’s conceptual foundation, clarity of purpose and description, methodological and analytical transparency and appropriateness, and critical reflection (criteria #1-3,6,7,8-11, 13) with a score of 3. We recognize opportunities to have strengthened the study design, sampling, and local stakeholder engagement (criteria #4,5,12), with more time allotted to the study’s implementation and scored these with a 2.

Table 2.

Quality Assessment With Diverse Studies (QuADS) Assessment

QuADS criteria	Score/Associated criteria statement	Justification
1. Theoretical or conceptual underpinning of the research	3/ Explicit discussion of the theories or concepts that inform the study, with application of the theory or concept evident through the design, materials and outcomes explored	This study is grounded in a conceptual framework developed and published a priori in September 2023
2. Statement of research aims	3/ Explicit and detailed statement of aim/s in the main body of report	The research aim and background rationale are clearly presented in each paper that reports findings
3. Clear description of research setting and target population	3/ Specific description of the research setting and target population of study	In this and other papers reporting findings of this research, the research setting and target population (i.e., municipalities with populations <10,000 in northern Ontario) is clearly articulated. Eligible interview participants are also clearly defined
4. The study design is appropriate to address the stated research aim/s	2/ The study design can address the stated research aim/s but there is a more suitable alternative that could have been used or used in addition	With more time and a larger research team, the study may have been able to recruit additional participants representing more stakeholder groups
5. Appropriate sampling to address the research aims	2/ Evidence of consideration of sample required to address the aim	The sample (number of representatives from each municipality and organization) is described
6. Rationale for choice of data collection tool/s	3/ Detailed explanation of rationale for choice of data collection tool/s	Development of data collection tools (interview scripts and SIM questionnaire) have been clearly described as part of the multi-method approach
7. The format and content of data collection tool is appropriate to address the stated research aim/s	3/ Structure and content of tool/s allow for detailed data to be gathered around all relevant issues required to address the stated research aims	Data collection tools are tailored to the research aim and facilitated responding to research question
8. Description of data collection procedure	3/ Detailed description of each stage of the data collection procedure, including when, where and how data were gathered such that the procedure could be replicated	Details provided in the current paper
9. Recruitment data provided	3/ Complete data allowing for full picture of recruitment outcomes	The full recruitment process is described in the current paper
10. Justification for analytic method selected	3/ Detailed justification for choice of analytic method selected	Rationale for selected analytic method clearly described in this paper and other papers presenting research findings
11. The method of analysis was appropriate to answer the research aim/s	3/ Method of analysis selected is the most suitable approach to attempt answer the research aim/s in detail	Reflexive TA as most appropriate analytic method for positionality of this researcher, conceptual framework, and research aims
12. Evidence that the research stakeholders have been considered in research design or conduct	2/ Evidence of stakeholder input informing the research	Stakeholder input at project initiation and at each stage (i.e., with advisory committee) described
13. Strengths and limitations critically discussed	3/ Thorough discussion of strengths and limitations of all aspects of study including design, methods, data collection tools, sample and analytic approach	Discussion of strengths and limitations present in each paper reporting research findings and incorporated into discussion of current paper

Political rigour, however, is a more recent concept and warrants deeper exploration, again as described by Temper et al. (2019). Accordingly, the remainder of this discussion will focus on this aspect of the study, drawing on the ten principles presented in Table 1 of accessibility, reflexivity, relevance, transparency, care-fullness, respectfulness, relationality, reciprocity, fallibility, and transfomativity/transgression. To support this analysis, we incorporate excerpts from AM’s research journal and reflect upon the theoretical considerations presented above of PNS, systems thinking, pragmatism, and transdisciplinarity.

Accessibility

The value of accessibility suggests that research should be widely understandable and serve as a tool for social learning. Accessibility aligns with our study’s pragmatic design, which emphasizes the uptake and use of findings. Prioritizing accessibility also supports understanding of the systems used and, along with the study’s transdisciplinary approach, enhances its relevance to a broader audience.

A key objective of this research was to ensure its utility to public health, emergency management, and local governance actors. To achieve this, we emphasized not only the preparation of traditional peer-reviewed publications but also integrated knowledge mobilization. This included sharing findings as they emerged, tailoring selective findings for specific audiences, and producing a comprehensive report and a policy brief. Presentations were also offered to research participants, advisors, and supporters. Bazeley (2016) describes reporting on qualitative research as “somewhere to the left of, or sitting on the fence between, art and science” (p.299). This understanding of qualitative writing as both an art and a science underpinned our approach, with a single primary author supported by advisory and supervisory committees.

The timely sharing of findings was facilitated by use of a single researcher rather than a hands-on research team. Although a multiple-researcher approach allows the integration of diverse perspectives, which can add value and depth, this approach can also require additional time investment as the team builds shared understanding, objectives, and develops a cohesive conclusion. Conversely, a transparent single researcher approach facilitates swift decision-making and reporting. This project’s aim to inform policy and practice in a timely manner was further supported by an opportunity to share findings with provincial government representatives mid-way through the reporting phase. In such a scenario, having multiple researchers driving the metaphorical research car could have hindered the responsiveness required for this timely reporting.

The adoption of virtual methods, such as video conferencing for interviews and email for sharing large municipal documents, made this research feasible in rural areas where participants were located hundreds of kilometres apart from one another—and from the researcher (Salmons, 2016). The use of the Structured Interview Matrix facilitated social learning by fostering active listening and deliberation among participants. By ensuring real-time access to the research through multiple formats, we aimed to make it accessible to potential users, and by encouraging open discussion and active engagement in the research, we sought to support meaningful social learning.

Reflexivity

The value of reflexivity, here, refers to the critical examination of the researchers’ practices, assumptions, and power relationships. This is a frequently emphasized value in qualitative research (Braun & Clarke, 2022), and that of critical change agents such as Paulo Freire (Freire, 1970). In qualitative research, self-reflection, or reflexivity, allows a researcher to continuously reflect upon the research process and their influence on it, as these change over time (Holmes, 2020). By being reflexive about one’s positionality, research can more transparently acknowledge the impact of this positionality. In her research journal, AM reflected on her positionality and its impact on her decision-making:

As much as you try to immerse yourself in the data, there’s so much that sometimes it really is where your head is at, what you’ve recently read, that influences the interpretation/decision in the moment....

Thinking about the interviews, now that they’re done: I am definitely positioned within this content, and I embraced that in the interviews to strengthen them, to build rapport with participants and have them appreciate the validity of their experiences.

AM’s research journal indicates active reflection during all stages of coding—acknowledging pre-conceived ideas, often asking “what is behind this?” and thinking back to the context and to the individual who shared the idea—all helped by a deep understanding of transcripts.

This reflection led to acknowledging potential research limitations, including the possibility that AM’s embedded positionality within the research geography and field of public health may have limited participants’ openness. Conversely, this same positionality may have fostered trust, making it a strength as well.

Reflexive practice also shaped the research approach. The Structured Interview Matrix was partly chosen for its potential to reduce power imbalances among participants. Maintaining a research journal provided a space for ongoing reflection which informed decision-making within the implementation of the research.

Relevance

Relevance, as an aspect of political rigour, emphasizes the importance of research being practically useful to people and aligns with the study’s theoretical framework in similar ways to accessibility, discussed above. To build relevance, this research integrated three participatory elements: informal consulting with the field before developing the research project, the use of an advisory committee to provide guidance and validate findings at each stage, and the integration of research participants/key knowledge users in the analysis of findings and development of recommendations.

AM’s journal entries reflect occupation with the study’s relevance, especially in supporting Indigenous wellbeing:

I’m doing the OCAP [ownership, control, access, and possession] Foundations course today and thinking about my research and how to respect OCAP. For example, how does it apply to urban Indigenous populations? Or, how does my research which impacts lands and waters impact First Nations? Or, how does my research approach, including having one First Nations representative, potentially impact or harm local First Nations communities? And the contrast/challenges with research timing are strong. Could I/should I have taken more time with this part?

Unfortunately, limitations related to number of participants and scope of sectors involved in the study may limit the generalizability of the findings. To further consider relevance, we explore the idea of generalizability of the research findings. Lewis et al. (2014) discuss three aspects of generalizability: representational (the extent to which findings can be generalized to the broader population in the case), inferential (whether findings can be generalized to other cases), and theoretical (whether findings suggest broadly generalizable theoretical implications). Because this case study did not have representation from the entire case study area (Northern Ontario), its representational generalizability may be limited to the communities and regions that did specifically take part. Because of the variation among rural places, inferential and theoretical generalizability may best be left to what Lewis et al. (2014) call ‘reader generalisation’, in which the researchers, instead of claiming generalizability of their findings, provide sufficient description of the cases to allow the reader to determine for themselves where findings might apply. We propose that this research may be relevant to rural communities with governance structures or characteristics like those in our study, also consistent with the concept of case-to-case generalization as described by Chenail (2010).

Transparency

Transparency here refers to clarity among all involved in the research as to the research process, structure, and outcome. In research that seeks to build concurrence among actors, foster cross- disciplinary trust, and foster change—particularly when using the less familiar approach of PNS—transparency is essential for building trust and maintaining credibility. The informed consent process helped with this aim, as all participants were provided the same overview of the study, including how their individual contribution fit into the larger project. Furthermore, regular updates were provided to members of the advisory committee, participants, and project supporters, communicating changes in timeline and identifying opportunities to engage with the findings.

Through her reflections, AM considered further how to maintain transparency of process and decision-making, and coherence between the data analysis and broader research question:

I quickly read some more about reflexive thematic analysis in NVivo and was reminded that the codes should be helping me answer my research questions...my research questions weren’t enough to make use of all the data, but there’s more about context that seems relevant. If I think more broadly to the overall project and next phases...all these data are indeed relevant.

I’m going to have to make my conceptual framework very clear.

I’m still struggling between my analysis versus what will be useful for the next phases of the project.

In this study involving a diverse range of actors working across different contexts, we recognized the value of transparency for its role in building trust among research participants and knowledge users (often the same individuals), and in the research findings themselves.

Care-fullness

Care-fullness refers to care for all those involved in and impacted by the research, including researchers. This study retrospectively explored the pandemic response and as a “lessons-learned” type analysis, might have been vulnerable to commenting in a judgemental manner on decisions made or on work that was undertaken, especially given AM’s proximity to the subject. However, AM was diligent and careful in her approach to analysis and reporting, maintaining focus on the development of useful, forward-looking, systems-based recommendations.

For example, in rural communities, “topic” experts can be elusive: local governance structures rarely accommodate depth of knowledge in any one area, meaning these communities often rely on urban-based knowledge. So, while local rural actors are experts in their context, they may not hold expertise in the many domains that affect them. Emergency management is an area that is, by many, addressed “off the side of one’s desk” as one of many other responsibilities. It is not unreasonable, then, to expect gaps between the latest research on the topic and practice, within these settings. Our approach has aimed to demonstrate care to individuals by carefully discussing this gap, where it arose, in relation to system drivers.

Care-fullness as described here is a pragmatic value to apply in a study seeking to produce energizing and action-oriented findings.

Respectfulness

The value of respectfulness speaks to maintaining respect for ways of knowing and various forms of knowledge. In the introduction, we discussed the value of pluralism in complex issues. As a researcher, AM struggled with her ways of knowing and the appreciation that others may not share similar perspectives, while wanting to generate research that was as meaningful to them as to her.

I’m still really struggling to figure out how to harmonize my acknowledgement of a plurality of perspectives with my push that we move to a salutogenic model.

Two things happening concurrently. First is how I explain the phenomenon and acknowledge a multiplicity of perspectives.... As I go about discussing it, I’ll be open to others having different framings/understandings. We all come from different places and we know what we know and, pragmatically speaking, we can get on with the work regardless.

If I use an inductive approach, then I dive in and see what comes out. If I use a deductive approach, then I’m looking for things in relation to the theoretical framework I’ve developed. I kind of want to do both—is that even possible?

She also tried to maintain awareness of how she was reacting and what that might suggest about her hidden assumptions: “Try to notice when I’m having emotional reactions to content or when I feel very sure.”

This value was important to AM’s research as it helped prevent any single perspective from being positioned as superior. It helped to emphasize the inherent value of plural viewpoints and the importance of respecting their co-existence within the complex experience of the COVID-19 pandemic.

Relationality

Relationality helps ensure the research is grounded in its context and is a core element of the PNS approach used in this study. AM was herself grounded as the researcher in the study context. While we consider her positionality as a content and context expert a strength of the study, it may also be a potential limitation, having potentially impacted participants’ willingness to express critical views about public health, or reluctance to speak openly to a colleague.

According to Holmes (2020), there are many advantages to a researcher’s having an “insider position” related to design of methodological tools, increased trust among participants, and ability to more thoroughly interpret responses. There are also disadvantages, however, that this case study had to anticipate and mitigate, and in some cases accept, like a reluctance among participants to disclose some information to a colleague. In fact, AM’s relation to participants varied by community and sector. In all interviews, AM emphasized her northern-ness to demonstrate commitment to and understanding of the context in which participants work. She also was upfront, both in interviews and within the informed consent process, about her connection to one local public health agency, reassuring participants that their relationship with this organization would not be impacted at all by their participation (or non-participation) in the study.

Reciprocity

Reciprocity implies an exchange for mutual benefit, and in the context of political rigour, Temper et al. (2019) suggest this can be achieved, in part, through co-design. While this study included some consultation during its design, co-design was not a central feature; instead, reciprocity was pursued through a strong emphasis on utility. This study aimed to be of specific value to research participants and the communities in which they live, by functioning as a post-pandemic reflection of use to participating and supporting organizations but for which they did not have capacity to undergo independently. In her journal, AM reflected:

Today in the workshop one person said, “this is the debrief we needed, but never got to have,” and others agreed. It felt great to feel that this was useful to them, addressing needs they had identified but did not have capacity to address.

Since the report’s publication, AM has sought opportunities to reciprocate with the communities who participated by speaking with others in the region to answer questions, brainstorming ways to use the findings in their own work addressing systemic issues in rural Northern Ontario. The research generated was seen as valuable and useful to participants, as well as to the researchers.

Fallibility

Fallibility acknowledges that research can fail and highlights the value of such failure, particularly as a source of learning. Feeling very much that she was learning as she was doing the research, AM reflected often on its potential fallibility:

This all feels so inefficient because I’m not experienced enough.

Trying to organize recommendations by sector/actor doesn’t work because the influences are too messy. Nobody “owns” any of these.

Feels important to go slowly, not condense things too quickly. With a complex systems perspective, answers to questions aren’t easily bounded or identified.

She also reflected often through the process of trying new things, taking care to honour the data and ensure she was applying the analytical approach appropriately. The process of RTA involved significant learning and back-and-forth, which was also described as beneficial within the process. For example, flipping between re-reading transcripts and coding;

Today I started clustering codes using paper slips.... Immediately it feels different. It’s more open, I’m not stuck the way I was using excel or NVIVO. I'm making new clusters that weren’t occurring to me using those tools, and starting to consider possible themes that hadn’t occurred to me before.

I’m still reading through the interviews, but not worrying about doing it all in one go before trying to cluster codes. Being able to flip from the data to the codes seems to help me come up with new ideas and process the information more easily.

I really see the reality of my actively creating themes—they don’t just emerge. I’m choosing where to place codes, how to link them to ideas that are already forming, and in doing so, not linking them to other ideas.

For codes, I’m often asking myself, “what’s behind this?” and thinking back to the context and the individual. Makes good knowledge of the transcripts valuable.

Presenting at conferences has helped me explore, clarify and then be explicit about my mindset and theoretical foundations…applying learning as I go.

I’m going to go back over the codes one more time, especially for the first couple interviews, because I think my style of coding evolved since the beginning.

Already discussed somewhat above, the single-researcher approach used in this study also relates to fallibility. Risks were mitigated by the researcher’s receiving advice from a community-based advisory committee and guidance from doctoral supervisory committee. This allowed her to avoid some of the tensions and time-consuming investments in consensus on study objectives and analysis that are required of a multi-researcher study. Bazeley (2016) points out that multiple-research teams, with varied disciplinary training, underlying individual philosophies, and areas of competency, can generate too much plurality against which to then sufficiently make sense of the research, working against the aim of generating integrated findings in a multi-method study. Interdisciplinary fields have probability of researchers coming from diverse disciplinary backgrounds with disparate trainings and understandings of research practices and associated strategies. The work to reconcile these differences in order to build consensus on data analysis and reporting would be significant.

PNS and pragmatic research can be unpredictable and require frequent adaptation, making fallibility an important value to embrace. Early career researchers may especially benefit from doing so, to support them in accepting, adapting to, and learning from experiences where external factors affect timelines or methods, or when insight into findings emerges over time—AM, for example, has continued to develop new insights into her doctoral work since the publication of her thesis.

Transformativity/Transgression

The final value proposed by Temper et al. for political rigour is transformativity/transgression, which they describe as a means to disrupting oppressive power relations. This research sought to support transformativity and transgression in the system being addressed and for the researcher, which we attempted by maintaining a critical lens throughout the research. Doing so during the recommendations phase was particularly challenging, as AM worked to honour the recommendations made by participants, as context experts, with her own transformative goals. She achieved this by reporting on the initial, participant-developed recommendations and her own final research recommendations separately, after much reflection:

How do I balance the recommendations provided from the field, with my acknowledgement of the power and knowledge limitations that also affect them?

This balance was also helped by a systems thinking approach, called Critical System Heuristics (CSH), that can be used to explore values and assumptions behind a complex phenomenon yet rarely brought to the fore. CSH was developed by Werner Ulrich beginning in the 1970s; he calls these often-hidden values and assumptions “boundary judgements” (Ulrich & Reynolds, 2020). Ulrich encourages researchers to become aware and critical of boundary judgements and offers a series of questions to aid in exploring both ideal and actual sources of motivation, control, knowledge, and legitimation for the phenomenon in question. Whether a researcher uses these questions or not, the general practice of clarifying the bounds of the system one is exploring and acknowledging the corresponding variability of values and facts is the conceptual core of this approach. AM used this approach of exploring boundary judgements that may be hidden in participant recommendations to inform the final set of research recommendations.

This study included critical perspectives of the professional “expert” roles deployed within the COVID-19 response within rural communities, both in general by questioning the nature of one’s knowledge, and in terms of accountabilities, i.e., where one is employed and who they speak for.

Finally, our study analysis infused aspects of critical discourse analysis (CDA) in interpreting interview data. CDA has been well articulated by thinkers such as Fairclough and Wodak who point to cultural and economic power relations’ influence on language and further inspired by the thinking of Michel Foucault (Foucault, 1971; Wodak & Meyer, 2009). For Foucault, power relations and social structures are maintained by power dispersed among societal institutions and networks, and these both mainstream discourses and silence counter discourses, many of which are so entrenched that we no longer notice them. AM worked to bring these ideas to her analysis.

Although transformation is central to disrupting the power struggles inherent in the wicked problems addressed by this research, and its acknowledgement in the project’s conceptual framework, it did not emerge explicitly in the final research set of recommendations. Nevertheless, transformation serves as a means of enacting those very recommendations.

Conclusion

Through this paper, we have aimed to demonstrate that PNS as a paradigm makes it possible to research complex issues while embracing a plurality of perspectives, and producing timely, actionable findings. Furthermore, such research may be implemented imperfectly, encounter unanticipated changes, yet still achieve meaningful findings. By applying two sets of quality criteria—for scientific and political rigour—we hope to bring attention to the ability of research to address complex societal issues, and demonstrate the feasibility of such multi-phased, iterative research being done with integrity. We hope that we have drawn attention to the value of this research, and helped to make it easier and more common for people to participate in and use research to address complex social problems.

Research to foster understanding and action on complex issues related to the global poly-crisis is accompanied by tensions: among conflicting ideologies, associated with power imbalances, epistemologies, and worldviews. Applying PNS, seeking scientific rigour, and making political values explicit can make such research possible. Where scientific rigour is commonly taught to and was relatively straightforward to assess, the values underpinning political rigour prompted deeper post-hoc reflection. In retrospect, the ten values explored here resonated with AM’s approach and were, to large extent, intuitively integrated. We believe these values are useful to others working on wicked problems and suggest that researchers consider, or co-develop, a tailored set of values at the outset, as Temper et al. (2019) recommend, to help cultivate political rigour.

The reflexive approach described here can support early career researchers in learning through practice. It encourages critical examination of assumptions and openness to a wider range of findings, also beneficial to researchers at all stages. Wicked problems are, by nature, elusive and complex, involving many actors and divergent perspectives, which can be challenging to navigate. Active reflection along with attention to other values explored here can serve as a compass.

As communities face more frequent disruptions such as extreme weather and political or economic instability, the need grows for research that enables nuanced and contextualized understandings and supports local action for health and wellbeing. The values of political rigour, and by extension the case study described above, align with the commitments proposed in the RADIX Disaster Studies Manifesto (RADIX, n.d.) which emphasizes disaster related research that is locally-driven, generative, ethically conducted and widely accessible.

Our study demonstrates that timely, meaningful research to address complex problems can be done in participatory ways, and by people embedded in the field. In rural places especially, people wear many hats and contribute how they can, which may involve being both practitioner and academic; making it easier for rural practitioners to bridge between academy and practice can contribute to increased engagement in addressing rural issues in particular.

Supplemental Material

Supplemental Material - Post-Normal Science, Post-COVID: Multi-Method Approaches to Actionable Research

Supplemental Material for Post-Normal Science, Post-COVID: Multi-Method Approaches to Actionable Research by Amanda Mongeon, Leith Deacon, Kate Mulligan, Rana Telfah in International Journal of Qualitative Methods.