Abstract
Urban planners are often tasked with locating, assessing, and integrating multilevel policy, regulatory, administrative, and governance information. Through a recent systematic literature review (SLR) of Canadian municipal interventions targeting women’s equity, this paper answers the following research question: how can planners use an SLR methodology to increase the rigor and transparency of precedent research within practice? This article highlights the contributions and limitations of applying an SLR methodology to research within practice and demonstrates how planners can modify and apply this approach to standardize the collection and assessment of precedent to contribute to informed planning decisions.
Introduction
Urban planning considers numerous socio-spatial elements, including new and existing land use, transit and mobility, recreation, housing, and economic development. Multilevel planning interventions include both overarching (e.g., Official and Strategic Plans) and subsidiary plans (e.g., housing policies, neighborhood plans, transit strategies) administered by planning departments, either alone or in partnership with other municipal and regional authorities. Despite the complexity of planning practice, there is little substantive, methodological guidance available to practitioners attempting to investigate the potential scope, mechanism, and magnitude of multilevel planning options (Davoudi 2006; Faludi and Waterhout 2006; Head 2010). Systematizing the planning research process allows planners to effectively consider a wider variety of planning interventions, supporting evidence-based practice in a way that “can therefore be integrated with collaborative and participatory approaches that place a greater emphasis on jointly defining problems and consensus-based solutions” (Krizek, Forysth, and Slotterback 2009, 460). In this paper, we explore how planners can modify and apply an systematic literature review (SLR) methodology—frequently used in disciplines such as public health, medicine, business, and education—to increase the rigor and transparency of multilevel policy and precedent research.
This process-building, methodological paper, provides a recent SLR of municipal planning initiatives and policies relating to women’s equity as an exemplar. The purpose of this study is to explore the potential of applying a modified SLR methodology within planning practice to increase the rigor and transparency of precedent research relating to complex socio-spatial issues. We anticipate that this methodology will allow practitioners to capture and assess evidence from a wider variety of sources—including policy, regulatory, administrative, and governance interventions—contributing to a better understanding of the intentions and implications of a wider range of planning initiatives and supporting more informed decision-making. The study outlines the search results, quality assessment considerations, and synthesis challenges that can be expected following the application of a traditional SLR methodology to precedent research. It demonstrates that, while modifications are required to adapt an academic SLR methodology to practice, this approach offers a means of making professional research more comprehensive.
We establish the need to increase the methodological transparency and rigor of conventional policy research within planning practice. As this paper focuses on reconceptualizing an academic methodology as a professional process, we provide a detailed discussion of the methods used, the results of the initial and revised search strategies, and the modifications necessary for the effective use of this methodological approach in practice. We describe the approach to data collection and quality assessment and provide recommendations for synthesizing narrative data to allow for future alignment with census and demographic data. The paper concludes with a discussion of the strengths and limitations of applying this methodology in planning practice, and suggests directions for future evidence-based practice and associated research.
Context and Rationale
SLRs can be defined as a type of research synthesis intended to collect, assess, and integrate existing evidence relevant to a specific research question, either as justification for a policy decision or to synthesize existing knowledge into new theory (Oliver, Dickson, and Bangpan 2015; Pawson et al. 2005). SLRs are not a traditional part of urban planning scholarship and training. As a result, planners often lack knowledge of how this particular methodology would enable them to conduct rigorous and replicable policy reviews within practice and support more reliable decision-making (Davoudi 2006; Krizek, Forysth, and Slotterback 2009; Xiao and Watson 2019).
Planners frequently wrestle with complex, systemic issues regarding interconnected matters of safety, sustainability, public health, and equity—referred to in the planning literature as “wicked problems”—throughout the policymaking process (Balassiano 2011; Grant-Smith and Osborne 2016). For instance, a policy directed toward housing affordability could affect individuals differently depending on their income, physical ability, age, race, gender, or family structure. As a result, planners are often faced with designing and justifying operational solutions to address the needs of a diverse population based on a presumed theory of change that links intervention to outcome along a theoretical causal pathway. However, existing academic systematic reviews relating to the urban environment, including examples related to safety (Alderton et al. 2020; Lorenc et al. 2013), sustainability (Luederitz, Lang, and Von Wehrden 2013), and public health (Ige et al. 2019; Pineo et al. 2020; Smith et al. 2017), overwhelmingly consider the urban environment as a setting for, rather than a key contributor to, the phenomena being studied. Also, while such academic studies may provide evidence of trends at the community level, the inherent epistemological paradox of applying research from multiple disciplines to a complex systemic issue means that social outcomes are often difficult or impossible to attribute to municipal initiatives. Furthermore, these reviews often overlook the organization-level information necessary to design, justify, and implement effective multilevel policy interventions in practice (Mavrot, Hadorn, and Sager 2019; Parida and Brown 2018). While various disciplines, including public health, medicine, business, and education have developed guidelines on how to conduct SLRs, urban planning has not yet established such guidance for either academic researchers or practitioners (Faludi and Waterhout 2006; Krizek, Forysth, and Slotterback 2009; Lavis 2009; Xiao and Watson 2019). Furthermore, the methodological guidance that does exist refers almost exclusively to the review of academic research and, while guidelines relating to SLRs of gray literature are available (Benzies et al. 2006; Hopewell, Clarke and Mallett 2005; Mahood, Van Eerd, and Irvin 2014), they do not offer substantive, procedural direction for planners reviewing policy, regulatory, administrative, or governance interventions within practice. As a result, there is a need for an applied research methodology that will enable practitioners to evaluate policy and precedent research within an operational context to synthesize information that is implementable or enforceable in practice.
For planners, recommending policy actions without sufficient grounding in practical, implementation-based knowledge poses two key challenges. First, practitioners must draw conclusions about the anticipated scope and magnitude of a policy’s impact based on their understanding of the intent and historical implications of similar policies (Davoudi 2006; Lavis 2009). Second, they must apply evidence from other locations to community data, while taking into consideration operational requirements, departmental jurisdiction, enforcement capabilities, and the local political climate (Krizek, Forysth, and Slotterback 2009; Oliver, Dickson, and Bangpan 2015). Due to the unpredictable and political nature of urban development, a planning intervention that was effective in one municipality may have unexpected results in another. Using a modified SLR methodology within practice would enable planners to broaden the scope of precedent research to include a wider range of relevant interventions, thereby providing a more complete view of available approaches. This would allow practitioners to avoid cherry picking, while better understanding the intentions, implications, and mechanisms of different precedents for use in the decision-making process.
Including an analysis of planning initiatives as part of an evidence-based decision-making process recognizes that operational considerations—including staff capacity, governance structure, political support, funding, or enforcement capacity—are an integral part of the design, justification, and implementation of a municipal intervention. Due to the collaborative and political nature of urban development, the impact of a planning intervention is not always aligned with its intention; therefore, the first step for practitioners is often to gather information about the types of interventions other jurisdictions have implemented to understand the technical, administrative, and financial requirements of a variety of approaches. However, in the absence of a systematic methodology for identifying, assessing, and comparing municipal interventions, practitioners often focus precedent research on a few prominent examples (Auld et al. 2014). Furthermore, due to a lack of peer-reviewed literature about the practice-based aspects of municipal initiatives, it is difficult to develop a nuanced understanding of the scope and impact of diverse planning options in relation to the settings, conditions, and context within which they take place (Moghadam and Rafieian 2019; Parida and Brown 2018).
This paper demonstrates that a modified SLR methodology can provide planners with a means of understanding and comparing diverse, multilevel interventions on a single topic, including an assessment of the source of authority (e.g., government-led, industry-led, or hybrid), type of intervention (e.g., policy, regulation, guideline, governance approach, or partnership), target group (e.g., citizens, industry, or administration), and operational activities (e.g., planning, implementation, or evaluation) (Auld et al. 2014; Head 2010; Oliver, Dickson, and Bangpan 2015). We argue that a modified SLR methodology supports practitioners as they conduct rigorous, replicable, and transparent policy and precedent research, while developing operational evidence about the design and administration of multilevel planning interventions.
Method
Urban planning practice includes and informs a diversity of activities, including the development and administration of policies and regulations, shared governance approaches through partnerships and advisory committees to Council, and public education and engagement activities. Effective precedent research requires analysis of all planning instruments enacted within a strategic initiative (e.g., policies, regulations, plans, strategies, frameworks, governance approaches, partnerships), the implementation context, and the population or jurisdictional target group (Mavrot, Hadorn, and Sager 2019). For brevity, these activities will be referred to as “planning practice” and “multilevel interventions,” respectively, in this article. In this section, we describe the methods used to conduct a systematic review of multilevel policy interventions, including capturing, assessing, and synthesizing diverse types of municipal documentation.
Research Protocol
One of the primary characteristics of a SLR is the creation of an a priori research protocol documenting the research question, rationale, and methods to be used throughout the review. The protocol establishes the rigor, transparency, and trustworthiness of a study’s methods by detailing how information will be identified, screened, assessed, and integrated. In many cases, systematic review protocols will be published before research begins to establish ownership of the review, reduce the risk of duplication, and provide greater transparency when evaluating or updating an existing review. Numerous protocol registries exist, most with a focus on health-related research, including the Campbell Collaboration, Cochrane Collaboration, PROSPERO, and the Joanna Briggs Institute (JBI). No designated registry currently exists for systematic reviews of urban policy research; however, a protocol could be submitted to a City Council as part of a staff recommendation related to a particular municipal objective. As Council records are part of the public domain, this approach could help establish transparency and accountability, which would facilitate the use of the evidence gathered to inform the decision-making process.
The protocol used in the exemplar was designed based on the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) standardized guidelines (PRISMA 2020). PRISMA offers a twenty-seven-item checklist that is intended to guide the preparation and reporting of a robust systematic review. Because this study includes intervention approaches aimed at addressing a specific social objective, the research protocol also incorporates a Population, Intervention, Comparison, Outcome (PICO) Framework, commonly used in evidence-based practice to improve the conceptual clarity of a research question. Combining these approaches enabled the development of a protocol aimed at conducting Canadian municipal research that meets the requirements for methodological rigor consistent with the JBI guidance for systematic reviews and research synthesis (The JBI 2017). Due to the lack of precedent for research protocols that are applicable to planning practice, we have included the protocol developed for this study in Table 1.
Research Protocol Developed for Exemplar Study.
Note: PICO = Population, Intervention, Comparison, Outcome.
Document Identification and Screening
Research protocols typically identify the specific academic databases that will be searched in a systematic review; however, because Canada does not have a comprehensive repository of municipal policies and interventions, municipal websites were chosen as a proxy for academic databases. This section describes the process used for document identification and screening based on recommended SLR methods for gray literature. Gray literature is defined as open-source information generally produced by organizations for which publishing is not a primary activity, including all levels of government, business, and industry (Benzies et al. 2006; Hopewell, Clarke, and Mallett 2005; Mahood, Van Eerd, and Irvin 2014). This section also outlines the modifications necessary to capture the full diversity of municipal interventions, implementation contexts, and additional variables affecting outcomes.
Given typical methods of precedent research within planning practice, the exemplar focused on collecting and analyzing publicly available documentation of municipal initiatives. Two rounds of document identification and screening were completed: the first round was based on the initial research protocol and sought documentation of multilevel policy and regulatory interventions. The second round expanded the allowable range of document types and focused on locating evidence of non-policy interventions, including alternative governance approaches, committees to Council, partnerships, gender budgeting, and facility programming. Both rounds of document identification investigated the same forty-eight Canadian cities, including all thirteen provincial and territorial capital cities, the federal capital, and all Census Metropolitan Areas (CMAs) (Statistics Canada 2018). The study investigated eight cities in British Columbia, four in Alberta, two in Saskatchewan, one in Manitoba, nineteen in Ontario, five in Quebec, three in New Brunswick, and one each in Nova Scotia, Newfoundland, Prince Edward Island, Yukon, Northwest Territories, and Nunavut. Using the Statistics Canada CMA requirements as criteria for city selection helped ensure geographic diversity, while maintaining transparency and ensuring consistency in the size and complexity of the study areas.
First round
Prior to the start of our review, peer-reviewed research was used to identify key areas where women’s equity is recognized to intersect with the built environment. Gray literature from prominent international gender organizations was then searched to establish priority areas where municipalities were already attempting to address women’s equity through policy and related interventions, including access to services, transit and mobility, safety and freedom from violence, health and hygiene, and security of tenure (City of Vienna 2011; Council of Europe 2016; Inter-American Development Bank 2014; Organisation for Economic Co-operation and Development 2018; UN Habitat & Global Utmaning 2021; World Bank 2020). The results of these searches were used to inform the development of the research protocol, which outlined strategies for document identification and data extraction. The first round of document identification sought to capture any policies, plans, and strategies that were explicitly focused on advancing women’s equity. To ensure that our study captured all of the aforementioned priority areas affecting women’s equity, we also searched for municipal interventions related to accessibility, transportation, safety, community well-being, and housing.
Given the inconsistent nature of Internet searches, we determined that using Google and similar search engines to seek municipal documentation was unlikely to be exhaustive; therefore, efforts were focused on conducting structured searches of each municipal website using the internal site query function. To support transparency and replicability, we followed the same process of searching, identifying, and screening records for every municipality. Multiple inquiries were undertaken on each municipal website using the following search terms: “women,” “gender,” “gender mainstreaming,” “official community plan,” “housing,” “safety,” “transportation,” “well-being,” and “accessibility.” Searches were limited to English-language plans, policies, and regulations published between 1995—the year the Beijing Declaration was adopted by the United Nations (UN Women 1995) and Canada adopted its first Federal Plan for Gender Equality (Status of Women Canada 1995)—and 2021 when the review took place. The search parameters resulted in the initial identification of 257 plans, policies, and regulations. Before downloading, each document underwent a full-text word search for the keywords: “women,” “girls,” “gender,” and “equity.” Of the 257 documents examined, 175 included one or more incidences of these terms and were downloaded for content screening. Any documents that did not include one or more references to the keywords were excluded from the study.
All downloaded documents underwent content screening before progressing to an in-depth review. In a typical systematic review, screening would be conducted based on the title and abstract of a document. Policy and regulatory documents rarely include an abstract; therefore, screening consisted of a detailed word search within the full-text of each document, including an examination of the contextual use of keywords (Gaber and Gaber 2005). For example, the research team found many instances where “equity” and “gender equity” appeared as high-level aspirations—often in an introduction or executive summary—but had no associated objectives, policy statements, or indicators within the text. Similarly, the term “women” was often used to describe the local gender distribution, regardless of whether the document specifically considered women’s needs in its goals, objectives, or indicators. Through content screening, 124 documents were excluded, including 82 where women were not identified as a target population, 26 where “equity” referred to a different equity-deserving group, and 16 documents that were not relevant to this study. The initial search revealed many results that did not meet the protocol requirements for intervention type, indicating that many cities have gender equity work underway that is not captured in policy or regulatory documents.
Second round
The first round of document identification resulted in the collection of fifty-one documents for detailed analysis, including only seven directly focused on women’s equity. It was clear from the first round that the search strategy proposed in the research protocol was too narrow and did not capture the full range of municipal interventions aimed at advancing women’s equity. As a result, a second round of data collection was proposed and the research protocol was modified to capture a more diverse range of interventions and records, including guidelines and toolkits; department structures and staff allocations; advisory committees to Council and alternative governance approaches; municipal strategic plans; partnerships; needs assessments; and gender budgets. Council minutes were included in cases where gender-based budgets or interventions were approved but lacked documentation elsewhere.
The second round of document collection was again conducted using the query function on each municipal website and limited to English-language documents published between 1995 and 2021. To broaden the search results, we first conducted a word search for “gender,” “women,” and “equity” using each municipality’s query function. While the first round focused only on plans, policies, and regulations, the researchers now investigated all query hits, including Council minutes, evaluations, consultant reports, public notices, and staff training materials. Next, the research lead screened each municipality’s public listing of policies, plans, and regulations for any equity-driven interventions and searched each municipality’s corporate strategic plan for objectives, priorities, and expenditures related to women’s equity.
To better understand the range of administrative and governance-based approaches, we reviewed each municipality’s department listing and organizational chart, searching for a dedicated Women’s Office; Equity, Diversity, and Inclusion Office; and/or dedicated staff allocations. Next, we searched each municipality’s list of advisory committees to Council for documentation of Women’s Advisory Committees, or Equity, Diversity, and Inclusion Committees. Finally, publicly available Council minutes and reports were scanned for any action items related to women’s equity, gender mainstreaming, gender budgeting, or other targeted initiatives that lacked documentation elsewhere.
As in the first round of document identification, all search results underwent a full-text search for two or more occurrences of the words “women,” “girls,” “gender,” and “equity” before downloading. In total, an additional 147 documents were downloaded for content screening. We again conducted a full-text, contextual search for the words “women,” “girls,” “gender,” and “equity.” This resulted in 107 of the 147 documents being excluded, including 41 duplicates from the first round of document collection, 43 documents where women were not identified as a target population, 6 human resource policies focused solely on hiring and/or internal complaint procedures, and 17 documents which were not relevant to this study. As shown in Figure 1, the two rounds of document identification and screening resulted in the inclusion of ninety-one municipal records for full-text analysis and quality assessment at the end of document review.
Flowchart summarizing the SLR process for municipal research, informed by PRISMA guidelines.
Data Extraction
Data extraction and quality assessment were completed over a ten-month period. Using MAXQDA software, each document was categorized by type, topic, authority, location, and date. The research lead created a deductive coding matrix based on previous research, and a detailed, full-text review of each document was completed. Data were extracted and coded across eight categories of interventions: municipal directives, governance, service provision, data collection, gender budgeting, partnerships, public engagement, and leadership. Municipal initiatives were further divided into four operational sub-categories: plans and policies, regulations, strategies and frameworks, and guidelines and toolkits. Data were also extracted and coded relating to specific gender-based indicators, including but not limited to childcare, mobility, security of tenure, safety and freedom from violence, recreation, and social participation. Finally, data concerning intersectional identity characteristics were collected relating to age, income, ability, family structure, orientation, gender identity, race, religion, and participation in sex work. The deductive coding approach allowed us to identify the diversity of approaches taken to address a single objective, while isolating any specific objectives or intersectional characteristics not addressed through the multilevel interventions.
Snowball references
Snowball references were collected during document review to better understand the academic and industry references making a core contribution to the development of municipal interventions. A double-sided snowball approach is a systematic, retrospective approach to identifying documents that consist of compiling and analyzing all bibliographic entries from study documents to identify “landmark” papers (Contandriopoulos et al. 2010). Every document was searched for bibliographic references, which were extracted and recorded alphabetically in an Excel table. Once the detailed review was complete and all snowball references were collected, duplicate references were grouped and tallied. Due to the tremendous heterogeneity of document types, target groups, and geographical contexts of the references collected, a threshold of two occurrences was set, and any references appearing two or more times in different documents were downloaded for further review. Through the double-sided snowball process, 485 bibliographic references were identified: 215 were referenced only once, 201 were in French, 51 were specific to a single city or region (e.g., references to past Council reports, demographic reports, or reports from local organizations), and 11 were documents already included in the review. This process resulted in a large number of exclusions (n = 478) to obtain a limited number of inclusions (n = 7); therefore, the effort required to collect and sort the snowball references may have been better directed toward other research activities, such as articulating the theory of change underlying each intervention approach. Future reviews may consider omitting the snowball approach in favor of including landmark industry publications and guidelines identified through previous research or in consultation with an alternative panel of expert practitioners.
Quality Assessment
The heterogeneity of records included in the review compelled the creation of a study-specific quality assessment checklist. The checklist was developed based on guidance for critically appraising qualitative research (Lockwood, Munn, and Porritt 2015), mixed-methods research (Hong et al. 2018), and gray literature (Tyndall 2010). Quality assessment focused primarily on relevance, salience, and accountability. As these factors are heavily context-dependent, legitimacy was assessed based on the source of the information (e.g., municipal staff, Council, consultant, or partner organization); whether the document clearly defines objectives aimed at influencing professional discourse or practice (relevance); whether the document can influence the physical configuration or socio-spatial management of the urban environment (salience); and whether the document contains a monitoring and evaluation requirement (accountability).
The quality assessment checklist was completed for every document, and at the end of each checklist, documents were recorded as “included,” “excluded,” or “requiring further information.” Through the quality assessment process, thirty full-text documents were excluded, and four documents were flagged as requiring further information, three of which were later included in the study. The quality assessment process allowed the research team to develop a coherent strategy to assess the reviewed documents as records of diverse municipal interventions, regardless of the heterogeneous document types, formats, audience, and scope.
Data Synthesis
The purpose of data synthesis was to identify the specific aspects of women’s equity most commonly targeted through municipal planning interventions and develop a series of conditional indicator statements associated with a positive outcome for each objective (Ebrahim 2019; Pawson et al. 2005). We combined two complementary data synthesis techniques—meta-narrative mapping and realist review—to theoretically align data synthesis within a pragmatic knowledge exchange process. Meta-narrative mapping supports pragmatic knowledge exchange by grouping interventions into practice-driven categories and then evaluating their contributions against evidence-based objective and indicator data (Contandriopoulos et al. 2010; Greenhalgh 2004; Timulak 2014). Realist review is a central approach to policy evaluation and aims to “unpack the mechanism of how complex programmes work (or why they fail) in particular contexts and settings” (Pawson et al. 2005, S1:21). Used together, these techniques enabled us to assess convergences and divergences in the causal pathways and outcomes of diverse intervention types in relation to a specific set of objectives.
To compare and evaluate multilevel intervention approaches, it is necessary to first identify key objectives that are addressed through each initiative and align these with qualitative and quantitative indicators of success. We used a convergent segregated meta-narrative mapping approach to synthesize the predominantly narrative municipal data into theoretically driven objectives and indicators (Gaber and Gaber 2005; Greenhalgh 2004; Xiao and Watson 2019). Qualitative objective and indicator data were recorded verbatim in an Excel pivot table, while quantitative indicator data were codified into textual descriptions in accordance with guidance from the JBI (Lizarondo et al. 2020).
Figure 2 provides a simplified example of a meta-narrative mapping approach in relation to the following priority objective: “By 2025, women’s sense of safety will be increased by at least 10 per cent” (City of Vancouver 2018, 20). This objective is divided into several sub-objectives including, but not limited to, decreasing gender-based violence in public spaces and increasing services for women experiencing marginalization. Each of these sub-objectives can then be divided into qualitative indicators, such as women’s self-reported feelings of safety, and qualitized indicators, including incidences of sexual assault and victimization, the number of women experiencing homelessness, and the number of women engaging in sex work. Qualitizing statistical data allows for integration with qualitative data, while maintaining transparency and decreasing the risk of error associated with assigning numerical values to qualitative data. This approach will allow the research team to conduct linear regressions on any longitudinal data in future phases of the study, such as changes in victimization rates or the number of women experiencing homelessness, to examine associations and effect sizes between variables.
Convergent segregated meta-narrative indicator mapping.
To preserve the integrity of the data, qualitative and qualitized objectives and indicators were first recorded as independent data points in an Excel pivot table, resulting in the collection of 430 policy objectives and 151 indicators. Indicators and objectives that were outside the jurisdiction of an urban planning department were removed from the synthesis, and the remainder were assembled into an impact matrix according to the aforementioned priority areas for gender-based planning (i.e., access to services; transit and mobility; safety and freedom from violence; health and hygiene; and security of tenure). Mapping the meta-narrative objective and indicator data against the diversity of intervention approaches allowed us to create a series of conditional indicator statements, which may be used to inform policy decision-making in the future. Conditional statements often take the form “‘If A, then B’ or ‘In the case of C, D is unlikely to work’” (Pawson et al. 2005, S1:24) and may be useful for planners evaluating a variety of intervention options. The impact matrix allowed the research team to isolate the objectives and indicators associated with a particular priority area and then cross-reference these against each intervention approach. This allowed us to identify the specific instruments most commonly used to address each objective and develop a theory of change for each category of intervention. This impact matrix could be expanded and refined in the future to better understand how a particular intervention may affect populations with diverse, intersectional identity characteristics.
Discussion
SLRs are commonly used to establish a foundation of evidence to inform a decision-making process. Because urban planning interventions have significant social, political, and economic implications, practice-based precedent research would benefit from upholding similar standards of rigor, salience, and transparency to academic reviews. Applying a modified systematic review methodology in an operational context revealed three key benefits to practitioners: (1) broadened scope of included documents, (2) identification of (mis)alignment between interventions, and (3) development of a quality assessment strategy for diverse interventions.
Broadened Scope of Included Documents
Using a systematic approach to identifying, assessing, and synthesizing municipal records permitted the inclusion of documents from a wider mix of intervention types than a conventional precedent analysis allows. While minor modifications to the protocol were necessary between round 1 and round 2 to fully capture the diverse range of intervention approaches taken by various jurisdictions, an SLR approach provided the structure necessary to methodically investigate municipal interventions as a multilevel system, while ensuring geographic diversity relating to the size and complexity of precedent cases. In the absence of the SLR structure, precedent selection may be biased toward one intervention type or toward a limited number of well-known case studies.
Identification of (Mis)Alignment between Interventions
Using a modified SLR approach gives planners the ability to readily identify where distinct levels of policy may not be aligned. For example, if a municipality adopts an overarching women’s equity policy that identifies security of tenure as a primary objective, but the City’s housing policy does not mention women as a target population, then the City may face issues with enforcement, and women’s housing needs may remain unmet. Similar to the plan scorecard approach used by Berke et al. (2021), using a modified SLR methodology allowed the research team to identify the degree to which multilevel interventions (e.g., regulations, plans, policies, programs, governance structures, budgets) were coordinated with respect to a particular objective. This approach could support practitioners in collecting, analyzing, and integrating appropriate precedents from other jurisdictions as evidence when designing, justifying, and implementing complex multilevel interventions.
Development of a Quality Assessment Strategy for Diverse Interventions
Quality assessment of municipal interventions differed both in process and content from quality assessments of academic literature. While academic studies provide detailed descriptions of the methodology used in gathering and evaluating evidence, municipal documents focus on objectives and outcomes, often omitting discussion of the process by which conclusions were reached. To address this difference and evaluate interventions against a common set of quality criteria, the research team developed a custom quality assessment checklist, which enabled us to consider the source of authority, level of intervention, target group, and stage of activity for a variety of municipal records. By linking the use of a quality appraisal checklist to an articulated theory of change, practitioners may strengthen the evidence base for decision-making in cases when there is uncertainty about the strengths and limitations of multiple intervention options in relation to a particular policy objective.
Modifications for Practice
The research team recognizes that conducting a traditional SLR within planning practice may not be practical due to time and budget constraints. We, therefore, provide the following recommendations to support planners in scaling and modifying an SLR methodology for use in practice:
● Clearly define the scope of the review, including the population of interest, intervention type, key objectives, and indicators of success.
● Avoid cherry picking municipal precedent by defining geographic parameters for inclusion, including population size, capital city status, governance structure, and/or jurisdiction.
● Use a consistent and transparent approach to searching and identifying relevant intervention approaches, including keyword searches, department directories, Council minute and budget screening, and Committee listings.
● Create and apply a standardized quality assessment checklist, which includes the source of authority, level of intervention, target audience, and stage of activity.
● Use a systematic approach to synthesize and report findings. Where possible, summarize the mechanism or factors necessary for success.
By defining specific parameters for screening municipalities and interventions into a precedent study, and by standardizing how information is collected, assessed, and analyzed, practitioners can effectively manage the time required to undertake an SLR methodology within practice. We argue that the benefits of increasing the rigor and transparency of research findings by using a modified SLR methodology outweigh any additional time required.
Limitations
While a modified SLR methodology allows practitioners to capture, assess, and integrate evidence from a variety of municipal records, this approach is not without limitations. As there is no centralized database of municipal policies in Canada, it can be challenging to efficiently search and identify documentation relating to municipal programs and interventions. Document identification requires in-depth hand-searching and, while it is easier to locate policy documents than it is to find budget, governance, or partnership information, successful searches often rely on a municipal department’s ability to effectively disseminate information to the public. Furthermore, the absence of bibliographic meta-data within the documents themselves can make searching more difficult and decrease the potential for study replicability. We note that these challenges are related less to the search methods used than to the indexing and search functionality of the municipal websites themselves. For this reason, it is important that the exact search terms and process are documented, including a summary of documents screened out of the study, so that the reader can judge the validity of the study’s conclusions based on how the included records were identified, screened, and assessed.
In addition, as municipal websites change over time, replicability may not be possible when updating systematic reviews of municipal records. The research team noted several instances where documentation of programs and policies on municipal websites had been added or removed between the first and second rounds of document collection. Precedent research frequently relies on publicly available information; therefore, future study designs may consider adding the ease of locating public documentation to the quality assessment criteria. This concern could also be addressed in future studies by introducing an expert panel of planning practitioners, using the Delphi Method, to identify landmark precedents and advise the research team on alternate approaches that merit inclusion.
The greatest challenge to using an SLR methodology, however, may lie in data synthesis. Municipal interventions rely heavily on context, yet locations that are in close, geographic proximity may differ significantly from a policy and governance perspective. In addition, because municipal policies and interventions are often internally administered, the day-to-day challenges of implementing various initiatives may be difficult to evaluate. For this reason, the local political, economic, administrative, and cultural context must be considered to understand how and why certain interventions are successful in one context and not another. These factors may make this approach “ontologically more suited to case studies than to any other method” (Contandriopoulos et al. 2010, 453). The use of a systematic review methodology allows planners to consider the benefits and limitations of multiple intervention types, which address complex objectives with multiple, overlapping components. As the use of precedent is a key aspect of planning practice, researchers and practitioners may be well-served to consider the potential of using a modified SLR methodology to support the development of comprehensive case studies for practice.
If planning professionals hope to develop innovative approaches to planning issues and “push the knowledge frontier, we must know where the frontier is” (Xiao and Watson 2019, 93). While the application of an SLR methodology does not address the political relationships that influence policy outcomes—and thus cannot guarantee better results—this approach can help ensure that policymakers have access to comprehensive information about the intention and implications of multilevel interventions to support more informed decision-making.
Despite these limitations, the added value of systematically capturing and assessing a broad range of policy documentation outweighs the negative implications of using a process that is not entirely replicable. While the results of this study may have greater applicability to nations that share similar governance models to Canada, we believe that there is sufficient analytic generalizability in the methodological approach to contribute to research in other jurisdictions.
Conclusion
As a contribution to existing methodological knowledge, this paper highlights the potential for an SLR methodology to be effectively modified for use in planning practice. The key benefit of applying an SLR methodology within practitioner-led research is the ability to capture and assess evidence from a wider variety of sources, which may enrich the overall findings and better inform evidence-based decision-making. Using an SLR approach within practice encourages planners to consider a broader range of initiatives against an equal set of criteria. This may decrease practitioner bias toward one intervention type and could support the development of multilevel interventions, whereby a number of complimentary approaches are used to address municipal objectives based on the scope and strengths of each. Future research may consider how an SLR methodology could be expanded in practice to include other sources of gray and academic literature to create plausible use cases that better capture organizational contexts and geographic variables to support evidence-based decision-making and evaluation. Additional applications of this methodology may include state of the evidence reviews; assessing the aggregate impacts and effects of combining governance and regulatory approaches; and evaluating trade-offs regarding effectiveness, populations of interest, and accountability that may not be apparent when examining single policies or interventions alone.
While an SLR methodology may not guarantee better results when implementing a policy or initiative, it will contribute to a better understanding of the intentions and implications of a wider range of planning initiatives and support more informed decision-making. With appropriate modifications, an SLR methodology provides a robust approach to increasing the rigor and transparency of precedent research within planning practice.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Kristin Agnello would like to acknowledge the support of the Vanier Canada Graduate Scholarship Program and of Mitacs through the Mitacs Accelerate Program.
