Abstract
In this exploratory paper, we introduce a new method of plan quality evaluation to measure strength of policy focus. We demonstrate how the method can be applied to evaluate the strength of farmland protection policy in two local governments in the Province of British Columbia, Canada: the City of Richmond and Metro Vancouver Regional District. The method aims to not only describe strengths and weaknesses but also measure aspects of quality that help explain why a legislative framework is good. We suggest that this approach complements methods to evaluate effectiveness of farmland protection policies.
Introduction
Serious efforts to protect farmland from urban encroachment and other non-farm developments began in the 1950s (Bunce 1998). Although we have learned a great deal about factors that contribute to the success of these programs, the pressure to convert more agricultural land persists, and there is still more to learn about what makes one farmland protection program better than another. In this paper, we have two objectives, which are to demonstrate how a new method can be used to evaluate the strength of legislative frameworks to protect farmland and to demonstrate how this method can complement studies of effectiveness.
Questions about what makes a farmland protection program better relate to questions about what makes a good plan, which are central to the field of plan evaluation (see Berke and Godschalk 2009; Lyles and Stevens 2014). In general terms, plan evaluation concerns the systematic acquisition and assessment of plans, planning processes, and planning outcomes. While the majority of studies in this field have focused on evaluating outcomes as a measure of plan effectiveness, the field of plan evaluation also studies the quality of plans. Evaluation of plan quality is “the process by which plan content analysis data is linked to normative criteria of what constitutes a better plan” (Lyles and Stevens 2014, 434) based on established standards of plan quality (Stevens 2013). Because plan quality evaluation has a primary focus on a plan—as it is written—as the object of evaluation, this approach provides a direct means to evaluate the efficacy of land use policies and programs to protect agricultural land.
Past studies to evaluate farmland protection programs and policies have focused on measures of effectiveness, most often by analyzing changes in land uses and the consequent conversion or alienation of agricultural land. A wide range of early studies of farmland protection policies and programs generated valuable insights. For example, researchers concluded that approaches to preserving farmland must involve more than land use policy, be multi-faceted, and be legally defensible (e.g., American Farmland Trust 1997; Bryant and Russwurm 1982; Coughlin et al. 1980; T. Daniels and Bowers 1997; T. L. Daniels and Nelson 1986; Pierce and Furuseth 1982). Elements of a multi-faceted approach related to agricultural land use planning can include local comprehensive plans, agricultural zoning, exclusive agricultural districts, large lot zoning (minimum parcel sizes), buffers (edge planning), and rural residential zones on lower quality soils, to name a few. Studies also highlight the importance of protecting farmland and managing urban growth as mutually reinforcing (e.g., Alterman 1997; Bryant and Russwurm 1982; Coughlin et al. 1980; Nelson 1992). Having a high level of public support is another important factor supported by the results (e.g., Coughlin et al. 1980; Furuseth and Pierce 1982). Lessons from past studies, combined with an array of recent studies (e.g., American Farmland Trust 2020; T. Daniels and Payne-Riley 2017; Liu and Lynch 2011; Mullinix et al. 2013; Newman, Powell, and Wittman 2015; Nixon and Newman 2016; Schilling et al. 2014; Stobbe et al. 2011), provide insights to both individual planning tools and legislative frameworks as a whole.
In contrast to the extent of studies of the effectiveness of farmland protection programs and policies, very little attention has been given to evaluating the quality of farmland protection programs and policies. Methodologically, most studies rely on what Mundie (1982) refers to as programmatic summaries. Similarly, based on a review of studies completed in Oregon regarding the conservation of farm and forest lands, Gosnell et al. (2011) found that a majority of early studies were descriptive examinations of historical trends of land conversion and fragmentation, usually based on available data. Notwithstanding possible limitations, these studies direct attention to examining the elements of legislative frameworks as objects of study.
Thus, there are two aspects to this exploratory paper. Our primary purpose is to demonstrate how a new method of plan quality evaluation can be used to measure the strength of farmland protection as a focus of public policy. Through this method, we direct attention from measuring effectiveness to measuring efficacy, which is a shift from measuring produced effects to measuring the power to produce effects. The method employs ordinal measures of four, inter-related variables that are used to calculate an index of overall strength. The second aspect of this paper is to demonstrate potential benefits of using plan quality evaluation as a complement to studies of effectiveness. For these purposes, we evaluated two local governments in British Columbia, the City of Richmond and the Metro Vancouver Regional District. The city is a member municipality of the regional government. We believe that different approaches for studying efficacy and effectiveness can be used together to better understand and improve the quality of farmland protection policy.
The paper is organized as follows. We begin by situating our approach within the field of plan quality evaluation. We then review recent studies that have assessed the success of British Columbia’s farmland protection policy and describe our study sites. In the “Method” section, we explain our coding protocol and scoring rubric, as well as highlight exploratory aspects of our approach. After presenting our results, we discuss potential benefits, challenges, and limitations of the new method.
The methods and results of this evaluation of local governments in British Columbia complement an evaluation of the provincial legislative framework for agricultural land use planning. Both papers are premised on the methodological framework established by Connell and Daoust-Filiatrault (2018).
Plan Quality Evaluation
Evaluation concerns the systematic acquisition and assessment of information to provide useful feedback about the significance, worth, or condition of some object or intervention. For the field of plan evaluation, objects of assessment include plans, planning processes, and planning outcomes. These objects of study correspond with types of evaluation that occur at different stages of development, as follows: efficiency of plan development, efficacy of plan quality, and effectiveness of plan implementation (refer to Figure 1). As already noted, most studies of plan evaluation examine outcomes as measures of plan effectiveness.

Plan quality in relation to the field of plan evaluation.
The method presented in this paper concerns the sub-field of plan quality evaluation, which has a primary focus on a plan as the object of evaluation. The plan can be a single document or a set of documents. The method employs content analysis linked to normative criteria of how good or bad something is. In other words, to evaluate plan quality is to evaluate efficacy, which is “the power to produce a desired result or effect” (Merriam-Webster, n.d.). Thus, the power of a plan to produce an effect is embodied in the object of evaluation, whereas effectiveness relates to outcomes as objects of evaluation.
Plan quality evaluation can be divided into three areas of investigation, each distinguished by their purpose of study and methods, as shown in Figure 1. One dimension of plan quality is the extent of documentation. A comprehensive document will include goals, fact bases, policies, and content related to public participation, implementation, and monitoring (Berke and Godschalk 2009; Lyles and Stevens 2014). Plan quality is also concerned with communicative aspects of discourse, regarding the persuasiveness of a plan, or ability to engage and motivate the reader, and to garner the attention and response of policy makers and the public (Bunnell and Jepson 2011; Norton 2008). The corresponding principles, as operationalized by Norton (2008), are legibility, accuracy, legitimacy, and sincerity. The evaluation of policy focus stems from a need to analyze the message of a plan as an object of evaluation that is separate from the organization of a plan (Norton 2005, 2008). The coding protocol for measuring the strength of policy focus is premised on the following four principles: maximize the stability of policy focus, integrate public priorities across jurisdictions, minimize uncertainty that undermines the policy intent, and accommodate flexibility without contributing unnecessarily to uncertainty (Connell 2020; Connell and Daoust-Filiatrault 2018). Although each area of plan quality evaluation has important differences and can be assessed separately (e.g., Hossu et al. 2020), they all use plans as objects of study and employ methods of content analysis.
Based on their review of twenty years of studies of plan quality, Lyles and Stevens (2014) document cumulative efforts to develop coding protocols and rubrics to evaluate the extent of documentation (comprehensiveness). These methods have been applied to study such areas as hazard mitigation, environmental management, coastal management, growth management, sustainable development, and climate change. An evaluation of farmland protection fits among these areas of study, but there is a substantive difference between evaluating the comprehensiveness of farmland protection policies and the strength of policy focus. We can illustrate this difference by looking at Stevens’ (2013) evaluation of the quality of statutory land use plans for local governments in British Columbia. The purpose of Stevens’ research was to measure the comprehensiveness of plan documentation. Following established methods, the coding protocol was based on sixty-one items that covered implementation and monitoring, interorganizational coordination, participation, organization and presentation, and required elements under provincial legislation. Among the sixty-one items, only one was related directly to agriculture: “Does the plan contain at least one specific policy or action related to food and/or agriculture?” In contrast, a plan quality evaluation designed to measure the strength of policy focus would include only items related to farmland protection, as “clear evidence of an emphasis in plan policies favoring” (Norton 2005, 60) the intended issue.
Study Area
In British Columbia, when provincial legislation to protect agricultural land was enacted in 1973, the intervention was viewed as a “bold attack” (Rawson 1976, 42) that garnered mixed support and assorted views of its success (Garrish 2003). The legislation created the Agricultural Land Commission (ALC) and the Agricultural Land Reserve (ALR). The ALC is a quasi-judicial tribunal with a mandate to preserve agricultural land and responsibility to govern the ALR. The ALR is the name of the province-wide restrictive zone, within which agriculture is a priority use, farming and ranching are encouraged, and non-farm uses are restricted.
Among the first studies of the success of the provincial legislative framework, Manning and Eddy (1978) found that the legislation helped maintain viable farm units. Recent studies approach questions of effectiveness from many perspectives. Using hedonic land pricing models to assess the ALR’s effect on development pressures, Eagle et al. (2014) found that the ALR is effective at preserving farmland. Stobbe, Cotteleer, and van Kooten (2009) found that hobby farm uses led to increased prices of ALR, potentially undermining the goal of agricultural land preservation. By examining parcels that had been identified as underused for agriculture, Mullinix et al. (2013) found that the use of ALR land for non-agricultural purposes contributed to the effective loss of agricultural land and posed a threat to the maintenance of the ALR. Nixon and Newman (2016) used various spatial analytic processes to identify changes in land use in the Lower Mainland of British Columbia and then explored socio-political explanation for the patterns of change. Overall, most studies have found evidence of success to support the effectiveness of the legislative framework.
As in other jurisdictions, analysis of land use changes are measured as an important indicator of outcomes. Different from other jurisdictions, the province-wide restrictive zoning implemented in British Columbia (and in Québec) provides unique opportunities for evaluating effectiveness. In British Columbia, the legislative framework includes the ability for landowners and local governments to apply to remove, include, or subdivide agricultural land and for non-farm uses. In particular, the ability to examine applications to exclude land from the ALR, and the ALC’s decisions, provide a means to assess the success of the legislative framework to protect farmland (ALC 1983). Pierce (1981) was a forerunner of this approach. More recently, Stobbe et al. (2011) examined thirty years of applications to exclude land from the ALR. They considered spatial, political, and other factors that affect ALC decisions and found that parcel size and proximity to a highway were among significantly influential factors, while political factors were not statistically influential.
The legislative framework has also been the subject of non-academic studies. The ALC has used its database of decisions for internal reviews. For the City of Kelowna, the ALC and Ministry of Agriculture (British Columbia Ministry of Agriculture and Lands 2008) used past decisions regarding the exclusion of parcels from the ALR to reveal what the agricultural landscape may have looked like if all exclusion applications had been allowed. They found that 24 percent (2,367 hectares) of the original ALR land area would have been permanently removed, with a corresponding negative impact on the city’s agricultural industry.
Collectively, the studies of the effectiveness of British Columbia’s legislative framework provide a rich context in which to explore how an evaluation of plan quality might complement these studies. In this paper, we apply the proposed method to local governments, which are responsible for land use bylaws, including statutory plans, zoning bylaws, and other implementing bylaws such as development permit areas and edge planning. These bylaws concern not only lands designated as agricultural, but also activities on and development of non-agricultural lands that pose threats to agricultural land and farming activities. Urban growth management, including growth boundaries, are essential planning tools that help to protect farmland. However, local governments are not entirely free to do what they want. Depending on the province or state, senior-level legislative frameworks both enable and constrain what a local government can do. Furthermore, in British Columbia, there are also legislated relations between a regional government and its member municipalities.
To demonstrate how the method of plan quality evaluation can be applied to local governments, we selected the City of Richmond and Metro Vancouver Regional District as study areas. The results of an evaluation of the provincial legislative framework (Connell forthcoming) are included in this paper to show how farmland protection policies are integrated across the three levels of government in British Columbia.
In the remainder of this section, we provide a brief description of each study area, including the province of British Columbia. The legislative frameworks for the Metro Vancouver Regional District, the City of Richmond, and the Province of British Columbia are summarized in Online Appendix A.
British Columbia
In Canada, the responsibility for land use planning is shared among different levels of government, including federal, provincial, regional, and municipal governments. The divisions of power between the federal and provincial governments are determined by sections 91 and 92 of Canada’s Constitution Act, 1867, including the responsibility for land use planning as a matter that is local and concerns property rights. Local governments in Canada are not recognized in the Constitution Act, 1867; they are created by each province and have no inherent authority except that given to them by the provinces.
The primary responsibility for land use planning lies with provinces. In turn, provinces delegate most land use planning responsibilities to municipalities and regional governments through a municipal government act or planning act that enables and constrains what local governments can and must do. Generally, the authority delegated by each province to local governments is consistent across Canada. However, the level of provincial oversight and specific concerns for protecting farmland vary by province.
British Columbia has a somewhat strong legislative framework. The current provincial legislative framework includes the following key elements: Agricultural Land Commission Act (ALC Act, S.B.C. 2002, c. 36), Agricultural Land Reserve General Regulation (B.C. Reg. 171/2002), Agricultural Land Reserve Use Regulation (B.C. Reg. 30/2019), Local Government Act (LGA, R.S.B.C. 2015, c. 1), Land Title Act (R.S.B.C. 1996, Pt. 7, c. 250, s. 86(1)), and Farm Practices Protection (Right to Farm) Act (R.S.B.C. 1996, c. 131). As already noted, the ALC is a quasi-judicial tribunal that is responsible for implementing the Act. The ALR is based on restrictive land zoning, is province-wide, and includes all classes of agricultural land that cover about 4.6 million hectares, which includes most of the agricultural land in the province, but less than 5 percent of the provincial land base (ALC 2013). Only 1.1 percent of the provincial land base is considered prime agricultural land (Canada Land Inventory Classes 1–3) (ALC 2013).
The provincial legislative framework for agricultural land use planning includes legislated requirements for the integration of the provincial interest in protecting farmland across jurisdictions. The framework requires municipal and regional governments to adopt bylaws that are consistent with provincial legislation. Local government bylaws “must ensure consistency” with the ALC Act, regulations, and orders of the ALC (ALC Act s. 46 (2)). Furthermore, a bylaw that is inconsistent, to the extent of the inconsistency, has no force or effect (ALC Act s. 46 (4)).
Although the provincial legislative framework has legislated requirements for integration across jurisdictions, conformity with these legal requirements is undermined by the absence of provincial authority to approve statutory plans and zoning bylaws of local governments. When developing a statutory plan that affects ALR lands, a municipality must only “consult” with the ALC (LGA Pt. 13 s. 475(4)) and, before adoption, refer the plan to the ALC “for comment” (LGA Pt. 13 s. 477(3)(b)).
Metro Vancouver Regional District
The regional level of government in British Columbia is a “regional district” and often includes both incorporated areas (cities, towns, villages) as well as unincorporated electoral areas. A regional district performs three roles: it is the general-purpose local government for unincorporated areas, provides a governance mechanism for inter-municipal co-operation, and serves as regional governments for both voluntary activities decided upon by locally elected officials and a limited number of activities mandated by the provincial government (Union of British Columbia Municipalities [UBCM] 2008).
A regional district is governed by a board of directors (rather than a council), which includes municipal officials appointed to the board by their municipal councils and members from electoral areas who are elected directly. Generally, a regional district relies on inter-municipal co-operation and compromise; the level of influence a regional district has over its member municipalities varies widely.
Under the Local Government Act, a regional district has authority for areas that are not in a municipality (LGA Pt 14 s. 456 (b)). For areas under their control, a regional district may adopt a statutory plan (Pt. 14 Div. 4) and regulate land use through zoning bylaws (Pt. 14 Div. 5). The statutory, or comprehensive, plan is called an Official Community Plan (OCP). A regional district can also adopt a regional growth strategy (RGS) for the purpose of guiding decisions on growth, change, and development in all of its area (LGA Pt. 13 s. 429 (1)). A RGS should work toward avoiding urban sprawl (s. 428 (1) (a)) and “maintaining the integrity of a secure and productive resource base, including the agricultural land reserve” (s. 428 (1) (e)).
Metro Vancouver Regional District (known as Metro Vancouver) is located in southwestern British Columbia. Metro Vancouver has twenty-one member municipalities, including the City of Vancouver and the City of Richmond. Metro Vancouver also includes an Electoral Area and First Nations Treaty land. Metro Vancouver is the most populated region in the province, with a population of 2,463,431 and is expected to grow (Metro Vancouver Regional District 2011). At the same time, natural features constrain the land base, such as mountains and extensive floodplains. Consequently, the demand for urban development in the region is very high, contributing to significant pressure to develop agricultural lands.
Regarding the land base, 22 percent (60,893 hectares) of Metro Vancouver’s total land base is in the ALR (Ministry of Agriculture 2014). Disproportionately, although Metro Vancouver’s ALR land comprises only 1.5 percent of the total ALR land in the province, the region generates 27 percent of the province’s gross farm income (Metro Vancouver Regional District 2014). Farm products include greenhouse vegetables and flowers, potatoes, blueberries, cranberries, poultry, eggs, and dairy. Challenges to farming include high land prices, natural barriers, land speculation, non-farm uses on ALR land, and urban–rural conflicts.
City of Richmond
In British Columbia, a municipality is a general-purpose local government that provides a wide range of services and regulates a variety of activities (UBCM 2008). Members of municipal council are elected directly by constituents of its jurisdiction. A municipality may adopt a statutory plan (Pt. 14 Div. 4) and regulate land use through zoning bylaws (Pt. 14 Div. 5).
In addition to the legislated requirements for integration that apply to regional districts and municipalities, as noted above, the provincial legislative framework also requires integration between regional districts and its member municipalities. A RGS must be accepted by the affected municipalities and, thereafter, the RGS becomes binding (LGA Pt. 13, s. 436). Affected municipalities, as part of their statutory plan must then adopt a “regional context statement” that specifically identifies how the OCP relates to the RGS. Furthermore, a regional context statement and the OCP must be consistent; however, there are no standards for demonstrating consistency.
The City of Richmond, which is a member municipality of Metro Vancouver, is the fourth largest city in British Columbia. The City has a total population of 198,309 and, by 2041, is expected to grow by 40 percent. To accommodate this growth, roughly 42,000 new housing units will be required, with 61 percent of this growth expected in the City center (City of Richmond 2012). Richmond is facing a very high demand for urban development with very high pressure on the agricultural land base.
About 39 percent (4,993 hectares) of the city’s total land base is protected as ALR land (0.1% of all provincial ALR land) with 62 percent (3,072 ha) of this land being farmed (City of Richmond 2012). A major farming challenge in Richmond is drainage, given that most of the city is located in a flood zone; other issues concerning agricultural land in Richmond include rural–urban conflict, high land values, and non-farm uses on ALR land (City of Richmond 2003).
Method
The method of plan quality evaluation used in this study is designed to assess the strength of policy focus. 1 We propose that evaluating the strength of policy focus to protect farmland is a valuable approach that enriches descriptive summaries of legislative frameworks and complements efforts to evaluate the effectiveness of farmland protection programs and policies.
The premise of measuring strength of policy focus is a new development within the field of plan quality evaluation. Our development of the coding protocol and scoring rubric began in 2013, as part of a national research program to study principles and beneficial practices of agricultural land use planning in Canada. 2 The researchers wanted to analyze and compare legislative frameworks among and across provincial, regional, and municipal governments. The aim was to not only identify strengths and weaknesses but also to develop an analytical framework to explain why one legislative framework is better or worse than another. As more assessments were completed as part of the national project, and as the sets of data and results expanded, the scoring protocol was subject to continuous review and refinement. To date, the researchers have completed evaluations of over fifty legislative frameworks covering provincial, regional, and municipal levels of government.
For our purpose, we presume that protecting farmland is a desired outcome of a government, and a very strong policy focus is a preferred outcome. Thus, a very strong legislative framework is designed to protect farmland; a very weak framework exposes agricultural land to non-farm development. This range of options provides a scale of measurement, but does not presume that a government must have very strong farmland protection. Within the constraints and obligations of its senior government’s legislative framework, a local government usually has some discretion over the extent to which it seeks to protect farmland. The method aims to evaluate the extent to which a local government exercises this discretion.
The method of evaluation centers on a government’s legislative framework. The first step of the method is to identify the elements of a legislative framework that are relevant to agricultural land use planning. For local governments in British Columbia, a legislative framework can include statutory plans, zoning and other implementing bylaws, agricultural advisory committees (AACs), as well as enforceable and aspirational policies. The latter can include agricultural land use inventories, agricultural sector strategies, as well as food systems strategies.
The second step is to measure the strength of policy focus based on content analysis. Our coding protocol is premised on four principles: maximize stability, integrate public priorities across jurisdictions, minimize uncertainty, and accommodate flexibility. For the scoring, each principle is scored for the presence and strength of specified items. An ordinal five-point scale from very weak (1) to very strong (5) is used, as described in Online Appendix B. Here, we describe how we operationalize the four principles as criteria, which relies on theoretical interpretation.
To maximize stability is to make a commitment to protecting farmland that cannot be easily changed, especially not at the whim of shifting political interests. It is important to express policy in clear, concise language that is entrenched through the legislative means (e.g., laws, regulations, statutory plans, implementing bylaws). Correspondingly, one can use the following questions as guides to analyze the content of a legislative framework:
To what extent does the local government maximize stability? Is the commitment to protect farmland clear, direct? Is the commitment entrenched in a statutory plan? Is the commitment expressed as part of a long-term vision, over-arching goal, or guiding principle?
Integrating public priorities across jurisdictions is like a “policy thread” (Smith 1998) that weaves together legislative frameworks. Three components help to ensure that lower level policies are set within the context of broader public priorities. (1) Statutory requirements: the level of integration of lower level policy with upper level policy as required by legislation. Senior government policies that advance integration often require lower level governments “to be consistent with” or “to conform to” upper level policies. (2) Statutory authority: the extent of the jurisdiction’s authority over agricultural land use planning. This component concerns whether or not, or the extent to which, senior levels of government retain (or delegate) approval of local government statutory plans and implementing bylaws. (3) Demonstrated consistency: from a local government perspective, this aspect refers to the extent to which a local government recognizes the senior-level legislative requirements for protecting farmland. Corresponding questions include the following:
To what extent does the local government integrate its commitment to protect farmland with upper level governments? Are all relevant elements of the upper level legislative framework identified? Are relevant elements of the upper level legislative framework described? If applicable, is the commitment to protect farmland connected directly with upper level interests in protecting farmland?
Uncertainty relates to expectations about how a legislative framework will be applied to future land use decisions. In this regard, loopholes, ambiguous language, and open-ended conditions are factors that contribute to uncertainty; the presence of such factors weakens a legislative framework. Uncertainty also concerns urban pressures on agricultural land. In this regard, urban growth management policies that include designated growth boundaries and density targets are significant factors that help to minimize uncertainty. Ideally, policies that direct urban development to designated areas signify that agricultural lands will not be developed for non-farm uses. Additional policy tools, such as covenants and designating specific areas for future development under specific conditions, also help to minimize uncertainty. Recognizing “right to farm” legislation is also important because it protects farmers’ rights to use their land for normal operations; the legislation provides a legal basis for land use zoning and use of planning tools that support agricultural activities, such as covenants. Corresponding questions include the following:
To what extent does the local government minimize uncertainty? Is there a commitment to containing urban growth (e.g., urban boundaries, density targets)? How firm is the commitment to maintain the designated urban boundaries? Are there any loopholes, qualifying statements, or vague language that undermine the commitment to protect farmland?
Inevitably, governments must address conflicts over potential uses of land, and land use planning seeks a balance that best manages these issues. The need for “flexibility” is often raised in this context. The principle of accommodating flexibility recognizes this need and that certain policies can function as mechanisms that moderate the restrictive effects of maximizing stability and minimizing uncertainty. Policy mechanisms that can accommodate flexibility without undermining stability or contributing to uncertainty include agricultural impact assessments, edge planning (farm/non-farm interface areas), and development permit areas. To help ensure that these kinds of policy mechanism are applied with specific regard for the interest in protecting agricultural land, it is important that a legislative framework includes and refers to appropriate governance mechanisms, such as quasi-judicial commissions, AACs, and application processes. Corresponding questions include the following:
How well does the local government accommodate flexibility? Is there an AAC? Are planning tools used to manage interface areas (e.g., edge planning, buffers, development permit areas)? Is an agricultural impact assessment required when considering non-farm developments that affect agricultural land and farming activities? Are non-farm developments directed to lands with lower agricultural capability?
The third step is to measure overall strength of policy focus. We follow a common approach used in plan quality evaluation, which is to calculate an index of overall plan quality based on category scores (Lyles and Stevens 2014). The index is then used to rank plans on the premise that higher scores indicate a higher quality (better) plan (Guyadeen 2019; Horney et al. 2017; Stevens 2013). Our index of overall strength of policy focus is based on the sum of non-equal weighted scores for each principle, as presented in Online Appendix C.
The weighting of the four principles accounts for the relative importance of each principle as well as relationships among the principles. Stability is the cornerstone of the strength of policy focus. As such, it carries more weight than the other three principles. A score for maximizing stability is doubled. A local government’s ability and opportunity to integrate vertically depend on the strength of their corresponding upper level legislative framework(s). To account for this relationship, a higher weighting is given for integrating with stronger upper level frameworks. That is, a higher level of integration by a local government with a strong provincial/state legislative framework serves to enhance the strength of the local framework. It is better for a local government to make an effort to minimize uncertainty, and worse if it does not. Correspondingly, the weighting for minimizing uncertainty expands the 1 to 5 scale into a 0.5 to 7.5 scale. A government should consider accommodating flexibility only after it addresses the other three principles. On this basis, the weighting for this principle is premised on the scores for the other three principles, whereby a local government is penalized for not taking care of the other three principles. A score for flexibility is adjusted down when the sum of scores for the three principles are lower. The sum of weighted scores (maximum = 32.5) determines overall strength of policy focus, which is based on a seven-point scale from very weak to very strong. The breakpoints between overall ratings, for example, between strong and very strong, should be interpreted with caution. There is no meaningful difference between, for example, 18.49 and 18.5, yet these two overall scores lead to different overall ratings. In situations like this, there is room for interpretation.
Results
The purpose of this project was to apply a new method of policy analysis to local governments in British Columbia. Our aim was to measure the efficacy or strength of policy focus for protecting farmland in Metro Vancouver and the City of Richmond. We first present a summary of the legislative frameworks and of the overall results before presenting the results for the set of principles for each local government. For each level of government, we review key elements of the legislative framework with examples of statements that inform the evaluation scores for each principle.
A summary of the results of our analyses for Metro Vancouver and the City of Richmond are shown in Table 1. For the overall scores for strength of policy focus on protecting farmland, the province is somewhat strong and the two local governments are rated as strong. Among the individual principles, all three levels of government have very strong ratings for maximizing stability and all three have ratings of strong or very strong for minimizing uncertainty. Notably, although the legislative frameworks for protecting farmland in the two local governments are both strong, there are differences among the four principles.
Strength of Local Legislative Framework: Province of British Columbia.
* = very weak; ***** = very strong.
The following results illustrate how multiple levels of government can be evaluated to produce individual measures of overall strength of policy focus. The results also illustrate how the method reveals similarities and differences.
Metro Vancouver Regional District
Overall, Metro Vancouver’s legislative framework for the protection of agricultural land is strong. A combination of RGS, Regional Food System Strategy (RFSS), and Regional Food System Action Plan (RFSAP) supports a clear commitment to protect farmland in conjunction with urban containment boundaries (UCBs) for all of its member municipalities. The strengths of each principle vary, but there are no weak elements. As a regional district, Metro Vancouver is a federation; it has no authority independent of its member municipalities. To some extent, the legislative framework is a negotiated outcome. For Metro Vancouver, its control over region-wide sewage and water infrastructure has been a source of leverage to enact very strong urban containment districts. This element of its legislative framework shows up as a strength under minimize uncertainty. At the same time, a regional district is limited with regard for its influence regarding planning tools that help to accommodate flexibility.
Metro Vancouver’s legislative framework for protecting agricultural land is rated very strong in maximizing stability. As stated in the RGS, “The challenge for the Regional Growth Strategy is to protect the agricultural land base and to encourage its active use for food production” (Metro Vancouver Regional District 2011, 6). The RGS is strongly linked to a broad set of key documents, including the food strategy and action plan. In addition, the Metro Vancouver Regional Land Use Inventory Report provides comprehensive data about the agricultural sector and land base in the region. Collectively, these documents support a clear commitment to protect farmland, which provides a foundational element of a comprehensive approach. The RFSS identifies “disappearing farmland” as a growing concern, while the RFSAP identifies steps to address this concern.
Metro Vancouver is strong in integrating across jurisdictions. The RGS has multiple, detailed references to the ALC Act. It also refers to efforts to create consistency between the RGS and ALC policies, and has specific statements of what Metro Vancouver should work on in collaboration with the province. For example, the RGS states that the Agricultural land designation is intended to “reinforce provincial and local objectives to protect the agricultural land base of the region” (Metro Vancouver Regional District 2011, 9). The RGS (Strategy 2.3.10) states that the province, in consultation with municipalities, establishes policies that set maximum residential floor area and setback regulations for development within the ALR.
The RGS also supports integration between the region and its member municipalities. As per Strategy 2.3.6, the role of member municipalities is to adopt regional context statements that address an extensive list of policy areas, including discouraging non-farm uses that do not complement agriculture and subdivision of agricultural land leading to farm fragmentation, and managing the agricultural–urban interface to protect the integrity and viability of agricultural operations. Specific references to local governments appear elsewhere. Strategy 2.3 states that local governments should include policies that “encourage the use of agricultural land, with an emphasis on food production.” As well, Strategy 1.3 “Protect Rural areas from urban development” states that municipalities should include policies that support agricultural uses within the ALR.
The RFSS and supporting Action Plan refer to the ALC Act and the Water Act. As well, the roles and responsibilities of different groups (such as the province, federal government, the Regional District, and municipalities) are outlined in the RFSS. The RFSAP enhances integration, stating that municipalities should create development permit areas for farmland. In the RFSS and RFSAP, and the RGS, there are references to the Farm Practices Protection (Right to Farm) Act, but only a vague mention of municipal AACs.
Metro Vancouver is very strong in minimizing uncertainty for the protection of agricultural land. The RFSS and Action Plan work to further their commitment, and consistency is found throughout these two documents and the RGS. The implementation of UCBs in every municipality is a significant policy. The stated purpose of the UCB (Metro Vancouver Regional District 2011, 9) is “to establish a stable, long-term, regionally defined area for urban development” that, among other things, reinforces the protection of agricultural land. Another important policy is Strategy 2.3.4, which states that Metro Vancouver will work with the ALC “to protect the region’s agricultural land base.” Furthermore, Strategy 2.3.2 states that Metro Vancouver will monitor the status of agricultural land “with the objective of promoting agricultural viability and food production” in collaboration with the province and ALC.
Metro Vancouver is rated moderate in accommodating flexibility. The strongest contribution is the regional AAC, which brings agricultural issues into land use decisions. However, the AAC is not mentioned in the RGS or RFSS and Action Plan, which would enhance the significance of this committee. The RGS includes only a few policy statements that accommodate flexibility. These statements include references to municipalities managing the agricultural–urban interface and to the province establishing and enforcing maximum residential floor area and setback regulations for development within the ALR. The Food Strategy and Action Plan documents contain more extensive recommendations that would help to accommodate flexibility.
City of Richmond
The City of Richmond’s local legislative framework for farmland protection is strong. The City has a very clear, direct commitment to protecting farmland that is supported by a comprehensive set of policies. The results demonstrate consistent levels of strength across all four principles.
The City of Richmond’s legislative framework is very strong with regard for maximizing stability. Richmond’s OCP emphasizes the importance of farmland protection, acknowledges the City’s historical roots in agriculture, and recognizes agriculture as a “heritage asset.” The OCP Vision states that, as part of achieving a sustainable city, the City will have “protected and productive agricultural lands.” As well, keeping farmland intact and productive is a stated direction for planning. In the section dedicated to Agriculture and Food (7.0), the over-arching theme is protecting farmland and enhancing its viability. References that identify many different benefits to farmland protection are also made throughout the document, including sections on Climate Change Adaptation, Sustainable Resource Use, and Resilient Economy. Furthermore, Richmond’s Area and Sub-Area Plans also increase stability, with especially strong language found in the Hamilton and McLennan Sub-Area Plans. For example, the only two goals of the McLennan sub-area plan are to preserve agricultural lands and minimize urban–rural conflicts.
Another legislative document contributing to stability is Richmond’s Agricultural Viability Strategy (AVS) (and corresponding Profile and Survey Reports), an enforceable policy that is referenced several times in the City’s OCP. The AVS aims to strengthen and enhance the agricultural sector in Richmond. As stated, “The purpose of the AVS is not to remove land from the Agricultural Land Reserve unless there is a substantial net benefit to agriculture and there is consultation with agricultural stakeholders” (City of Richmond 2003, 5).
Integration for the City of Richmond is strong. There are numerous references to provincial and regional legislative documents. Specifically, there is substantial integration between the Metro Vancouver RGS, the AVS, and OCP. To ensure Richmond’s OCP aligns closely with the RGS, the required regional context statements list specific strategies and goals identified by Metro Vancouver and match them to policies in their OCP. One specific strategy from the RGS is Strategy 2.3, which aims to “protect the supply of agricultural land and promote agricultural viability with an emphasis on food production.” In support of this strategy, the City notes that their agricultural policies are in line with this strategy and that other documents, such as the AVS, also support this regional goal. Beyond references and the alignment of goals between documents, Richmond also emphasizes a desire for co-operation with the Region.
Richmond’s legislative framework is rated as strong with regard for minimizing uncertainty. In particular, the UCB restricts growth to urban areas while protecting farmland. The Area and Sub-Area Plans containing ALR land all recognize the importance of protection and created policies with this in mind, contributing to consistency.
Richmond’s legislative framework rated strong in accommodating flexibility. Their municipal AAC complements the regional AAC. Richmond’s zoning accommodates flexibility by providing three different types of agricultural zones and a specific golf course zone. Richmond also has a number of Site Specific Zones in their zoning bylaw, three of which pertain to agriculture. The City recognizes the importance of protecting intensive agricultural areas and also allowing other uses on what is probably less desirable farmland. The final piece of flexibility can be seen in their desire to establish development permit areas specifically for protecting farmland. Another way in which the City accommodates flexibility is through their ALR landscape buffers, which are intended to “accommodate and encourage development while minimizing the impacts of new developments on agricultural land.”
Discussion: Potential Benefits and Limitations
Through this paper, our aim was to introduce a new method to measure the strength of policy focus using four principles as criteria. We applied this method to evaluate the legislative frameworks of Metro Vancouver Regional District and the City of Richmond. After a brief discussion of the results of the assessment, we discuss potential benefits of the method and how the results of the method might complement other studies of farmland protection policies and programs.
Whereas all three levels of government included in this evaluation are known for their efforts to protect agricultural land, the results illustrate how these commitments are embedded in legislative frameworks. As presented in Table 1, the very strong commitments to protecting farmland are evident in scores for maximizing stability, while differences among the results across all four principles help to reveal a more nuanced understanding of what makes them strong. As well, the results indicate that the method itself can account for different formulations of policy focus for protecting agricultural land.
The results reflect other differences between the two study areas. As a regional district, Metro Vancouver is a federation for which its member municipalities must negotiate for differing interests and priorities. As an entity, Metro Vancouver has control over region-wide sewage and water infrastructure, which translates into sufficient leverage to enact very strong UCBs for all its member municipalities, which appears as a strength among the four principles under minimize uncertainty. At the same time, a regional district is limited with regard for its influence regarding planning tools that help to accommodate flexibility. On the contrary, the City of Richmond has a strong rating for accommodating flexibility. The overall strong rating for the city is consistent with Newman, Powell, and Wittman (2015), who concluded that the legislative framework generally stabilized land use in the area.
The results for all three levels of government lend insight about the integration of public priorities across jurisdictions. Recall that the rubric for overall strength accounts for relations between levels of government, whereby the score is higher when a lower level government integrates well to a stronger upper level government. In our results, all three governments have very strong commitments to protecting agricultural land, while the municipal and regional governments have strong ratings for integration. These combinations contribute to the overall strength of the local government legislative frameworks. Of note, these strong ratings occur in the absence of provincial approval authority for local statutory plans.
A potential benefit of the approach outlined in this paper is to use strength of policy focus as a complement to studies of effectiveness. For this purpose, we consider the recent study by Stobbe et al. (2011), which examined spatial, political, and other factors that affect ALC decisions on applications to exclude land from the ALR. Although the authors analyzed political factors, including the political party in power at the time and the municipal government’s position regarding the application, they did not consider the quality of agricultural land use policy, either at the provincial or local levels. By making this observation, we are not arguing that the study is flawed; we highlight this omission only as a means to illustrate how plan quality can be incorporated as an independent variable.
The purpose of the Stobbe et al. (2011) study was to understand factors that affect decisions to change land zoning from agriculture to other uses. Among a suite of factors, the authors considered whether the proposed use of the land had the backing or opposition of the four municipal governments included in their study areas. Given the purpose of public policy to guide decisions (for both the municipal governments and the ALC), there is presumptive value to consider the influence of relevant policies in application decisions. Furthermore, the municipal governments’ positions on the specific applications may or may not be consistent with their respective legislative frameworks. As such, there may be value to include the strength of each municipality’s legislative framework as an independent variable.
The strength of policy focus for protecting farmland could also have been used to select the study areas. Presently, the strengths of the four legislative frameworks that were included in the study are not known. In addition to understanding how similar or different strengths of policy focus might have affected the results, study sites could have been selected based on these criteria. For example, study sites could have been selected because they all had strong legislative frameworks, thus accounting for, and perhaps minimizing or eliminating, one potential influence on application decisions. Alternatively, sites could have been selected for having different strengths of policy focus. Although including strength of policy focus may not affect the results of the study, including this factor as an independent variable would help to account for an important element influencing application decisions.
Challenges and Limitations
At this stage of development of the method, future applications by different researchers will help to address several important challenges and limitations regarding the coding protocol and the scoring rubric. One broad challenge concerns the inconsistent use of “efficacy” in plan evaluation studies. When the term is used, it is rarely defined and often operationalized as effectiveness (e.g., Allred and Chakraborty 2015; Nixon and Newman 2016). An exception is Curran and Stobbe (2010, 9), who operationalize the term as follows: “How closely does the policy or program meet the objectives of Metro Vancouver for this project (protect agricultural lands, improve the viability of farming, increase actively farmed lands)?” Statements from their results of evaluating efficacy include the following: “potential for significant impact on protecting farmland”; “helps establish clear expectations about what will happen on land”; “provides direct means to preserve farmland and increase actively farmed land”; and “enables permanent protection of purchased lands and the creation of ag potential in urban areas.” These statements are consistent with an understanding of efficacy as the power to produce a desired effect. Importantly, the inconsistent use of terms in the literature may reflect a broader debate that questions the premise of plan quality evaluation as worthy of study. Namely, the ability to analytically separate plan quality from plan implementation is questioned. If a plan as-it-is-written cannot be separated from outcomes then, presumably, there is no benefit to evaluating the strength of policy focus. However, this argument would flow both directions: if outcomes cannot be separated from the quality of the plan, then studies of effectiveness would be challenged to draw conclusions regarding the need to strengthen policy as a means to attain better outcomes.
The majority of studies of plan quality evaluate the comprehensiveness of a plan’s documentation, for which standard approaches have been developed and used for statistical analysis (Lyles and Stevens 2014; Stevens 2013). These approaches typically employ a combination of binary (present/not-present) and ordinal measures for individual items, as well as an index for an overall score. The index is calculated based on equal weighting of categories. Our method to measure the strength of policy focus differs from these standards for measuring comprehensiveness. Instead of scoring individual items, we scored only each of the four principles on a five-point ordinal scale and then used non-equal weighting to calculate an index for overall strength of policy focus. Our plan quality index for overall score and individual characteristics are not intended for statistical analysis.
For the coding protocol, the four principles were developed to account for different views of what makes one plan better than another. Based on feedback, we found that some people consider the principles too simple to be useful, some people find them too complicated to be helpful, and some people find the principles to be intuitively powerful. We have come to accept that the principles may not satisfy everyone’s perspective and we remain open to feedback. While there is room to debate whether the terminology is appropriate, we believe it is more important to refine what land use planning policies and tools align with each principle. A related matter is to develop a core consensus regarding the ideal legislative framework to protect agricultural land. As discussed below, there may be merit in developing a theoretical ideal that transcends jurisdictional constraints. This course of evaluation would have to account for inter-jurisdictional differences, especially at the country-to-country level, where substantive differences relate to constitutional authority for land use planning and private property rights.
Regarding the scoring rubric, the method employs elements that deviate from standards used to measure the quality of a plan’s documentation. These new elements include scoring at the category level instead of individual items, and the use of non-equal weighting to calculate an index. These exploratory elements warrant questions about the validity and reliability of the method that can only be addressed through future applications that aim to replicate the results of this study and to apply the method in new jurisdictions. As well, applications of the method that aim to incorporate tests of statistical significance and of inter-coder reliability would require continued refinement and quantification of the coding protocol.
The results for our study sites, as presented in Table 1, point to questions about how can or should the method account for differences across levels of local government. Should there be a single ideal benchmark of “very strong” for each principle against which all local governments are measured regardless of jurisdiction, that is, a theoretical ideal that transcends jurisdictions? Or should the benchmark that represents “very strong” be adjusted for each site based on the planning authority within a particular jurisdiction? Our position is that a theoretical ideal is the most useful. However, although the literature points to elements and characteristics of a quality legislative framework for protecting farmland, there is no consensus on an ideal legislative framework. Consensus is also unlikely given legal conditions that differ among jurisdictions. The scale we developed drew from our experience of completing assessments of legislative frameworks for different levels of government. However, all of these assessments were completed in Canada and may not be fully transferable to other countries.
A single benchmark means that municipal and regional governments would be evaluated using the same criteria, which is the method we used for our two study areas. For a single site study, there may be advantages to creating site-specific benchmarks, as the scores would reveal the strength of policy focus of a local government against what is possible for that government. However, this customized approach eliminates the possibility of comparing multiple sites that operate with different types of planning authority. In contrast, a primary advantage of using a theoretical ideal is the opportunity for comparative analysis under different legislative frameworks, with the possibility of comparing sites not only within a single province or state, but also across provinces/states, and between different countries.
Thus, comparing and interpreting results across levels of government warrant an important qualification. Although the same seven-point scale for overall strength is used for all levels of government, these measures are based on different rubrics, as a particular formulation of policy focus reflects, in part, the legislative frameworks that enables and constrains the authority of a local government. As such, results across levels of jurisdiction cannot be compared directly. That is, the overall strength of the province cannot be compared directly with Metro Vancouver or Richmond. Likewise, regional and municipal governments cannot be compared directly. Rather, each level of government can be compared directly only among governments at the same level. To elaborate, the reason is that the ideal “very strong” legislative framework is different for each level of government, in accordance with each level of government’s authority for land use planning. Thus, each level of government must be evaluated by what is possible, and ideal, within the governing legislative framework.
Conclusion
In this paper, our purpose was to demonstrate how a new method to evaluate the strength of policy focus for protecting farmland can be applied to local governments. The method is based on a coding protocol using four principles of policy analysis and on a rubric that employs ordinal scales and non-equal weighting to calculate an index of overall strength. We applied the method to the municipality of the City of Richmond and the regional government of Metro Vancouver. The results show that both local governments have strong legislative frameworks, with each government using its land use planning authority and different planning tools to achieve these outcomes.
We propose that evaluating the strength of a policy focus to protect farmland is a valuable approach that enriches descriptive summaries of legislative frameworks and complements efforts to evaluate the effectiveness of farmland protection programs and policies. Our aim was to develop a method that not only identifies strengths and weaknesses of legislative frameworks but also provides an analytical framework to help explain why one legislative framework is better or worse than another. In this regard, a potential advantage of this method rests upon the combination of the four principles: maximize the stability of policy focus, integrate public priorities across jurisdictions, minimize uncertainty that undermines the policy intent, and accommodate flexibility without contributing unnecessarily to uncertainty. The scoring rubric is designed to value the merit of each principle while accounting for relations among them.
The underlying premise of evaluating the strength of policy focus is that a higher quality plan is more likely to be implemented and achieve intended outcomes. However, as Brody, Carrasco, and Highfield (2006, 300) stress, there is an important caveat: “Even if a policy is present and mandatory in a plan, there is no guarantee this policy is implemented as a regulation and enforced by the local jurisdiction.” While this caveat may be considered a limitation of evaluating policy focus, it also speaks to an opportunity. To measure the strength of policy focus is to evaluate the power of a legislative framework to produce outcomes. Thus, this method is different from, and is intended as a complement to, evaluations of effectiveness that measure outcomes. Ideally, evaluations of efficacy and effectiveness are completed together.
The field of plan quality evaluation is relatively young and has benefited from continuous refinements by different researchers (Lyles and Stevens 2014). We hope that the evaluation of policy focus, as a nascent dimension of plan quality, will benefit from ongoing developments in the broader field of plan evaluation and from future applications by other researchers.
Supplemental Material
online_supp_Connell – Supplemental material for Evaluating the Strength of Local Legislative Frameworks to Protect Farmland: City of Richmond and Metro Vancouver, British Columbia
Supplemental material, online_supp_Connell for Evaluating the Strength of Local Legislative Frameworks to Protect Farmland: City of Richmond and Metro Vancouver, British Columbia by David J. Connell in Journal of Planning Education and Research
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for this study was provided by Social Sciences and Humanities Research Council of Canada (435-2013-1726).
Supplemental Material
Supplemental material for this article is available online.
Notes
Author Biography
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
