Plan quality evaluation researchers typically evaluate plans in relation to whether they contain certain desirable features. Best practice dictates that plans be evaluated by at least two readers and that researchers report a measure of the extent to which the readers agree on whether the plans contain the desirable features. Established practice for assessing this agreement has been subject to criticism. We summarize this criticism, discuss an alternative approach to assessing agreement, and provide recommendations for plan quality evaluation researchers to follow to improve the quality of their data and the manner in which they assess and report that quality.
AyturSemra A.SatinskySara B.EvensonKelly R.RodriguezDaniel A.2011. “Pedestrian and Bicycle Planning in Rural Communities: Tools for Active Living.” Family & Community Health34 (2): 173–81.
2.
BaerWilliam C.1997. “General Plan Evaluation Criteria: An Approach to Making Better Plans.” Journal of the American Planning Association63 (3): 329–44.
3.
BakerIngridPetersonAnnBrownGregMcAlpineClive. 2012. “Local Government Response to the Impacts of Climate Change: An Evaluation of Local Climate Adaptation Plans.” Landscape and Urban Planning107 (2): 127–36.
4.
BassettEllenShandasVivek. 2010. “Innovation and Climate Action Planning—Perspectives from Municipal Plans.” Journal of the American Planning Association76 (4): 435–50.
5.
BaynhamMaggieStevensMark. 2013. “Are We Planning Effectively for Climate Change? An Evaluation of Official Community Plans in British Columbia.” Journal of Environmental Planning and Management Published electronically March 15, 2013. doi:10.1080/09640568.2012.756805.
6.
BerkePhilipFrenchSteven P.1994. “The Influence of State Planning Mandates on Local Plan Quality.” Journal of Planning Education and Research13 (4): 237–50.
7.
BerkePhilipGodschalkDavid. 2009. “Searching for the Good Plan: A Meta-analysis of Plan Quality Studies.” Journal of Planning Literature23 (3): 227–40.
8.
BerkePhilip R.1994. “Evaluating Environmental Plan Quality: The Case of Planning for Sustainable Development in New Zealand.” Journal of Environmental Planning and Management37 (2): 155–70.
9.
BerkePhilip R.CrawfordJanDixonJennyEricksenNeil. 1999. “Do Cooperative Environmental Planning Mandates Produce Good Plans? Empirical Results from the New Zealand Experience.” Environment and Planning B: Planning and Design26 (5): 643–64.
10.
BerkePhilip R.DixonJennyEricksenNeil. 1997. “Coercive and Cooperative Intergovernmental Mandates: A Comparative Analysis of Florida and New Zealand Environmental Plans.” Environment and Planning B: Planning and Design24 (3): 451–68.
11.
BerkePhilip R.EricksenNeilCrawfordJanDixonJenny. 2002. “Planning for Indigenous People: Human Rights and Environmental Protection in New Zealand.” Journal of Planning Education and Research22 (2): 115–34.
12.
BerkePhilip R.GodschalkDavid R.KaiserEdward J. with RodriguezDaniel A.2006. Urban Land Use Planning, 5th ed.Urbana: University of Illinois Press.
13.
BerkePhilip R.Manta-ConroyMaria. 2000. “Are We Planning for Sustainable Development? An Evaluation of 30 Comprehensive Plans.” Journal of the American Planning Association66 (1): 21–33.
14.
BerkePhilip R.RoenigkDale J.KaiserEdward J.BurbyRaymond. 1996. “Enhancing Plan Quality: Evaluating the Role of State Planning Mandates for Natural Hazard Mitigation.” Journal of Environmental Planning & Management39 (1): 79–96.
15.
BerkePhilipSmithGavinLylesWard. 2012. “Planning for Resiliency: Evaluation of State Hazard Mitigation Plans under the Disaster Mitigation Act.” Natural Hazards Review13 (2): 139–49.
16.
BerkePhilipSpurlockDanielleHessGeorgeBandLarry. 2013. “Local Comprehensive Plan Quality and Regional Ecosystem Protection: The Case of the Jordan Lake Watershed, North Carolina, U.S.A.” Land Use Policy31 (March): 450–9.
17.
BrodySamuel D.2003a. “Are We Learning to Make Better Plans? A Longitudinal Analysis of Plan Quality Associated with Natural Hazards.” Journal of Planning Education and Research23 (2): 191–201.
18.
BrodySamuel D.2003b. “Implementing the Principles of Ecosystem Management through Local Land Use Planning.” Population and Environment24 (6): 511–40.
19.
BrodySamuel D.CarrascoVirginiaHighfieldWesley E.2006. “Measuring the Adoption of Local Sprawl Reduction Planning Policies in Florida.” Journal of Planning Education and Research25 (3): 294–310.
20.
BrodySamuel D.HighfieldWesCarrascoVirginia. 2004. “Measuring the Collective Planning Capabilities of Local Jurisdictions to Manage Ecological Systems in Southern Florida.” Landscape and Urban Planning69 (1): 33–50.
21.
BunnellGeneJepsonEdwad J.Jr.2011. “The Effect of Mandated Planning on Plan Quality.” Journal of the American Planning Association77 (4): 338–53.
22.
BurbyRaymond J.DaltonLinda C.1994. “Plans Can Matter! The Role of Land Use Plans and State Planning Mandates in Limiting the Development of Hazardous Areas.” Public Administration Review54 (3): 229–38.
23.
BurbyRaymond J.MayPeter J., with BerkePhilip R.DaltonLinda C.FrenchSteven P.KaiserEdward J.1997. Making Governments Plan: State Experiments in Managing Land Use. Baltimore, MD: Johns Hopkins University Press.
24.
CarminesEdward G.ZellerRichard A.1979. Reliability and Validity Assessment. Newbury Park, CA: Sage.
25.
CohenJacob. 1960. “A Coefficient of Agreement for Nominal Scales.” Educational and Psychological Measurement20 (1): 37–46.
26.
ComptonD’LaneLoveTony P.SellJane. 2012. “Developing and Assessing Intercoder Reliability in Studies of Group Interaction.” Sociological Methodology42 (1): 348–64.
27.
DaltonLinda C.BurbyRaymond J.1994. “Mandates, Plans, and Planners: Building Local Commitment to Development Management.” Journal of the American Planning Association60 (4): 444–61.
28.
DavisBraxton C.2004. “Regional Planning in the US Coastal Zone: A Comparative Analysis of 15 Special Area Plans.” Ocean and Coastal Management47 (1–2): 79–94.
29.
DeyleRobert E.SmithRichard A.1998. “Local Government Compliance with State Plan Mandates: The Effects of State Implementation in Florida.” Journal of the American Planning Association64 (4): 457–69.
30.
EdwardsMaryHainesAnna. 2007. “Evaluating Smart Growth: Implications for Small Communities.” Journal of Planning Education and Research27 (1): 49–64.
Evans-CowleyJennifer S.GoughMeghan Z.2008. “Evaluating Environmental Protection in Post-Hurricane Katrina Plans in Mississippi.” Journal of Environmental Planning and Management51 (3): 399–419.
33.
Evans-CowleyJennifer S.GoughMeghan Z.2009. “Evaluating New Urbanist Plans in Post-Katrina Mississippi.” Journal of Urban Design14 (4): 439–61.
34.
EvensonKelly R.SatinskySara B.RodriguezDaniel A.AyturSemra A.2012. “Exploring a Public Health Perspective on Pedestrian Planning.” Health Promotion Practice13 (2): 204–13.
35.
FreelonDeen G.2010. “ReCal: Intercoder Reliability Calculation as a Web Service.” International Journal of Internet Science5 (1): 20–33.
36.
GodschalkDavid R.BeatleyTimothyBerkePhilipBrowerDavid J.KaiserEdward J.1999. Natural Hazard Mitigation: Recasting Disaster Policy and Planning. Washington, DC: Island Press.
37.
HaminElisabeth M.2011. “Integrating Adaptation and Mitigation in Local Climate Change Planning.” In Climate Change and Land Policies, edited by IngramGregory K.HongYu-Hung. Cambridge, MA: Lincoln Institute of Land Policy.
38.
HayesAndrew F.KrippendorffKlaus. 2007. “Answering the Call for a Standard Reliability Measure for Coding Data.” Communication Methods and Measures1 (1): 77–89.
39.
HochCharles. 2007. “How Plan Mandates Work: Affordable Housing in Illinois.” Journal of the American Planning Association73 (1): 86–99.
40.
HorneyJennifer A.NaimiAshley I.LylesWardSimonMattSalvesenDavidBerkePhilip. 2012. “Assessing the Relationship between Hazard Mitigation Plan Quality and Rural Status in a Cohort of 57 Counties from 3 States in the Southeastern U.S.” Challenges3 (2): 183–93.
41.
JonesDavid K.EvensonKelly R.RodriguezDaniel A.AyturSemra A.2010. “Addressing Pedestrian Safety: A Content Analysis of Pedestrian Master Plans in North Carolina.” Traffic Injury Prevention11 (1): 57–65.
42.
KaiserEdward J.DaviesJohn. 1999. “What a Good Plan Should Contain: A Proposed Model.” Carolina Planning24 (2): 29–41.
43.
KangJung EunPeacockWalter G.HuseinRahmawati. 2010. “An Assessment of Coastal Zone Hazard Mitigation Plans in Texas.” Journal of Disaster Research5 (5): 520–28.
44.
KrippendorffKlaus. 2004. “Reliability in Content Analysis: Some Common Misconceptions and Recommendations.” Human Communication Research30 (3): 411–33.
45.
KrippendorffKlaus. 2011. “Agreement and Information in the Reliability of Coding.” Communication Methods and Measures5 (2): 93–112.
46.
KrippendorffKlaus. 2013. Content Analysis: An Introduction to Its Methodology, 3rd ed.Thousand Oaks, CA: Sage.
47.
LombardMatthewSnyder-DuchJenniferBrackenCheryl Campanella. 2002. “Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability.” Human Communication Research28 (4): 587–604.
48.
Manta-ConroyMariaBerkePhilip R.2004. “What Makes a Good Sustainable Development Plan? An Analysis of Factors That Influence Principles of Sustainable Development.” Environment and Planning A36 (8): 1381–96.
49.
MilesMatthew B.HubermanA. Michael. 1984. Qualitative Data Analysis: A Sourcebook of New Methods. Newbury Park, CA: Sage.
50.
MilesMatthew B.HubermanA. Michael. 1994. Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA: Sage.
51.
NelsonArthur C.FrenchSteven P.2002. “Plan Quality and Mitigating Damage from Natural Disasters: A Case Study of the Northridge Earthquake with Planning Policy Considerations.” Journal of the American Planning Association68 (2): 194–207.
52.
NortonRichard K.2005a. “Local Commitment to State-Mandated Planning in Coastal North Carolina.” Journal of Planning Education and Research25 (2): 149–71.
53.
NortonRichard K.2005b. “More and Better Local Planning: State-Mandated Local Planning in Coastal North Carolina.” Journal of the American Planning Association71 (1): 55–72.
54.
NortonRichard K.2008. “Using Content Analysis to Evaluate Local Master Plans and Zoning Codes.” Land Use Policy25 (3): 432–54.
55.
OloniluaOluponmile O.IbitayoOlurominiyi O.2011. “Toward Multihazard Mitigation: An Evaluation of FEMA-Approved Hazard Mitigation Plans under the Disaster Mitigation Act of 2000.” Journal of Emergency Management9 (1): 37–49.
56.
PrestonBenjamin L.WestawayRichard M.YuenEmma J.2011. “Climate Adaptation Planning in Practice: An Evaluation of Adaptation Plans from Three Developed Nations.” Mitigation and Adaptation Strategies for Global Change16 (4): 407–38.
ScottWilliam A.1955. “Reliability of Content Analysis: The Case of Nominal Scale Coding.” Public Opinion Quarterly19 (3): 321–25.
59.
SrivastavaRahulLaurianLucie. 2006. “Natural Hazard Mitigation in Local Comprehensive Plans: The Case of Flood, Wildfire and Drought Planning in Arizona.” Disaster Prevention and Management15 (3): 461–83.
60.
StoneBrianVargoJasonHabeebDana. 2012. “Managing Climate Change in Cities: Will Climate Action Plans Work?”Landscape and Urban Planning107 (3): 263–71.
61.
TalenEmily. 1996. “Do Plans Get Implemented? A Review of Evaluation in Planning.” Journal of Planning Literature10 (3): 248–59.
62.
TalenEmilyKnaapGerrit. 2003. “Legalizing Smart Growth: An Empirical Study of Land Use Regulation in Illinois.” Journal of Planning Education and Research22 (4): 345–59.
63.
TangZhenghong. 2008. “Evaluating Local Coastal Zone Land Use Planning Capacities in California.” Ocean & Coastal Management51 (7): 544–55.
64.
TangZhenghongBrodySamuel D.2009. “Linking Planning Theories with Factors Influencing Local Environmental-Plan Quality.” Environment and Planning B36 (6): 522–37.
65.
TangZhenghongBrodySamuel D.QuinnCourtneyChangLiangWeiTing. 2010. “Moving from Agenda to Action: Evaluating Local Climate Change Action Plans.” Journal of Environmental Planning and Management53 (1): 41–62.
66.
TangZhenghongLindellMichael K.PraterCarlaWeiTingHusseyChristopher M.2011. “Examining Local Coastal Zone Management Capacity in U.S. Pacific Coastal Counties.” Coastal Management39 (2): 105–32.
67.
TermorshuizenJolande W.PaulOpdamvan den BrinkAdri. 2007. “Incorporating Ecological Sustainability into Landscape Planning.” Landscape and Urban Planning79 (3–4): 374–84.
68.
WesmanAlexander G.1971. “Writing the Test Item.” In Educational Measurement, edited by ThorndikeRobert L.Washington, DC: American Council in Education.
69.
WheelerStephen M.2008. “State and Municipal Climate Change Plans.” Journal of the American Planning Association74 (4): 481–96.
70.
WimmerRoger D.DominickJoseph R.1983. Mass Media Research: An Introduction. Belmont, CA: Wadsworth.
