During the last twenty years, more than forty-five publications have sought to measure and evaluate the quality of plans using content analysis methods. We examine reasons for this growth in the literature and its contributions and limitations. We also examine whether the research methods described in these publications conform to recommended practices in the methodological literature on content analysis to determine whether plan quality researchers are likely to be generating reliable and reproducible plan quality data. We provide seven recommendations plan quality researchers can follow to address these weaknesses and improve the reliability and reproducibility of their data.
AlexanderE. R.FaludiA.1989. “Planning and Planning Implementation: Notes on Evaluation Criteria.” Environment and Planning B: Planning and Design16 (2): 127–40.
2.
AltshulerA. A.1967. The City Planning Process: A Political Analysis. Ithaca, NY: Cornell University Press.
3.
AyturSemra A.SatinskySara B.EvensonKelly R.RodriguezDaniel A.2011. “Pedestrian and Bicycle Planning in Rural Communities: Tools for Active Living.” Family & Community Health34 (2): 173–81.
4.
BaerWilliam C.1997. “General Plan Evaluation Criteria: An Approach to Making Better Plans.” Journal of the American Planning Association63 (3): 329–44.
5.
BakerIngridPetersonAnnBrownGregMcAlpineClive. 2012. “Local Government Response to the Impacts of Climate Change: An Evaluation of Local Climate Adaptation Plans.” Landscape and Urban Planning107 (2): 127–36.
6.
BassettEllenShandasVivek. 2010. “Innovation and Climate Action Planning—Perspectives from Municipal Plans.” Journal of the American Planning Association76 (4): 435–50.
7.
BaynhamMaggieStevensMark R.2014. “Are We Planning Effectively for Climate Change? An Evaluation of Official Community Plans in British Columbia.” Journal of Environmental Planning and Management57 (4): 557–87.
8.
BerelsonB.1971. Content Analysis in Communications Research. New York: Hafner.
9.
BerkePhilip R.1994. “Evaluating Environmental Plan Quality: The Case of Planning for Sustainable Development in New Zealand.” Journal of Environmental Planning and Management37 (2): 155–70.
10.
BerkeP. R.BackhurstM.DayM.EricksenN.LaurianL.CrawfordJ.DixonJ.2006. “What Makes Plan Implementation Successful? An Evaluation of Local Plans and Implementation Practices in New Zealand.” Environment and Planning B: Planning and Design33:581–600.
11.
BerkePhilip R.CrawfordJanDixonJennyEricksenNeil. 1999. “Do Cooperative Environmental Planning Mandates Produce Good Plans? Empirical Results from the New Zealand Experience.” Environment and Planning B: Planning and Design26 (5): 643–64.
12.
BerkePhilip R.DixonJennyEricksenNeil. 1997. “Coercive and Cooperative Intergovernmental Mandates: A Comparative Analysis of Florida and New Zealand Environmental Plans.” Environment and Planning B: Planning and Design24 (3): 451–68.
13.
BerkePhilip R.EricksenNeilCrawfordJanDixonJenny. 2002. “Planning for Indigenous People: Human Rights and Environmental Protection in New Zealand.” Journal of Planning Education and Research22 (2): 115–34.
14.
BerkePhilipFrenchSteven P.1994. “The Influence of State Planning Mandates on Local Plan Quality.” Journal of Planning Education and Research13 (4): 237–50.
15.
BerkePhilipGodschalkDavid. 2009. “Searching for the Good Plan: A Meta-analysis of Plan Quality Studies.” Journal of Planning Literature23 (3): 227–40.
16.
BerkePhilip R.GodschalkDavid R.KaiserEdward J., with RodriguezDaniel A.2006. Urban Land Use Planning, 5th ed.Urbana: University of Illinois Press.
17.
BerkePhilip R.Manta-ConroyMaria. 2000. “Are We Planning for Sustainable Development? An Evaluation of 30 Comprehensive Plans.” Journal of the American Planning Association66 (1): 21–33.
18.
BerkePhilip R.RoenigkDale J.KaiserEdward J.BurbyRaymond. 1996. “Enhancing Plan Quality: Evaluating the Role of State Planning Mandates for Natural Hazard Mitigation.” Journal of Environmental Planning & Management39 (1): 79–96.
19.
BerkePhilipSmithGavinLylesWard. 2012. “Planning for Resiliency: Evaluation of State Hazard Mitigation Plans under the Disaster Mitigation Act.” Natural Hazards Review13 (2): 139–49.
20.
BerkePhilipSpurlockDanielleHessGeorgeBandLarry. 2013. “Local Comprehensive Plan Quality and Regional Ecosystem Protection: The Case of the Jordan Lake Watershed, North Carolina, USA.” Land Use Policy31: 450–59.
21.
BoswellMichael R.GreveAdrienne I.SealeTammy L.2013. Local Climate Action Planning. Washington, DC: Island Press.
22.
BrodySamuel D.2003a. “Are We Learning to Make Better Plans? A Longitudinal Analysis of Plan Quality Associated with Natural Hazards.” Journal of Planning Education and Research23 (2): 191–201.
23.
BrodySamuel D.2003b. “Implementing the Principles of Ecosystem Management through Local Land Use Planning.” Population and Environment24 (6): 511–40.
24.
BrodySamuel D.CarrascoVirginiaHighfieldWesley E.2006. “Measuring the Adoption of Local Sprawl Reduction Planning Policies in Florida.” Journal of Planning Education and Research25 (3): 294–310.
25.
BrodySamuel D.HighfieldWesley E.2005. “Does Planning Work? Testing the Implementation of Local Environmental Planning in Florida.” Journal of the American Planning Association71 (2): 159–75.
26.
BrodySamuel D.HighfieldWesCarrascoVirginia. 2004. “Measuring the Collective Planning Capabilities of Local Jurisdictions to Manage Ecological Systems in Southern Florida.” Landscape and Urban Planning69 (1): 33–50.
27.
BunnellGeneJepsonEdwad J.Jr.2011. “The Effect of Mandated Planning on Plan Quality.” Journal of the American Planning Association77 (4): 338–53.
28.
BurbyRaymond J.2003. “Making Plans that Matter: Citizen Involvement and Government Action.” Journal of the American Planning Association69 (1): 33–49.
29.
BurbyRaymond J.DaltonLinda C.1994. “Plans Can Matter! The Role of Land Use Plans and State Planning Mandates in Limiting the Development of Hazardous Areas.” Public Administration Review54 (3): 229–38.
30.
BurbyRaymond J.MayPeter J., with BerkePhilip R.DaltonLinda C.FrenchSteven P.KaiserEdward J.1997. Making Governments Plan: State Experiments in Managing Land Use. Baltimore: Johns Hopkins University Press.
31.
DaltonLinda C.BurbyRaymond J.1994. “Mandates, Plans, and Planners: Building Local Commitment to Development Management.” Journal of the American Planning Association60 (4): 444–61.
32.
DavisBraxton C.2004. “Regional Planning in the US Coastal Zone: A Comparative Analysis of 15 Special Area Plans.” Ocean and Coastal Management47 (1–2): 79–94.
33.
Department of Homeland Security, Office of the Inspector General. 2012. Survey of Hazard Mitigation Planning OIG-12-109. Washington, DC: Office of the Inspector General.
34.
DeyleRobert E.SmithRichard A.1998. “Local Government Compliance with State Plan Mandates: The Effects of State Implementation in Florida.” Journal of the American Planning Association64 (4): 457–69.
35.
DeyleRobert E.ChapinTimothy S.BakerEarl J.2008. “The Proof of the Planning Is in the Platting: An Evaluation of Florida’s Hurricane Exposure Mitigation Planning Mandate.” Journal of the American Planning Association74 (3): 349–70.
36.
DillmanD. A.SmythJ. D.ChristiaL. M.2008. Internet, Mail and Mixed-Mode Surveys: The Tailored Design Method. New York: John Wiley and Sons.
37.
EdwardsMaryHainesAnna. 2007. “Evaluating Smart Growth: Implications for Small Communities.” Journal of Planning Education and Research27 (1): 49–64.
Evans-CowleyJennifer S.GoughMeghan Z.2008. “Evaluating Environmental Protection in Post-Hurricane Katrina Plans in Mississippi.” Journal of Environmental Planning and Management51 (3): 399–19.
40.
Evans-CowleyJennifer S.GoughMeghan Z.2009. “Evaluating New Urbanist Plans in Post-Katrina Mississippi.” Journal of Urban Design14 (4): 439–61.
41.
EvensonKelly R.SatinskySara B.RodriguezDaniel A.AyturSemra A.2012. “Exploring a Public Health Perspective on Pedestrian Planning.” Health Promotion Practice13 (2): 204–13.
42.
FEMA (Federal Emergency Management Agency). 2008. Local Multi-Hazard Mitigation Planning Guidance Under the Disaster Mitigation Act of 2000 (Blue Book). Washington, DC: FEMA
43.
FEMA (Federal Emergency Management Agency). 2011. Local Mitigation Plan Review Guide. Washington, DC: FEMA.
44.
FriedmannJohn. 1993. “Toward a non-Euclidean Mode of Planning.” Journal of the American Planning Association59 (4): 482–85.
45.
GodschalkDavid R.BeatleyTimothyBerkePhilipBrowerDavid J.KaiserEdward J.1999. Natural Hazard Mitigation: Recasting Disaster Policy and Planning. Washington, DC: Island Press.
46.
HaminElisabeth M.2011. “Integrating Adaptation and Mitigation in Local Climate Change Planning.” In Climate Change and Land Policies, edited by IngramGregory K.HongYu-Hung. Cambridge: Lincoln Institute of Land Policy.
47.
HochCharles. 2007. “How Plan Mandates Work: Affordable Housing in Illinois.” Journal of the American Planning Association73 (1): 86–99.
48.
HopkinsLewis D.2001. Urban Development: The Logic of Making Plans. Washington, DC: Island Press.
49.
HorneyJennifer A.NaimiAshley I.LylesWardSimonMattSalvesenDavidBerkePhilip. 2012. “Assessing the Relationship between Hazard Mitigation Plan Quality and Rural Status in a Cohort of 57 Counties from 3 States in the Southeastern U.S.” Challenges3 (2): 183–93.
JonesDavid K.EvensonKelly R.RodriguezDaniel A.AyturSemra A.2010. “Addressing Pedestrian Safety: A Content Analysis of Pedestrian Master Plans in North Carolina.” Traffic Injury Prevention11 (1): 57–65.
52.
KaiserEdward J.GodschalkDavid R.1995. “Twentieth Century Land Use Planning: A Stalwart Family Tree.” Journal of the American Planning Association61 (3): 365–85.
53.
KangJung EunPeacockWalter G.HuseinRahmawati. 2010. “An Assessment of Coastal Zone Hazard Mitigation Plans in Texas.” Journal of Disaster Research5 (5): 520–28.
54.
KassarjianHarold, H.1997. “Content Analysis in Consumer Research.” Journal of Consumer Research4 (1): 8–18.
55.
KolbeRichard H.BurnettMelissa S.1991. “Content-Analysis Research: An Examination of Applications with Directives for Improving Research Reliability and Objectivity.” Journal of Consumer Research18 (2): 243–50.
56.
KrippendorffKlaus. 2004. Content Analysis: An Introduction to Its Methodology, 2nd ed.Thousand Oaks, CA: Sage.
57.
LaurianL.CrawfordJ.DayM.KouwenhovenP.MasonG.EricksenN.BeattieL.2010. “Evaluating the Outcomes of Plans: Theory, Practice, and Methodology.” Environment and Planning B, Planning & Design37 (4): 740.
58.
LaurianL.DayM.BackhurstM.BerkeP.CrawfordJ.DixonJ.ChapmanS.2004. “What Drives Plan Implementation? Plans, Planning Agencies and Developers.” Journal of Environmental Planning and Management47 (4): 555–77.
59.
LombardMatthewSnyder-DuchJenniferBrackenCheryl Campanella. 2002. “Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability.” Human Communication Research28 (4): 587–604.
60.
Manta-ConroyMariaBerkePhilip R.2004. “What Makes a Good Sustainable Development Plan? An Analysis of Factors That Influence Principles of Sustainable Development.” Environment and Planning A36 (8): 1381–96.
61.
MastopH.FaludiA.1997. “Evaluation of Strategic Plans: The Performance Principle.” Environment and Planning B: Planning and Design24:815–32.
62.
MilesMatthew B.HubermanA. Michael. 1984. Qualitative Data Analysis: A Sourcebook of New Methods. Newbury Park, CA: Sage.
63.
MilesMatthew B.HubermanA. Michael. 1994. Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA: Sage.
64.
NelsonArthur C.FrenchSteven P.2002. “Plan Quality and Mitigating Damage from Natural Disasters: A Case Study of the Northridge Earthquake with Planning Policy Considerations.” Journal of the American Planning Association68 (2): 194–207.
65.
NeumanMichael. 1998. “Does Planning Need the Plan?”Journal of the American Planning Association64 (2): 208–20.
66.
NeumanMichael. 2003. “Do Plans and Zoning Matter?”Planning, December28–31.
67.
NortonRichard K.2005a. “Local Commitment to State-Mandated Planning in Coastal North Carolina.” Journal of Planning Education and Research25 (2): 149–71.
68.
NortonRichard K.2005b. “More and Better Local Planning: State-Mandated Local Planning in Coastal North Carolina.” Journal of the American Planning Association71 (1): 55–72.
69.
NortonRichard K.2008. “Using Content Analysis to Evaluate Local Master Plans and Zoning Codes.” Land Use Policy25 (3): 432–54.
70.
OhmBrian W.2013. Wisconsin Land Use and Planning Law. Madison: University of Wisconsin Law School.
71.
OliveriaV.PinhoP.2009. “Evaluating Plans, Processes and Results.” Planning Theory and Practice10 (1): 35–63.
72.
OloniluaOluponmile O.IbitayoOlurominiyi O.2011. “Toward Multihazard Mitigation: An Evaluation of FEMA-Approved Hazard Mitigation Plans under the Disaster Mitigation Act of 2000.” Journal of Emergency Management9 (1): 37–49.
73.
O’TooleRandall. 2007. The Best-Laid Plans: How Government Planning Harms Your Quality of Life, Your Pocketbook, and Your Future. Washington, DC: Cato Institute.
74.
PrestonBenjamin L.WestawayRichard M.YuenEmma J.2011. “Climate Adaptation Planning in Practice: An Evaluation of Adaptation Plans from Three Developed Nations.” Mitigation and Adaptation Strategies for Global Change16 (4): 407–38.
RyanBrett D.2011. “Reading through a Plan: A Visual Interpretation of What Plans Mean and How They Innovate.” Journal of the American Planning Association77 (4): 309–27.
77.
SingletonRoyce A.StraitsBruce C.2005. Approaches to Social Research. New York: Oxford University Press.
78.
SmithGavinLylesWardBerkePhilip. 2013. “The Role of Hazard Mitigation Planning in Building Local Capacity and Commitment: A Tale of Six States.” International Journal of Mass Emergencies and Disasters32 (2): 178–203.
79.
SrivastavaRahulLaurianLucie. 2006. “Natural Hazard Mitigation in Local Comprehensive Plans: The Case of Flood, Wildfire and Drought Planning in Arizona.” Disaster Prevention and Management15 (3): 461–83.
80.
SteelmanToddi A.HessGeorge R.2009. “Effective Protection of Open Space: Does Planning Matter?”Environmental Management44:93–104.
81.
StevensMark R.HanschkaSteve. 2014. “Multilevel Governance of Flood Hazards: Municipal Flood Bylaws in British Columbia, Canada.” Natural Hazards Review15 (1): 74–87.
82.
StevensMark R.LylesWardBerkePhilip R.2014. “Measuring and Reporting Intercoder Reliability in Plan Quality Evaluation Research.” Journal of Planning Education and Research34 (1): 77–93.
83.
StoneBrianVargoJasonHabeebDana. 2012. “Managing Climate Change in Cities: Will Climate Action Plans Work?”Landscape and Urban Planning107 (3): 263–71.
84.
TalenEmily. 1997. “Success, Failure and Conformance: An Alternative Approach to Planning Evaluation.” Environment and Planning B: Planning and Design24:573–87.
85.
TalenEmilyKnaapGerrit. 2003. “Legalizing Smart Growth: An Empirical Study of Land Use Regulation in Illinois.” Journal of Planning Education and Research22 (4): 345–59.
86.
TangZhenghong. 2008. “Evaluating Local Coastal Zone Land Use Planning Capacities in California.” Ocean & Coastal Management51 (7): 544–55.
87.
TangZhenghongBrodySamuel D.2009. “Linking Planning Theories with Factors Influencing Local Environmental-Plan Quality.” Environment and Planning B36 (6): 522–37.
88.
TangZhenghongBrodySamuel D.QuinnCourtneyChangLiangWeiTing. 2010. “Moving from Agenda to Action: Evaluating Local Climate Change Action Plans.” Journal of Environmental Planning and Management53 (1): 41–62.
89.
TangZhenghongLindellMichael K.PraterCarlaWeiTingHusseyChristopher M.2011. “Examining Local Coastal Zone Management Capacity in U.S. Pacific Coastal Counties.” Coastal Management39 (2): 105–32.
90.
TermorshuizenJolande W.OpdamPaulvan den BrinkAdri. 2007. “Incorporating Ecological Sustainability into Landscape Planning.” Landscape and Urban Planning79 (3–4): 374–84.
91.
University of North Carolina at Chapel Hill. n.d. “Beyond the Basics: Best Practices in Local Mitigation Planning.” Mitigationguide.org (accessed March 3, 2014).
92.
WheelerStephen M.2008. “State and Municipal Climate Change Plans.” Journal of the American Planning Association74 (4): 481–96.
