Studies of governments and local organizations using survey data have played a critical role in the development of urban studies and related disciplines. However, missing data pose a daunting challenge for this research. This article seeks to raise awareness about the treatment of missing data in urban studies research by comparing and evaluating three commonly used approaches to deal with missing data—listwise deletion, single imputation, and multiple imputation. Comparative analyses illustrate the relative performance of these approaches using the second-generation Integrated City Sustainability Database (ICSD). The results demonstrate the benefit of using an approach to missing data based on multiple imputation, using a theoretically informed and statistically supported set of predictor variables to develop a more complete sample that is free of issues raised by nonresponse in survey data. The results confirm the usefulness of the ICSD in the study of environmental and sustainability and other policy in U.S. cities. We conclude with a discussion of results and provide a set of recommendations for urban researcher scholars.
AbayomiKobiGelmanAndrewLevyMarc. 2008. “Diagnostics for Multivariate Imputations.”Journal of the Royal Statistical Society: Series C (Applied Statistics)57 (3): 273–91.
2.
AllenTammy D.EbyLillian T.LentzElizabeth. 2006. “Mentorship Behaviors and Mentorship Quality Associated with Formal Mentoring Programs: Closing the Gap Between Research and Practice.”Journal of Applied Psychology91 (3): 567–78.
3.
AllisonPaul D.2002. “Missing Data: Quantitative Applications in the Social Sciences.”British Journal of Mathematical and Statistical Psychology55 (1): 193–96.
4.
AndridgeRebecca R.LittleRoderick J. A.2010. “A Review of Hot Deck Imputation for Survey Non-response.”International Statistical Review78 (1): 40–64.
5.
BaeJungahFeiockRichard. 2013. “Forms of Government and Climate Change Policies in US Cities.”Urban Studies50 (4): 776–88.
6.
BetsillMichele M.2001. “Mitigating Climate Change in US Cities: Opportunities and Obstacles.”Local Environment6 (4): 393–406.
7.
BodnerTodd E.2008. “What Improves with Increased Missing Data Imputations?”Structural Equation Modeling15 (4): 651–75.
8.
BrehmJohn O.1993. “The phantom respondents: Opinion surveys and political representation.”Ann Arbor, MI: University of Michigan Press.
9.
BrickJ. MichaelKaltonGraham. 1996. “Handling Missing Data in Survey Research.”Statistical Methods in Medical Research5 (3): 215–38.
10.
DeMarisAlfred. 2014. “Combating Unmeasured Confounding in Cross-Sectional Studies: Evaluating Instrumental-Variable and Heckman Selection Models.”Psychological Methods19 (3): 380–97.
11.
DondersA. Rogier T.van der HeijdenGeert J. M. G.StijnenTheoMoonsKarel G. M.2006. “Review: A Gentle Introduction to Imputation of Missing Values.”Journal of Clinical Epidemiology59 (10): 1087–91.
12.
DowneyRonald G.KingCraig V.1998. “Missing Data in Likert Ratings: A Comparison of Replacement Methods.”The Journal of General Psychology125 (2): 175–91.
13.
FeiockRichard C.BaeJungah. 2011. “Politics, Institutions and Entrepreneurship: City Decisions Leading to Inventoried GHG Emissions.”Carbon Management2 (4): 443–53.
14.
FeiockRichard C.KrauseRachel M.HawkinsChristopher V.CurleyCali. 2014. “The Integrated City Sustainability Database.”Urban Affairs Review50 (4): 577–89.
15.
FeiockRichard C.LeeIn WonParkHyung JunLeeKeon-Hyung. 2010. “Collaboration Networks Among Local Elected Officials: Information, Commitment, and Risk Aversion.”Urban Affairs Review46 (2): 241–62.
16.
FoxJames AlanSwattMarc L.2009. “Multiple Imputation of the Supplementary Homicide Reports, 1976–2005.”Journal of Quantitative Criminology25 (1): 51–77.
17.
GallimoreJonathan M.BrownBarbara B.WernerCarol M.2011. “Walking Routes to School in New Urban and Suburban Neighborhoods: An Environmental Walkability Analysis of Blocks and Routes.”Journal of Environmental Psychology31 (2): 184–91.
18.
GrahamJohn W.OlchowskiAllison E.GilreathTamika D.2007. “How Many Imputations Are Really Needed? Some Practical Clarifications of Multiple Imputation Theory.”Prevention Science8 (3): 206–13.
19.
HawkinsChristopher V.KrauseRachelFeiockRichardCurleyCali. 2016. “Making Meaningful Commitments: Accounting for Variation in Cities’ Investments of Staff and Fiscal Resources to Sustainability.”Urban Studies53 (9): 1902–24.
20.
JohnsonDavid R.YoungRebekah. 2011. “Toward Best Practices in Analyzing Datasets with Missing Data: Comparisons and Recommendations.”Journal of Marriage and Family73 (5): 926–45.
21.
JonesMichael P.1996. “Indicator and Stratification Methods for Missing Explanatory Variables in Multiple Linear Regression.”Journal of the American Statistical Association91 (433): 222–30.
22.
KingGaryHonakerJamesJosephAnneScheveKenneth. 2001. “Analyzing Incomplete Political Science Data: An Alternative Algorithm for Multiple Imputation.”American Political Science Review95 (1): 49–69.
23.
KongAugustineLiuJun S.WongWing H.1994. “Sequential Imputations and Bayesian Missing Data Problems.”Journal of the American Statistical Association89 (425): 278–88.
24.
KrauseRachel M.2010. “Policy Innovation, Intergovernmental Relations, and the Adoption of Climate Protection Initiatives by U.S. Cities.”Journal of Urban Affairs33 (1): 45–60.
25.
KrauseRachel M.2012. “Political Decision-Making and the Local Provision of Public Goods: The Case of Municipal Climate Protection in the US.”Urban Studies49 (11): 2399–417.
26.
KrauseRachel M.2013. “The Motivations Behind Municipal Climate Engagement: An Empirical Assessment of How Local Objectives Shape the Production of a Public Good.”Cityscape15:125–41.
27.
KrauseRachel M.FeiockRichardHawkinsChristopher. 2016. “The Administrative Organization of Sustainability Within Local Government.”Journal of Public Administration Research and Theory26 (1): 113–27.
28.
LittleRoderick J. A.1988a. “A Test of Missing Completely at Random for Multivariate Data with Missing Values.”Journal of the American Statistical Association83 (404): 1198–1202.
29.
LittleRoderick J. A.1988b. “Missing-Data Adjustments in Large Surveys.”Journal of Business & Economic Statistics6 (3): 287–96.
30.
LittleRoderick J. A.2016. “Selection Model (Missing Data).”Wiley StatsRef: Statistics Reference Online. 1–5. doi:10.1002/9781118445112.stat07871
31.
LittleRoderick J. A.RubinDonald B.2014. Statistical Analysis with Missing Data. Hoboken: John Wiley & Sons.
32.
LubellMarkFeiockRichardHandySusan. 2009. “City Adoption of Environmentally Sustainable Policies in California’s Central Valley.”Journal of the American Planning Association75 (3): 293–308.
33.
MiyamaErikoManagiShunsuke. 2014. “Global Environmental Emissions Estimate: Application of Multiple Imputation.”Environmental Economics and Policy Studies16 (2): 115–35.
34.
MyersTeresa A.2011. “Goodbye, Listwise Deletion: Presenting Hot Deck Imputation as an Easy and Effective Tool for Handling Missing Data.”Communication Methods and Measures5 (4): 297–310.
35.
NosekBrian A.AlterGeorgeBanksGeorge C.BorsboomD.BowmanS. D.BrecklerS. J.BuckS.et al. 2015. “Promoting an Open Research Culture.”Science348 (6242): 1422–25.
36.
ParkJoohyungHaSejin. 2012. “Understanding Pro-environmental Behavior: A Comparison of Sustainable Consumers and Apathetic Consumers.”International Journal of Retail & Distribution Management40 (5): 388–403.
37.
PengChao-Ying JoanneHarwellMichaelLiouShow-MannEhmanLee H.2006. “Advances in Missing Data Methods and Implications for Educational Research.” In Real Data Analysis, edited by SawilowskyShlomo S.31–78. Charlotte, NC: Information Age Publishing.
38.
PortneyKent E.2003. Taking Sustainable Cities Seriously: Economic Development, the Environment, and Quality of Life in American Cities. Vol. 67. Cambridge: MIT Press.
39.
RoystonPatrickWhiteIan R.2011. “Multiple Imputation by Chained Equations (MICE): Implementation in Stata.”Journal of Statistical Software45 (4): 1–20.
40.
RubinDonald B.1978. “Multiple imputations in sample surveys-a phenomenological Bayesian approach to nonresponse.”Proceedings of the survey research methods section of the American Statistical Association1: 20–34.
41.
RubinDonald B.1987. “Multiple Imputationfor Nonresponse in Surveys.”New York: Wiley
42.
RubinDonald B.1996. “Multiple Imputation After 18+ Years.”Journal of the American Statistical Association91 (434): 473–89.
43.
RyffCarol D.KeyesCorey Lee M.1995. “The Structure of Psychological Well-Being Revisited.”Journal of Personality and Social Psychology69 (4): 719–27.
SchaferJoseph L.GrahamJohn W.2002. “Missing Data: Our View of the State of the Art.”Psychological Methods7 (2): 147–77.
47.
SchneiderTapio. 2001. “Analysis of Incomplete Climate Data: Estimation of Mean Values and Covariance Matrices and Imputation of Missing Values.”Journal of Climate14 (5): 853–71.
48.
Van der HeijdenGeert J. M. G.DondersA. Rogier T.StijnenTheoMoonsKarel G. M.2006. “Imputation of Missing Values Is Superior to Complete Case Analysis and the Missing-Indicator Method in Multivariable Diagnostic Research: A Clinical Example.”Journal of Clinical Epidemiology59 (10): 1102–1109.
49.
WhiteIan R.RoystonPatrickWoodAngela M.2011. “Multiple Imputation Using Chained Equations: Issues and Guidance for Practice.”Statistics in Medicine30 (4): 377–99.
50.
WhiteIan R.DanielRhianRoystonPatrick. 2010. “Avoiding Bias due to Perfect Prediction in Multiple Imputation of Incomplete Categorical Variables.”Computational Statistics & Data Analysis54 (10): 2267–75.
51.
YoungRebekahJohnsonDavid R.2010. “Imputing the Missing Y’s: Implications for Survey Producers and Survey Users.” In Proceedings of the AAPOR Conference Abstracts, 6242–48.
52.
ZahranSammyGroverHimanshuBrodySamuel D.VedlitzArnold. 2008. “Risk, Stress, and Capacity: Explaining Metropolitan Commitment to Climate Protection.”Urban Affairs Review43 (4): 447–74.
53.
ZhangPaul. 2003. “Multiple Imputation: Theory and Method.”International Statistical Review71 (3): 581–92.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
