Abstract
Housing unit listing is often used in countries that do not have household or person registries to create frames for household surveys. While several studies have reported the kinds of units and areas that are at risk of overcoverage and undercoverage in such frames, none has looked at variability in the listing process. This article explores this variability by comparing two frames created by trained field staff using the same methods and materials. The overall overlap rate between the two listings is 80%. In nearly all blocks, the listers created different frames, and in more than ten percent of the blocks, the two frames did not overlap at all. In this observational study, the overlap between the two frames is particularly low in the blocks listed using the traditional (from scratch) listing method. There is also evidence that sometimes one lister visited the wrong block. The results show that the listing process can introduce variance into survey data.
References
1.
Andersen, R., Kasper, J., and Frankel, M.R. (1979). Total Survey Error. San Francisco: Jossey-Bass.
2.
Barrett, D.F., Beaghen, M., Smith, D., and Burcham, J. (2002). Census 2000 Housing Unit Coverage Study. Proceedings of the Section on Survey Research Methods: American Statistical Association, 146-151.
3.
Barrett, D. F., Beaghen, M., Smith, D., and Burcham J. (2003). Census 2000 Housing Unit Coverage Study. Technical Report Census 2000 Evaluation O.3, U.S. Census Bureau, Washington, D.C.
4.
Biemer, P.P. and Lyberg, L.E. (2003). Introduction to Survey Quality. New York: Wiley-Interscience.
5.
Central Co-ordinating Team (2010). European Social Survey Round 3 2008/2009. Final Activity Report ESS4e03.0, City University London.
6.
Childers, D.R. (1992). The 1990 Housing Unit Coverage Study. Proceedings of the Section on Survey Research Methods: American Statistical Association, 506-511.
7.
Childers, D.R. (1993). Coverage of Housing in the 1990 Decennial Census. Proceedings of the Section on Survey Research Methods: American Statistical Association, 635-640.
8.
Cohen, J. (1969). Weighted Kappa: Nominal Scale Agreement with Provision for Scaled Disagreement or Partial Credit. Psychological Bulletin, 70, 213-220.
9.
Eckman, S. (2010). Errors in Housing Unit Listing and Their Effects on Survey Estimates. Ph. D. thesis, University of Maryland.
10.
Eckman, S. and English, N. (2012). Creating Housing Unit Frames from Address Databases: Geocoding Precision and Net Coverage Rates. Field Methods, 24, 399-408.
11.
Eckman, S. and Kreuter, F. (2011). Confirmation Bias in Housing Unit Listing. Public Opinion Quarterly, 75, 139-150.
12.
Fellegi, I.P. (1964). Response Variance and Its Estimation. Journal of the American Statistical Association, 59, 1016-1041.
13.
Groves, R.M. (1989). Survey Errors and Survey Costs. Hoboken, N.J: John Wiley and Sons.
14.
Harter, R., Eckman, S., English, N., and O’Muircheartaigh, C. (2010). Applied Sampling for Large-Scale Multi-Stage Area Probability Designs. Handbook of Survey Research, P.V. Marsden and J. Wright (eds). (Second ed). Bingley: Emerald, 169-197.
15.
Herzog, T.N., Scheuren, F.J., and Winkler, W.E. (2007). Data Quality and Record Linkage Techniques. New York: Springer.
16.
Jowell, R. and the Central Co-ordinating Team (2003). European Social Survey 2002/2003. Technical Report ESS1e06.1, London City University.
17.
Jowell, R. and the Central Co-ordinating Team (2005). European Social Survey Round 2 2004/2005. Technical Report ESS2e03.1, London City University.
18.
Jowell, R. and the Central Co-ordinating Team (2007). European Social Survey Round 3 2006/2007. Technical Report ESS3e03.2, London City University.
19.
Kwiat, A. (2009). Examining Blocks with Lister Error in Area Listing. Proceedings of the Section on Survey Research Methods: American Statistical Association, 2546-2557.
20.
Lepkowski, J.M., Mosher, W.D., Davis, K., Groves, R.M., and Hoewyk, J.V. (2010). The 2006- 2010 National Survey of Family Growth: Sample Design and Analysis of a Continuous Survey. Vital Health Statistics, 2(150).
21.
Lessler, J.T. and Kalsbeek, W.D. (1992). Nonsampling Error in Surveys. Hoboken, N.J.: John Wiley and Sons.
22.
Listi, G.A., Manhein, M.H., and Leitner, M. (2007). Use of the Global Positioning System in the Field Recovery of Scattered Human Remains. Journal of Forensic Science, 52, 11.
23.
Loudermilk, C.L. and Li, M. (2009). A National Evaluation of Coverage for a Sampling Frame Based on the Master Address File (MAF). Proceedings of the Section on Survey Research Methods: American Statistical Association, 1721-1734.
24.
Manheimer, D. and Hyman, H. (1949). Interviewer Performance in Area Sampling. Public Opinion Quarterly, 13, 83-92.
25.
Montaquila, J.M., Brick, J.M., and Curtin, L.R. (2010). Statistical and Practical Issues in the Design of a National Probability Sample of Births for the Vanguard Study of the National Children’s Study. Statistics in Medicine, 29, 1368-1376.
26.
Morton, K.B., Hunter, S.R., Chromy, J.R., and Martin, P.C. (2006). Population Coverage in the National Survey of Drug Use and Health. Proceedings of the Section on Survey Research Methods: American Statistical Association, 3441-3446.
27.
O’Muircheartaigh, C. and Campanelli, P. (1999). A Multilevel Exploration of the Role of Interviewers in Survey Non-Response. Journal of the Royal Statistical Society. Series A (Statistics in Society), 437-446.
28.
O’Muircheartaigh, C. and Marckward, A.M. (1980). An Assessment of the Reliability of World Fertility Study Data. Proceedings of the World Fertility Survey Conference, 3, 305-379.
29.
O’Muircheartaigh, C.A., English, E.M., and Eckman, S. (2007). Predicting the Relative Quality of Alternative Sampling Frames. Proceedings of the Section on Survey Research Methods: American Statistical Association, 551-574.
30.
Sando, T., Mussa, R., Sobanjo, J., and Spainhour, L. (2005). Quantification of the Accuracy of Low Priced GPS Receivers for Crash Location. Journal of the Transportation Research Forum, 44, 19-32.
31.
Schnell, R. and Kreuter, F. (2005). Separating Interviewer and Sampling-Point Effects. Journal of Official Statistics, 21, 389-410.
32.
Statistics Canada (2008). Methodology of the Canadian Labour Force Survey. Technical Report 71-526-X, Statistics Canada.
33.
U.S. Census Bureau (1993). Programs to Improve Coverage in the 1990 Census. Technical report. 1990 CPH-E-3.
34.
U.S. Census Bureau (2006). Technical Paper 66: Design and Methodology, Current Population Survey. Technical report.
35.
U.S. Census Bureau (2009). Design and Methodology American Community Survey. Technical report, U.S. Census Bureau, Washington, D.C.
36.
U.S. Census Bureau; American Community Survey (2008). Selected Housing Characteristics: 2008, 1-Year Estimates. Generated by author using American FactFinder ,http://factfinder.census.gov . (Accessed 28 December 2010).
37.
West, B. and Olson, K. (2010). How Much of Interviewer Variance is Really Nonresponse Error Variance?, Public Opinion Quarterly, 74, 1027-1045.