A Study of Interviewer Compliance in 2013 and 2014 Census Test Adaptive Designs

Abstract

Researchers are interested in the effectiveness of adaptive and responsive survey designs that monitor and respond to data using tailored or targeted interventions. These designs often require adherence to protocols, which can be difficult when surveys allow in-person interviewers flexibility in managing cases. This article describes examples of interviewer noncompliance and compliance in adaptive design experiments that occurred in two United States decennial census tests. The two studies tested adaptive procedures including having interviewers work prioritized cases and substitute face-to-face attempts with telephone calls. When to perform such procedures was communicated to interviewers via case management systems that necessitated twice-daily transmissions of data. We discuss reasons when noncompliance may occur and ways to improve compliance.

Keywords

Computer-assisted personal interviewing decennial census

References

Biemer, P., P. Chen, and K. Wang. 2013. “Using Level-of-Effort Paradata in Non-Response Adjustments with Application to Field Surveys.” Journal of the Royal Statistical Society Series A 176(1): 147–168. Doi: http://dx.doi.org/10.1111/j.1467-985X.2012.01058.x.

Blumerman, L., E. Moffett, M. Bentley, T. Boone, and M. Chapin. 2015. “2020 Census Operational Plan Overview and Operational Areas.” Presentation to the Census Bureau’s National Advisory Committee. October 8, 2015.

Campanelli, P., P. Sturgis, and S. Purdon. 1997. “Can You Hear Me Knocking? An Investigation into the Impact of Interviewers on Survey Response Rates. London, GB, National Centre for Social Research.

Coffey, S. 2013. “Implementing Adaptive Design for the National Survey of College Graduates.” Article presented to FedCASIC, Suitland, Maryland, March 20, 2013.

Durrant, G. and F. Steele. 2009. “Multilevel Modelling of Refusal and Non-Contact in Household Surveys: Evidence from Six UK Government Surveys.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 172(2): 361–381. Doi: http://dx.doi.org/10.1111/j.1467-985X.2008.00565.x.

Fowler, F. and T. Mangione. 1988. Standardized Survey Interviewing: Minimizing Interviewer-Related Error. Sage Publications: Newbury Park, CA.

Groves, R. and S. Heeringa. 2006. “Responsive Design for Household Surveys: Tools for Actively Controlling Survey Errors and Costs.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 169: 439–457. Doi: http://dx.doi.org/10.1111/j.1467-985X.2006.00423.x.

Groves, R. and K. McGonagle. 2001. “A Theory-Guided Interviewer Training Protocol Regarding Survey Participation.” Journal of Official Statistics 17(2): 249–265.

Hackman, J. and G. Oldham. 1976. “Motivation through the Design of Work: Test of a Theory.” Organizational Behavior and Human Performance 16: 250–279. Doi: http://dx.doi.org/10.1016/0030-5073(76)90016-7.

10.

Kawamoto, K., C. Houlihan, E. Balas, and D. Lobach. 2005. “Improving Clinical Practice Using Clinical Decision Support Systems: A Systematic Review of Trials to Identify Features Critical to Success.” British Medical Journal 330(7494): 765. Doi: http://dx.doi.org/10.1136/bmj.38398.500764.8F.

11.

Kirgis, N. and J. Lepkowski. 2013. “Design and Management Strategies for Paradata-Driven Responsive design: Illustrations from the 2006–2010 National Survey of Family Growth.” In Improving Surveys with Paradata: Analytic Uses of Process Information, edited by F. Kreuter, 121–144. Hoboken, New Jersey: John Wiley and Sons.

12.

Kreuter, F., A. Mercer, and W. Hicks. 2014. “Increasing Fieldwork Efficiency through Prespecified Appointments.” Journal of Survey Statistics and Methodology 2(2): 210–223. Doi: http://dx.doi.org/10.1093/jssam/smu005.

13.

Laflamme, F. and H. St-Jean. 2011. “Highlights from the First Two Pilots of Responsive Collection Design for CATI Surveys.” In Proceedings of the Joint Statistical Meetings, American Statistical Association. Available at: https://www.amstat.org/sections/srms/proceedings/y2011/Files/301087_66138.pdf (accessed March 2016).

14.

Luiten, A. and B. Schouten. 2013. “Tailored Fieldwork Design to Increase Representative Household Survey Response: An Experiment in the Survey of Consumer Satisfaction.” Journal of the Royal Statistical Society, Series A 176(1): 169–189. Doi: http://dx.doi.org/10.1111/j.1467-985X.2012.01080.x.

15.

Morton-Williams, J. 1993. Interviewer Approaches. England: Dartmouth Publishing Company Limited.

16.

O’Muircheartaigh, C. and P. Campanelli. 1999. “A Multilevel Exploration of the Role of Interviewers in Survey Non-Response.” Journal of the Royal Statistical Society, Series A 162(3): 437–446. Doi: http://dx.doi.org/10.1111/1467-985X.00147.

17.

Peytchev, A., S. Riley, J. Rosen, J. Murphy, and M. Lindblad. 2010. “Reduction of Nonrespose Bias in Surveys through Case Prioritization.” Survey Research Methods 4(1): 21–29. Doi: http://dx.doi.org/10.18148/srm/2010.v4i1.3037.

18.

Pickery, J. and G. Loosveldt. 2002. “A Multilevel Multinomial Analysis of Interviewer Effects on Various Components of Unit Nonresponse.” Quality and Quantity 36(4): 427–437. Doi: http://dx.doi.org/10.1023/A:1020905911108.

19.

Poehler, E., D. Cronkite, P. Sanchez, A. Wakim, G. Dusch, H. Walrath, R. King, and J. Jones. 2016. “2020 Research and Testing: 2014 Census Test Nonresponse Followup Panel Comparisons and Instrument Analysis.” Washington, DC: U.S. Census Bureau.

20.

Purdon, S., P. Campanelli, and P. Sturgis. 1999. “Interviewers Calling Strategies on Face-to-Face Interview Surveys.” Journal of Official Statistics 15(2): 199–216.

21.

Rosen, J., J. Murphy, A. Peytchev, S. Riley, and M. Lindblad. 2011. “The Effects of Differential Interviewer Incentives on a Field Data Collection Effort.” Field Methods 23(1): 24–36. Doi: http://dx.doi.org/10.1177/1525822X10383390.

22.

Schouten, B., M. Calinescu, and A. Luiten. 2013. “Optimizing Quality of Response through Adaptive Survey Designs.” The Hague: Statistics Netherlands.

23.

Tourangeau, R., F. Kreuter, and S. Eckman. 2012. “Motivated Underreporting in Screening Interviews.” Public Opinion Quarterly 76(3): 453–469. Doi: http://dx.doi.org/10.2307/41684579.

24.

Wagner, J. and K. Olson. 2011. “Where Do Interviewers Go When They Do What They Do? An Analysis of Interviewer Travel in Two Field Surveys.” In Proceedings of the Joint Statistical Meetings, American Statistical Association, Survey Research Methods Section, Miami, July 30-August 4, 2011.

25.

Wagner, J. 2013a. “Adaptive Contact Strategies in Telephone and Face-to-Face Surveys.” Survey Research Methods 7(1): 45–55. Doi: http://dx.doi.org/10.18148/srm/2013.v7i1.5037.

26.

Wagner, J. 2013b. “Using Paradata-Driven Models to Improve Contact Rates.” In Improving Surveys with Paradata: Analytic Uses of Process Information, edited by F. Kreuter, 145–170. Hoboken, New Jersey: John Wiley and Sons.

27.

Wagner, J., B. West, N. Kirgis, J. Lepkowski, W. Axinn, and S. Ndiaye. 2012. “Use of Paradata in a Responsive Design Framework to Manage a Field Data Collection.” Journal of Official Statistics 28(4): 477–499.

28.

Walejko, G., A. Keller, G. Dusch, and P. Miller. 2014. “2020 Research and Testing: 2013 Census Test Assessment.” Washington, DC: U.S. Census Bureau.