Behavior coding is a widely used method for pretesting survey questionnaires. It is based on the idea that certain types of interviewer-respondent interactions indicate question flaws if the number of such interactions exceeds some specified percentage of the pretest interviews. In this article, we examine statistical approaches for assessing the reliability of different observed percentages of problematic interactions.
Blair, J., F. Conrad, A. Ackermann, and G. Claxton.2006. The effect of sample size on cognitive interview findings. Proceedings of the Section on Survey Research Methods, Joint Statistical Meetings, American Statistical Association, August 6—10, in Alexandria, VA.
2.
Conrad, F.G., and J. Blair.2004. Data quality in cognitive interviews: The case of verbal reports. In Methods for testing and evaluating survey questions, edited by S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, and E. Singer, 67—87. Hoboken NJ: John Wiley.
3.
Czaja, R., and J. Blair.2005. Designing surveys: A guide to decisions and procedures, 2nd ed. Thousand Oaks, CA: Pine Forge Press.
4.
Fowler, F.1989. Coding behavior in pretests to identify unclear questions . Conference proceedings, Health Survey Research Methods, 9—12, May 2—4. Washington, DC: National Center for Health Services Research.
5.
———. 1993. Survey research methods, 2nd ed. Newbury Park, CA: Sage.
Groves, R.M., F.J. Fowler, M.P. Couper, J.M. Lepkowski, E. Singer, and R. Tourangeau.2004. Survey methodology. Hoboken, NJ: John Wiley.
8.
Oskenberg, L., C. Cannell, and G. Kalton.1991. New strategies for pretesting questions . Journal of Official Statistics7:349—66.
9.
Presser, S., and J. Blair.1994. Survey pretesting: Do different methods produce different results. In Sociological methodology, edited by P. Marsen, 73—104. Washington, DC: American Sociological Association.
10.
Zukerberg, A.L., D.R. Von Thurn, and J.C. Moore.1995. Practical considerations in sample size selection for behavior coding pretests. Proceedings of the Section on Survey Research Methods, Joint Statistical Meetings, American Statistical Association, 1116—1121, August 13—17, in Alexandria, VA.
