Exploring the Validity of Behavior Coding

Abstract

Behavior coding is used to detect issues in questionnaires by flagging interactions between survey respondents and interviewers when their interactions deviate from standardized interviewing. It has become a popular method for evaluating survey questionnaire quality and has been used mostly in Western countries, with little research being done in Asia. However, because of its dependence on overt behaviors observed during the coding, it was hypothesized that it might not be as effective in a country where different communicative norms are present. We conducted an experiment in Korea using two sets of survey questionnaires with embedded problematic questions in the experimental group to investigate the validity of behavior coding as reference to the control group. Using 200 audio recordings from a telephone survey, we investigated whether behavior-coded data were helpful in identifying embedded flaws. The results indicated behavior coding in Korean surveys may not be as effective as in Western countries.

Get full access to this article

View all access options for this article.

References

Bates

Good

. 1996. An evaluation of the 1995 test census Integrated Coverage Measurement interview: Results from behavior coding. In Proceedings of the Section on Survey Research Methods, Annual Meetings of the American Statistical Association, 836–41. Alexandria, VA: American Statistical Association.

DeMaio

Rothgerb

Hess

. 1998. Improving survey quality through pretesting. Washington, DC: U.S. Bureau of the Census.

Edwards

Fry

Zahnd

Lordi

Willis

Grant

. 2004. Behavior coding across multiple languages: The 2003 California Health Interview Survey as a case study. In Proceedings of the Section on Survey Research Methods, Annual Meetings of the American Statistical Association, 4766–72. Alexandria, VA: American Statistical Association.

Fowler

F. J.

2011. Coding the behavior of interviewers and respondents to evaluate survey questions. In Question evaluation methods: Contributing to the science of data quality, eds. Madans

Miller

Maitland

Willis

, 7–21. Hoboken, NJ: John Wiley.

Fowler

F. J.

Cannell

C. F.

. 1996. Using behavioral coding to identify cognitive problems with survey questions. In Answering questions: Methodology for determining cognitive and communicative processes in Survey research, eds. Schwarz

Sudman

, 15–36. San Francisco: Jossey-Bass.

Goerman

Childs

Clifton

. 2008. Explaining differences in inter-coder reliability between English and Spanish behavior coding research. In Proceedings of the section on survey research methods, annual meetings of the American Statistical Association, 4156–63. Alexandria, VA: American Statistical Association.

Hall

E. T.

2000. Context and meaning. In Intercultural communication: A reader, eds. by Samovar

L. A.

Porter

R. E.

, 34–43. Belmont, CA: Wadsworth.

Harkness

J. A.

2003. Questionnaire translation. In Cross-cultural survey methods, eds. by Harkness

J. A.

Van de Vijver

F. J. R.

Mohler

, 35–36. Hoboken, NJ: Wiley.

Harkness

J. A.

Villar

Edwards

. 2010. Translation, adaptation, and design. In survey methods in multilingual, multiregional, and multicultural contexts, eds. by Harkness

J. A.

Braun

Edwards

Johnson

T. P.

Lyberg

Mohzler

P. P.

Pennell

Smith

T. W.

, 117–40. Hoboken, NJ: Wiley.

10.

Hughes

K. A.

2004. Comparing pretesting methods: Cognitive interview, respondent debriefing, and behavior coding. Washington, DC: U.S. Census Bureau.

11.

Javeline

1999. Response effects in polite cultures: A test of acquiescence in Kazakhstan. Public Opinion Quarterly 63:1–28.

12.

Johnson

T. P.

Holbrook

Cho

Shavitt

Chavez

Weinerm

. 2011. Cultural variability in respondent processing of health survey questions. APHA presentation, Washington, DC, October 29–November 2.

13.

Jurgenson

Childs

. 2011. Exploring the reliability of behavior coding. In Proceedings of the Section on Survey Research Methods, Annual Meetings of the American Statistical Association, 5749–61. Alexandria, VA: American Statistical Association.

14.

Krosnick

1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5:213–36.

15.

Levenstein

Conrad

Blair

Tourangeau

Maitland

. 2007. The effect of probe type on cognitive interview results: A signal detection analysis. In Proceedings of the Section on Survey Research Methods, 3850–55. Alexandria, VA: American Statistical Association.

16.

Oksenberg

Cannell

Kalton

. 1991. New strategies for pretesting survey questions. Journal of Official Statistics 7:349–65.

17.

Park

Sha

Pan

. 2014. Investigating validity and effectiveness of cognitive interviewing as a pretesting method for non-English questionnaires: Findings from Korean cognitive interviews. International Journal of Social Psychology 17:643–58.

18.

Thrasher

J. F.

Quah

A. C. K.

Dominick

Borland

Driezen

Awang

Omar

Hosking

Sirirassamee

Boado

. 2011. Using cognitive interviewing and behavioral coding to determine measurement equivalence across linguistic and cultural groups an example from the International Tobacco Control Policy Evaluation Project. Field Methods 23:439–60.

19.

Tourangeau

Rips

Rasinski

. 2000. The psychology of survey response. New York: Cambridge University Press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.33 MB