If They Don’t Understand the Question,They Don’t answer. Language Mismatch in Face-to-Face Interviews

Abstract

The provision of translated field instruments is a crucial aspect to reduce response burden and subsequently increase data quality in surveys with a multi-linguistic target population such as surveys on recent immigrants. Failure to address this can result in a mismatch between the survey language and the respondent’s mother tongue. By using a survey on refugees in Germany, this article explores the correlation of the absence of the respondents’ mother tongue on item nonresponse – a crucial aspect of data quality. Further, this article investigates whether supplementary audio recordings in the same language as the written questions can reduce item nonresponse when the mother tongue is not available. To answer the research questions, all missing answers per individual are counted and analyzed by means of poisson regression analyses. In a second step, the likelihood of item-nonresponse for single items is estimated as well. Results show that a language mismatch as well as the usage of audio recordings increase item nonresponse.

Keywords

Respondent burden item nonresponse cross-cultural survey methods poisson regression

References

Ahlmark, N., M.H. Algren, T. Holmberg, M.L. Norredam, S.S. Nielsen, A.B. Blom, and K. Juel. 2015. “Survey nonresponse among ethnic minorities in a national health survey – a mixed-method study of participation, barriers, and potentials.” Ethnicity & Health 20(6): 611–632. DOI: https://doi.org/10.1080/13557858.2014.979768.

Benstead, L J. 2013. “Effects of interviewer–respondent gender interaction on attitudes toward women and politics: Findings from Morocco.” International Journal of Public Opinion Research 26(3): 369–282. DOI: https://doi.org/10.1093/ijpor/edt024.

Cenoz, J. 2013. Defining Multilingualism. Annual Review of Applied Linguistics 33: 3–18. DOI: https://doi.org/10.1017/S026719051300007X.

Converse, J.M., and S. Presser. 1986. Survey questions: Handcrafting the standardized questionnnaire. Newbury Park: Sage.

Couper, M.P., P. Berglund, N. Kirgis, and S. Buageila. 2016. “Using text-to-speech (TTS) for audio computer-assisted self-interviewing (ACASI).” Field Methods 28(2): 95–111. DOI: https://doi.org/10.1177/1525822X14562350.

Couper, M.P., R. Tourangeau, and T. Marvin. 2009. “Taking the audio out of audio-CASI.” Public Opinion Quartely 73(2): 281–303. DOI: https://doi.org/10.1093/poq/nfp025.

Coutts, E., and B. Jann. 2011. “Sensitive questions in online surveys: Experimental results for the randomized response technique (RRT) and the unmatched count technique (UCT).” Sociological Methods & Research 40(1): 169–193. DOI: https://doi.org/10.1177/0049124110390768.

Eckman, S., and F. Kreuter. 2018. “Misreporting to Looping Questions in Surveys: Recall, Motivation and Burden.” Survey Research Methods 12(1): 59–74. DOI: https://doi.org/10.18148/srm/2018.v12i1.7168.

Elliott, M.N., W.S. Edwards, D.J. Klein, and A. Heller. 2012. “Differences by survey language and mode among chinese respondents to a CAHPS health plan survey.” Public Opinion Quartely 76(2): 238–264. DOI: https://doi.org/10.1093/poq/nfs020.

10.

Glemser, A., S. Huber, and I. Leven. 2017. “Report 2016 from kantar public (Samples M1-4).” In SOEP Wave Report 2016, edited by J. Britzke and J. Schupp: 43–54. Berlin: SOEP–The German Socio-Economic Panel study at DIW Berlin. Available at: https://www.diw.de/documents/publikationen/73/diw_01.c.560446.de/wave_report_2016.pdf.

11.

Greenfield, M. 2009. “Reaching gypsies and travellers.” Primary Health Care 19(8): 26–27. DOI: https://doi.org/10.7748/phc2009.10.19.8.26.c7301.

12.

Harkness, J.A. 2003. “Questionnaire Translation.” In Cross-Cultural Survey Methods, edited by J.A. Harkness, F.J.R. van de Vijer and P.P. Mohler: 35–56. Hoboken: John Wiley & Sons.

13.

Harkness, J.A., F.J.R. van de Vijver, and P.P. Mohler. 2003. Cross-Cultural Survey Methods, Hoboken: John Wiley & Sons.

14.

Harkness, J.A. 2007. “Improving the comparability of translations.” In Measuring Attitudes Cross-Nationally: Lessons from the European Social Survey, edited by R. Jowell, C. Roberts, R. Fitzgerald, and G. Eva: 79–94. London: Sage.

15.

Harkness, J.A., B.-E. Pennell, and A. Schoua-Glusberg. 2004. “Survey questionnaire translation and assessment.” In Methods for Testing and Evaluating Survey Questionnaires, S. Presser, J.M. Rothgeb, M P. Couper, J.T. Lessler, E. Martin, J. Martin and E. Singer: 453–474. New Jersey: Wiley & Sons.

16.

Holbrook, A.L., Y.I. Cho, and T.P. Johnson. 2006. The Impact of Question and Respondent Characteristics and Mapping Difficulties. Public Opinion Quarterly 70 (4): 565–595. DOI: https://doi.org/10.1093/poq/nfl027.

17.

Jacobsen, J. 2018. “Language barriers during the fieldwork of the IAB-BAMF-SOEP Survey of Refugees in Germany.” In Surveying the Migrant Population: Consideration of Linguistic and Cultural Issues, edited by D. Behr: 75–84. Köln: Gesis.

18.

Jacobsen, J., and L.M. Fuchs. 2020. “Can We Compare Conceptions of Democracy in Cross-Linguistic and Cross-National Research? Evidence from a Random Sample of Refugees in Germany.” Social Indicators Research 151: 669–690. DOI: https://doi.org/10.1007/s11205-020-02397-6.

19.

Jacobsen, J., J. Klikar, and J. Schupp. 2017. Scales manual IAB-BAMF-SOEP Survey of Refugees in Germany – revised version. Berlin: DIW/SOEP. (SOEP Survey Papers 475: Series C – Data Documentation). Available at: https://www.diw.de/documents/publikationen/73/diw_01.c.572342.de/diw_ssp0475.pdf.

20.

Jann, B., J. Jerke, and I. Krumpal. 2012. “Asking sensitive questions using the crosswise model: An experimental survey measuring plagiarism.” Public Opinion Quarterly 76(1): 32–49. DOI: https://doi.org/10.1093/poq/nfr036.

21.

Jaworski, J., L. Pagel, and J. Schupp. 2017. “Zuordnung und Überführung erworbener Bildungsabschlüsse in das internationale Bildungs klassifikations system (ISCED).” Politikeratung kompakt 123: 21–28. DOI: http://hdl.handle.net/10419/176772.

22.

Jesske, B. 2018. “Surveying migrants in the context of the low-income panel PASS.” In Surveying the Migrant Population: Consideration of Linguistic and Cultural Issues, edited by D. Behr: 85–94. Köln: GESIS.

23.

Johnson, T.P, D.P. O’Rourke, N. Chavez, S. Sudman, R.B. Warnecke, and L. Lacey. 1997. “Social Cognition and Responses to Survey Questions among Culturally Diverse Populations,” in Survey Measurement and Process Quality, edited by L. Lyberg, P. Biemer, M. Collins, E.D. de Leeuw, C. Dippo, N. Schwarz, and D. Trewin: 87–113, New York: John Wiley & Sons.

24.

Johnson T.P.,Y.I. Cho, A. Holbrook, D. O’Rourke, R.B. Warnecke, and N. Chávez. 2006. “Culturalvariability in the effects of question design featureson respondent comprehension of health surveys.” Annals of Epidemiology 16(9): 661–668. DOI: https://doi.10.1016/j.annepidem.2005.11.011.

25.

Kieruj, N.D., J. Mulder, A. Wijnant, S. Douhou, and F. Conrad. 2013. Hearing voices: supporting online questionnaires with text-to-speech technology. Paper presented at the Priority Programme on Survey Methodology, Bremen.

26.

Kish, L. 1962. “Studies of interviewers variance for attitudinal variables.” Journal of the American Statistical Association 57(279): 91–115.

27.

Kleiner, B., O. Lipps, and E. Ferrez. 2015. “Language Ability and Motivation Among Foreigners inb Survey Responding.” Journal of Survey Statistics and Methodology 3(3): 339–360. DOI: https://doi.org/10.1093/jssam/smv015.

28.

Kosyakova, Y., L. Olbrich, J.W. Sakshaug, and S. Schwanhäuser. 2019. “Identification of interviewer falsification in the IAB-BAMF-SOEP Survey of Refugees in Germany.” FDZ Methodenreport 2019(2). DOI: https://doi.org/10.5164/IAB.FDZM.1902.en.v1.

29.

Kosyakova, Y., J. Skopek, and S. Eckman. 2015. “Do interviewers manipulate responses to filter questions? evidence from a multilevel approach.” International Journal of Public Opinion Research 27(3): 417–431. DOI: https://doi.org/10.1093/ijpor/edu027.

30.

Kreuter, F., S. McCulloch, S. Presser, and R. Tourangeau. 2011. “The effect of asking filter questions in interleafed versus grouped format.” Sociological Methods & Research 40(1): 88–104. DOI: https://doi.org/10.1177/0049124110392342.

31.

Kroenke, K., R.L. Spitzer, J.B.W. Williams, and B. Löwe. 2009. “An ultra brief screening scale for anxiety and depression: The PHQ-4.” Psychosomatics 50(6): 613–621. DOI: https://doi.org/10.1016/S0033-3182(09)70864-3.

32.

Kroh, M., S. Kühne, J. Goebel, and F. Preu. 2015. The 2013 IAB-SOEP Migration Sample (M1): Sampling design and weighting adjustment. Berlin: DIW/SOEP. (SOEP Survey Papers 271: Series C – Data Documentation). Available at: https://www.diw.de/documents/publikationen/73/diw_01.c.570750.de/diw_ssp0271.pdf.

33.

Kroh, M., S. Kühne, J. Jacobsen, M. Siegert, and R. Siegers. 2017. Sampling, nonresponse, and integrated weighting of the 2016 IAB-BAMF-SOEP Survey of Refugees (M3/M4). Berlin: DIW/SOEP. (SOEP Survey Papers 477: Series C – Data Documentation). Available at: https://www.diw.de/documents/publikationen/73/diw_01.c.572346.de/diw_ssp0477.pdf.

34.

Krosnick, J.A. 1991. “Response strategies for coping with the cognitive demands of aittude measures in surveys.” Applied Cognitive Psychology 5: 213–236. DOI: https://doi.org/10.1002/acp.2350050305.

35.

Krosnick, J.A. 1999. “Survey research.” Annual Review of Psychology 50: 537–567. DOI: https://doi.org/10.1146/annurev.psych.50.1.537.

36.

Krumpal, I. 2013. “Determinants of social desirability bias in sensitive surveys: a literature review.” Quality & Quantity 47(4): 2025–2047. DOI: https://doi.org/10.1007/s11135-011-9640-9.

37.

Kühne, S., J. Jacobsen, and M. Kroh. 2019. “Sampling in Times of High Immigration: The Survey Process of the IAB-BAMF-SOEP Survey of Refugees.” Survey Methods: Insights from the Field. DOI: https://doi.org/10.13094/SMIF-2019-00005.

38.

Kühne, S., and M. Kroh. 2017. The 2015 IAB-SOEP Migration Study M2: Sampling design, nonresponse, and weighting adjustment. Berlin: DIW/SOEP. (SOEP Survey Papers 473: Series C – Data Documentation). Available at: https://www.diw.de/documents/publikationen/73/diw_01.c.571195.de/diw_ssp0473.pdf.

39.

Lambert, D. 1992. “Zero-inflated pisson regression. With an application to defects in manufacturing.” Technometrics 34(1): 1–14. DOI: https://doi.org/10.2307/1269547.

40.

Lenzner, T., L. Kaczmirek, and A. Lenzner. 2010. “Cognitive Burden of Survey Questions and Response Times: A Psycholinguistic Experiment.” Applied Cognitive Psychology 24(7). DOI: https://doi.org/10.1002/acp.1602.

41.

Little, R.J., and D.B. Rubin. 2014. Statistical Analysis with Missing Data. 2nd Edition. Hoboken: John Wiley & Sons

42.

Lyberg, L., M. Stange, J. Harkness, P. Mohler, B.-E. Pennell, and L. Japec. 2014. “A review of quality issues associated with studying hard-to-survey populations.” In Hard-to-Survey Populations, edited by R. Tourangeau, B. Edwards, T.P. Johnson, K.M. Wolter and N. Bates: 82–107. Cambridge: Cambridge University Press.

43.

Massey, D.S. 2014. “Challenges to Surveying Immigrants.” In Hard-to-Survey Populations, edited by R. Tourangeau, B. Edwards, T.P. Johnson, K.M. Wolter and N. Bates: 270–292. Cambridge: Cambridge University Press.

44.

Mostafa, T., and R.D. Wiggins. 2015. “The impact of attrition and non-response in birth cohort studies: a need to incorporate missingness strategies.” Longitudinal and Life Course Studies 6(2): 131–146. DOI: https://doi.org/10.14301/llcs.v6i2.312.

45.

Park, H., M.M. Sha, and G. Willis. 2016. Influence of English-Language Proficiency on the Cognitive Processing of Survey Questions. Field Methods 28(4): 415–430. DOI: https://doi.org/10.1177/1525822X16630262.

46.

Pennell, B.-E., Y. Deshmukh, J. Kelley, P. Maher, J. Wagner, and D. Tomlin. 2014. Disaster Research: Surveying Displaced Persons. In Hard-to-Survey Populations, edited by R. Tourangeau, B. Edwards, T.P. Johnson, K.M. Wolter, and N. Bates, 111–133. Cambridge: Cambridge University Press.

47.

Pennell, B.-E., K.C. Hibben, L.E. Lyberg, P.P. Mohler, and G. Worki. 2017. “A total survey error perspective on surveys in multinational, multiregional, and multicultural contexts.” In Total Survey Error in Practice, edited by P.P. Biemer, E. de Leeuw, S. Eckman, B. Edwards, F. Kreuter, L.E. Lyberg, N.C. Tucker and B.T. West. Hoboken: Wiley & Sons.

48.

Perez, P., and B. Baffour. 2018. “Respondent mental health, mental disorders and survey interview outcomes.” Survey Research Methods 12(2): 161–176. DOI: https://doi.org/10.18148/srm/2018.v12i2.7225.

49.

Peytcheva, E. 2018. “Can the language of survey administration influence respondents’ answers?” In Advances in Comparative Survey Methods: Multinational, Multiregional, and Multicultural Contexts (3MC), edited by T.P. Johnson, B.-E. Pennell, I.A.L. Stoop and B. Dorer: 325-340. Hoboken, NJ: Wileys and Sons.

50.

Rother, N., D. Schacht, and J.A. Scheible. 2017. “Sprachpotenziale: Sprachkenntnisse und Alphabetisierungsgrad von Geflüchteten.” Politikberatung kompakt 123: 29–42. DOI: http://hdl.handle.net/10419/176772.

51.

SOEp. 2016a. Erhebungsinstrumente der IAB-BAMF-SOEP-Befragung von Geflüchteten 2016: Haushaltsfragebogen, Stichproben M3-M4. Berlin: DIW/SOEP. (SOEP Survey Papers 361: Series A – Survey Instruments). Available at: https://www.econstor.eu/bitstream/10419/183117/1/1032414014.pdf.

52.

SOEp. 2016b. Erhebungsinstrumente der IAB-BAMF-SOEP-Befragung von Geflüchteten 2016: Integrierter Personen- und Biografiefragebogen, Stichproben M3-M4. Berlin: DIW/SOEP. (SOEP Survey Papers 362: Series A – Survey Instruments). Available at: https://www.econstor.eu/bitstream/10419/183118/1/1032416866.pdf.

53.

SOEp. 2018. SOEP-Core v33.1 – Documentation of Person-related Status and Generated Variables in PGEN. Berlin: DIW/SOEP. (SOEP Survey Papers 483: Series D – Varaible Descriptions and Coding). Available at: https://www.diw.de/documents/publikationen/73/diw_01.c.581299.de/diw_ssp0483.pdf.

54.

Survey Research Center. 2016. Guidelines for Best Practice in Cross-Cultural Surveys. Available at: http://ccsg.isr.umich.edu/.

55.

Tourangeau, R., F. Kreuter, and S. Eckmann. 2015. “Motivated misreporting: shaping answers to reduce survey burden.” In Survey Measurements: Techniques, Data Quality and Sources of Error, edited by U. Engel: 24–41. Frankfurt a. M.: Campus.

56.

Tourangeau, R., and T.W. Smith. 1996. “Asking sensitive questions: The impact of data collection mode, question format, and questions context.” Public Opinion Quartely 60(2): 275–304. DOI: https://doi.org/10.1086/297751.

57.

Tourangeau, R., L.J. Rips, and K. Rasinski. 2000. The Psychology of Survey Response. Cambridge: Cambridge University Press.

58.

Turner, C.F., L. Ku, S.M. Rogers, L.D. Lindberg, J.H. Pleck, and F. Sonenstein. 1998. “Adolescent sexual behavior, drug use and violence: Increased reporting with computer survey technology.” Science 280(8): 867–873. DOI: https://doi.org/10.1126/science.280.5365.867.

59.

Von Gostomski, C.B., and M. Pupeter. 2008. “Zufallsbefragung von Ausländern auf Basis des Ausländerzentralregisters: Erfahrungen bei der Repräsentativbefragung “Ausgewählte Migrantengruppen in Deutschland 2006/2007” (RAM).” Methoden, Daten, Analysen 2(2): 149–177. DOI: http://nbn-resolving.de/urn:nbn:de:0168-ssoar-126673.

60.

Warnecke, R.B., T.P. Johnson, N. Chavez, S. Sudman, D.P. O’Rourke, L. Lacey, and J. Horm. 1997. Improving Question Wording in Surveys of Culturally Diverse Populations. Annual Epidemiological Psychology 7: 334–342. DOI: https://doi.org/10.1016/s1047-2797(97)00030-6.

61.

West, B.T., and A.G. Blom. 2017. “Explaining Interviewer Effects: A Research Synthesis.” Journal of Survey Statistics and Methodology 5(2): 175–211. DOI: https://doi.org/10.1093/jssam/smw024.

62.

Wenz, A., T. Al Baghal, and A. Gaia. 2021. “Language Proficiency Among Respondents: Implications for Data Quality in a Longitudinal Face-to-Face Survey.” Journal of Survey Statistics and Methodology 9(1): 73–93. DOI: https://doi.org/10.1093/jssam/smz045.

63.

Zavala-Rojas, D. 2018. “Exploring Language Effects in Crosscultural Survey Research: Does the Language of Administration Affect Answers About Politics?” Methods, data, analyses 12(1): 127–150. DOI: https://doi.org/10.12758/mda.2017.16.