Methodological studies usually gauge response quality in narrative open-ended questions with the proportion of nonresponse, response length, response time, and number of themes mentioned by respondents. However, not all of these indicators may be comparable and appropriate for evaluating open-ended questions in a cross-national context. This study assesses the cross-national appropriateness of these indicators and their potential bias. For the analysis, we use data from two web surveys conducted in May 2014 with 2,685 respondents and in June 2014 with 2,689 respondents and compare responses from Germany, Great Britain, the United States, Mexico, and Spain. We assess open-ended responses for a variety of topics (e.g., national identity, gender attitudes, and citizenship) with these indicators and evaluate whether they arrive at similar or contradictory conclusions about response quality. We find that all indicators are potentially biased in a cross-national context due to linguistic and cultural reasons and that the bias differs in prevalence across topics. Therefore, we recommend using multiple indicators as well as items covering a range of topics when evaluating response quality in open-ended questions across countries.
AndrewsM. (2005, May). Who is being heard? Response bias in open-ended responses in a large government employee survey. Paper presented at the 60th AAPOR Annual Conference, Miami Beach, FL.
2.
BarriosM.VillarroyaA.BorregoÁ.OlléC. (2011). Response rates and data quality in web and mail surveys administered to PhD holders. Social Science Computer Review, 29, 208–220.
3.
BeattyP. C.WillisG. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71, 287–311.
4.
BehrD.BandillaW.KaczmirekL.BraunB. (2014). Cognitive probes in web surveys: On the effect of different text box size and probing exposure on response quality. Social Science Computer Review, 32, 524–533.
5.
BehrD.MeitingerK.BraunM.KaczmirekL. (2017). Web probing—Implementing probing techniques from cognitive interviewing in web surveys with the goal to assess the validity of survey questions. GESIS—Survey Guidelines. Mannheim, Germany: GESIS—Leibniz-Institute for the Social Sciences.
6.
BerryJ. W. (1969). On cross-cultural comparability. International Journal of Psychology, 4, 119–128.
7.
ChristianL. M.DillmanD. A. (2004). The influence of graphical and symbolic language manipulations on responses to self-administered questions. Public Opinion Quarterly, 68, 57–80.
8.
CouperM. P.KennedyC.ConradF. G.TourangeauR. (2011). Designing input fields for non-narrative open-ended responses in web surveys. Journal of Official Statistics, 27, 65–85.
9.
De JongM. G.SteenkampJ.-B. E.FoxJ.-P.BaumgartnerH. (2008). Using item response theory to measure extreme response style in marketing research: A global investigation. Journal of Marketing Research, 45, 104–115.
10.
DenscombeM. (2008). The length of responses to open-ended questions: A comparison of online and paper questionnaires in terms of a mode effect. Social Science Computer Review, 26, 359–368.
11.
DjurssaM. (1994). North European business cultures: Britain vs. Denmark and Germany. European Management Journal, 12, 138–146.
12.
DryerM. S.HaspelmathM. (2013). The World Atlas of language structures online. Max Planck institute for evolutionary anthropology, Leipzip. Retrieved February 5, 2018, fromhttp://wals.info
13.
EmdeM.FuchsM. (2012). Using adaptive questionnaire design in open-ended questions: A field experiment. American Association for Public Opinion Research (AAPOR) 67th Annual Conference, San Diego, CA.
14.
FuchsM. (2009). Differences in the visual design language of paper-and-pencil surveys versus web surveys: A field experimental study on the length of response fields in open-ended frequency questions. Social Science Computer Review, 27, 213–227.
15.
GoyderJ. (1986). Surveys on surveys: Limitations and potentialities. Public Opinion Quarterly, 50, 27–41.
16.
GrisayA. (2002). Translation and cultural appropriateness of the test and survey material. In WuR. (Ed.), PISA (pp. 57–70). Paris, France: Organization for Economic Cooperation and Development.
17.
GudykunstW. B.LeeC. M. (2003). Cross-cultural communication theories. In GudykunstW. B. (Ed.), Cross-cultural and intercultural communication (pp. 7–33). Thousand Oaks, CA: Sage.
HallE. T. (1976). Beyond culture. Garden City, NY: Anchor Press.
20.
HarknessJ. A. (2003). Questionnaire translation. In HarknessJ. A.Van de VijverF. J.MohlerP. (Eds.), Cross-cultural survey methods (pp. 35–56). New York, NY: John Wiley.
21.
HarzingA. W. (2006). Response styles in cross-national survey research: A 26-country study. International Journal of Cross Cultural Management, 6, 243–266.
22.
Hofelich MohrA.SellA.LindsayT. (2016). Thinking inside the box: Visual design of the response box affects creative divergent thinking in an online survey. Social Science Computer Review, 34, 347–359.
23.
HollandJ. L.ChristianL. M. (2009). The influence of topic interest and interactive probing on responses to open-ended questions in web surveys. Social Science Computer Review, 27, 196–212.
24.
HouseJ.KasperG. (1981) Politeness markers in English and German. In CoulmasF. (Ed.), Conversational routine: Explorations in standardized communication situations and pre-patterned speech (pp. 157–185). The Hague, the Netherlands: Mouton.
25.
IsraelG. D. (2010). Effects of answer space size on responses to open-ended questions in mail surveys. Journal of Official Statistics, 26, 271–285.
26.
ISSP Research Group. (2016a). International social survey programme Citizenship II—ISSP 2014 (ZA6670 Data file Version 2.0.0). GESIS Data Archive, Cologne. doi:10.4232/1.12590
27.
ISSP Research Group. (2016b). International social survey programme family and changing gender roles IV—ISSP 2012 (ZA5900 Data file Version 4.0.0). GESIS Data Archive, Cologne. doi:10.4232/1.12661
28.
JohnsonT.HolbrookA.StavrakantonakiM. (2015). Cross-cultural equivalence of survey response latencies. 6th Conference of the European Survey Research Association (ESRA). Reykjavik, Iceland.
29.
JohnsonT.KulesaP.ChoY. I.ShavittS. (2005). The relation between culture and response styles: Evidence from 19 countries. Journal of Cross-Cultural Psychology, 36, 264–277.
30.
KaczmirekL.MeitingerK.BehrD. (2017). Higher data quality in web probing with EvalAnswer: A tool for identifying and reducing nonresponse in open-ended questions. GESIS—Leibniz-Institut für Sozialwissenschaften, Cologne. Retrieved fromhttp://nbn-resolving.de/urn:nbn:de:0168-ssoar-51100-0
31.
KeuschF. (2014). The influence of answer box format on response behavior on list-style open-ended questions. Journal of Survey Statistics and Methodology, 2, 305–322.
32.
KimM. S.WilsonS. R. (1994). A cross-cultural comparison of implicit theories of requesting. Communications Monographs, 61, 210–235.
33.
KittlerM. G.RyglD.MackinnonA. (2011). Special review article: Beyond culture or beyond control? Reviewing the use of Hall’s high-/low-context concept. International Journal of Cross Cultural Management, 11, 63–82.
34.
LimT.-S. (2002). Language and verbal communication across cultures. In GudykunstW. B. (Ed.), Handbook of international and intercultural communication (pp. 69–87). Thousand Oaks, CA: Sage.
35.
LiuM. (2016, November). Verbal communication styles and culture. Communication and Culture Online. doi:10.1093/acrefore/9780190228613.013.162
36.
MaloshonokN.TerentevE. (2016). The impact of visual design and response formats on data quality in a web survey of MOOC students. Computers in Human Behavior, 62, 506–515.
37.
MarinG.GambaR. J.MarinB. V. (1992). Extreme response style and acquiescence among Hispanics: The role of acculturation and education. Journal of Cross-cultural Psychology, 23, 498–509.
38.
MeitingerK. (2017). Necessary but insufficient: Why measurement invariance tests need online probing as a complementary tool. Public Opinion Quarterly, 81, 447–472.
39.
MeitingerK. (2018). What does the general national pride item measure? Insights from web probing. International Journal of Comparative Sociology, 59, 428–450.
40.
MeitingerK.BraunM.BehrD. (2018). Sequence matters in online probing: The impact of the order of probes on response quality, motivation of respondents, and answer content. Survey Research Methods, 12, 103–120.
41.
MetzlerA.KunzT.FuchsM. (2015). The use and positioning of clarification features in web surveys. Psihologija, 48, 379–408.
42.
MiddletonK. L.JonesJ. L. (2000). Socially desirable response sets: The impact of country culture. Psychology & Marketing Journal, 17, 149–163.
43.
MillerA. L.LambertD. A. (2014). Open-ended survey questions: Item nonresponse nightmare or qualitative data dream?Survey Practice, 7, 1–11.
44.
NettleD. (2012). Social scale and structural complexity in human languages. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 367, 1829–1836.
45.
NeuliepJ. W. (2017). Intercultural communication: A contextual approach. Thousands Oaks, CA: Sage.
46.
OlsonK.ParkhurstB. (2013). Collecting paradata for measurement error evaluations. In KreuterF., (Ed.), Improving surveys with paradata: Analytic uses of process information (pp. 43–72). Hoboken, NJ: John Wiley.
47.
OudejansM.ChristianL. M. (2010). Using interactive features to motivate and probe responses to open-ended questions. In DasM.EsterP.KaczmirekL. (Eds.), Social and behavioral research and the Internet: Advances in applied methods and research strategies (pp. 215–244). New York, NY: Routledge.
48.
PellegrinoF.CoupéC.MarsicoE. (2011). Across-language perspective on speech information rate. Language, 87, 539–558.
49.
PoncheriR. M.LindbergJ. T.ThompsonL. F.SurfaceE. A. (2008). A comment on employee surveys: Negativity bias in open-ended responses. Organizational Research Methods, 11, 614–630.
RosenbloomB.LarsenT. (2003). Communication in international business-to-business marketing channels: Does culture matter?Industrial Marketing Management, 32, 309–315.
52.
SchmitzL.WeberW. (2014). Are Hofstede’s dimensions valid? A test for measurement invariance of uncertainty avoidance. Interculture Journal: Online-Zeitschrift für interkulturelle Studien, 13, 11–26.
53.
SchonlauM.CouperM. P. (2016). Semi-automated categorization of open-ended questions. Survey Research Methods, 10, 143–152.
54.
ShaoA. T.HillJ. S. (1994). Global television advertising restrictions: The case of socially sensitive products. International Journal of Advertising, 13, 347–366.
55.
SilberH.StarkT.BlomA.KrosnickJ. (2019). Implementing a multi-national study of questionnaire design. In JohnsonT.PennelB.-E.StoopI.DorerB. (Eds.), Advances in comparative survey methods: Multinational, multiregional, and multicultural contexts (3MC) (pp. 161–180). Hoboken, NJ: Wiley.
56.
SmithT. W. (2010). The globalization of survey research. In HarknessJ. A.BraunM.EdwardsB.JohnsonT.LybergL.MohlerP.PennellB.-E.SmithT. (Eds.), Survey methods in multinational, multiregional, and multicultural contexts (pp. 475–484). Hoboken, NJ: Wiley.
57.
SmythJ. D.DillmanD. A.ChristianL. M.McBrideM. (2009). Open-ended questions in web surveys: Can increasing the size of answer boxes and providing extra verbal instructions improve response quality?Public Opinion Quarterly, 73, 325–337.
58.
StarkT. H.SilberH.KrosnickJ. A.BlomA. G.AoyagiM.BelchiorA. … LawsonK. (2018). Generalization of classic question order effects across cultures. Sociological Methods & Research. doi:10.1177/0049124117747304
59.
TourangeauR.ConradF. G.CouperM. P.YeC. (2014). The effects of providing examples in survey questions. Public Opinion Quarterly, 78, 100–125.
60.
TriandisH. C. (1988). Collectivism v. individualism: A reconceptualization of a basic concept in cross-cultural social psychology. In VermaG.BagleyC. (Eds.), Cross-cultural studies of personality, attitudes and cognition (pp. 60–95). London, England: Palgrave Macmillan.
61.
TriandisH. C.MarínG.LisanskyJ.BetancourtH. (1984). Simpatía as a cultural script of Hispanics. Journal of Personality and Social Psychology, 47, 189–227.
62.
Van de VijverF.ChasiotisA. (2010). Making methods meet: Mixed designs in cross-cultural research. In HarknessJ. A.BraunM.EdwardsB.JohnsonT.LybergL.MohlerP.PennellB.-E. & SmithT. (Eds.), Survey methods in multinational, multiregional, and multicultural contexts (pp. 455–473). Hoboken, NJ: Wiley.
63.
Van VaerenberghY.ThomasT. D. (2012). Response styles in survey research: A literature review of antecedents, consequences, and remedies. International Journal of Public Opinion Research, 25, 195–217.
64.
WellsT.VidalonM.DiSograC. (2010). Differences in length of survey administration between Spanish-language and English-language survey respondents. In JSM Proceedings, Survey Research Methods Section Proceedings of Survey Research Methods Section (pp. 6186–6191). Alexandria, VA: American Statistical Association. Retrieved from http://www.asasrms.org/Proceedings/y2010f.html
65.
WierzbickaA. (1991). Cross-cultural pragmatics: The semantics of human interaction. Berlin, Germany: Mouton de Gruyter.
66.
WillisG. B. (2004). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage.
67.
ZaharnaR. S. (1995). Understanding cultural preferences of Arab communication patterns. Public Relations Review, 21, 241–255.
68.
ZuellC.MenoldN.KörberS. (2015). The influence of the answer box size on item nonresponse to open-ended questions in a web survey. Social Science Computer Review, 33, 115–122.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
