Careless responding (CR) in surveys has been identified as a serious threat for the validity of survey data. It occurs when participants respond without following the instructions or reading the content of items. The purpose of this article is to assess and explain the amount of CR at the individual and country level. In our study, individual CR (measured by indices using a short Big 5 personality questionnaire) correlated with education level and personality traits. In addition, using a sample of 8,320 participants from 34 countries, CR at the country level was investigated and it was strongly correlated with the cultural dimensions: human development, individualism, gender inequality, and power distance. CR can be seen as a powerful predictor of differences between countries. Finally, a comparison between CR and response styles (extreme and midpoint answers, acquiescence, and socially desirable responses) was conducted. CR and response styles showed some overlap, but loaded on different factors. CR is compatible with extreme answers, midpoint answers, and acquiescence, but different from social desirability. Social desirability is the only response style which requires careful reading of items.
AhammerI. M. (1971). Desirability judgments as a function of item content, instruction set, and sex: A life-span developmental study. Human Development, 14, 195-207. doi:10.1159/000271213
2.
ArcherJ. (2006). Cross-cultural differences in physical aggression between partners: A social-role analysis. Personality and Social Psychology Review, 10, 133-153. doi:10.1207/s15327957pspr1002_3
3.
Benet-MartinezV.JohnO. P. (1998). Los Cinco Grandes across cultures and ethnic groups: Multitrait multimethod analyses of the big five in Spanish and English. Journal of Personality and Social Psychology, 75, 729-750. doi:10.1037//0022-3514.75.3.729
4.
BowlingN. A.HuangJ. L.BraggC. B.KhazonS.LiuM.BlackmoreC. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111, 218-229. doi:10.1037/pspp0000085
5.
BrislinR. W. (1980). Translation and content analysis of oral and written material. In TriandisH. C.BerryJ. W. (Eds.), Methodology: Vol. 2. Handbook of cross-cultural psychology (pp. 389-444). Boston, MA: Allyn & Bacon.
6.
BuckleyJ. (2009). Cross-national response styles in international educational assessments: Evidence from PISA 2006. Department of Humanities and Social Sciences in the Professions, New York University. Retrieved from https://edsurveys.rti.org/PISA/documents/Buckley_PISAresponsestyle.pdf
7.
CouchA.KenistonK. (1960). Yeasayers and naysayers: Agreeing response set as a personality variable. Journal of Abnormal and Social Psychology, 60, 151-174. doi:10.1037/h0040372
8.
CronbachL. J. (1946). Response sets and test validity. Educational and Psychological Measurement, 6, 475-494. doi:10.1177/001316444600600405
9.
CurranP. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4-19. doi:10.1016/j.jesp.2015.07.006
10.
DesimoneJ. A.HarmsP. D.DeSimoneA. J. (2014). Best practice recommendations for data screening. Journal of Organizational Behavior, 36, 171-181. doi:10.1002/job.1962
11.
DillonM. (2014). Introduction to sociological theory: Theorists, concepts, and their applicability to the twenty-first century (2nd ed.). Chichester, UK: Wiley-Blackwell.
12.
EbbelerC.GrauI.BanseR. (2017). Cultural and individual factors determine physical aggression between married partners: Evidence from 34 countries. Journal of Cross-Cultural Psychology, 48, 1098-1118. doi:10.1177/0022022117719497
13.
EysenckS. B. G.EysenckH. J. (1963). An experimental investigation of “desirability” response set in a personality questionnaire. Life Science, 2, 343-355. doi:10.1016/0024-3205(63)90168-1
14.
FischerA. H.MansteadA. S. R. (2000). The relation between gender and emotions in different cultures. In FischerA. (Ed.), Studies in Emotion and Social Interaction. Gender and emotion: Social psychological perspectives (pp. 71-94). Cambridge, UK: Cambridge University Press.
15.
FischerR. (2004). Standardization to account for cross-cultural response bias: A classification of score adjustment procedures and review of research in JCCP. Journal of Cross-Cultural Psychology, 35, 263-282. doi:10.1177/0022022104264122
16.
FischerR. (2012). Value isomorphism in the European Social Survey: Exploration of meaning shifts in values across levels. Journal of Cross-Cultural Psychology, 43, 883-898. doi:10.1177/0022022111413276
17.
HampsonS. E. (1998). When is an inconsistency not an inconsistency? Trait reconciliation in personality description and impression formation. Journal of Personality and Social Psychology, 74, 102-117. doi:10.1037/0022-3514.74.1.102
18.
HeJ.BartramD.InceogluI.Van de VijverF. J. R. (2014). Response styles and personality traits: A multilevel analysis. Journal of Cross-Cultural Psychology, 45, 1028-1045. doi:10.1177/0022022114534773
19.
HeJ.BuchholzJ.KliemeE. (2017). Effects of anchoring vignettes on comparability and predictive validity of student self-reports in 64 cultures. Journal of Cross-Cultural Psychology, 48, 319-334.
20.
HeJ.Van de VijverF. J. R. (2015). Effects of a general response style on cross-cultural comparisons: Evidence from the Teaching and Learning International Survey. Public Opinion Quarterly, 79, 267-290. doi:10.1093/poq/nfv006
21.
HeJ.Van de VijverF. J. R.Dominguez EspinosaA.MuiP. H. C. (2014). Toward a unification of acquiescent, extreme, and midpoint response styles: A multilevel study. International Journal of Cross Cultural Management, 14, 306-322. doi:10.1177/1470595814541424
22.
HofstedeG. H. (2001). Culture’s consequences: Comparing values, behaviors, institutions, and organizations across nations (2nd ed.). Thousand Oaks, CA: Sage.
23.
HuangJ. L.CurranP. G.KeeneyJ.PoposkiE. M.DeShonR. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99-114. doi:10.1007/s10869-011-9231-8
24.
HuangJ. L.LiuM.BowlingN. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100, 828-845. doi:10.1037/a0038510
25.
JohnsonJ. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103-129. doi:10.1016/j.jrp.2004.09.009
26.
KamC. C. S.MeyerJ. P. (2015). How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organizational Research Methods, 18, 512-541. doi:10.1177/1094428115571894
27.
KungF. Y. H.KwokN.BrownD. J. (2018). Are attention check questions a threat to scale validity?Applied Psychology, 67, 264-283. doi:10.1111/apps.12108
28.
KurtzJ. E.ParrishC. L. (2001). Semantic response consistency and protocol validity in structured personality assessment: The case of the NEO-PI-R. Journal of Personality Assessment, 76, 315-332. doi:10.1207/S15327752JPA7602_12
29.
KyllonenP. C.BertlingJ. (2013). Innovative questionnaire assessment methods to increase cross-country comparability. In RutkowskiL.von DavierM.RutkowskiD. (Eds.), A handbook of international large-scale assessment data analysis: Background, technical issues, and methods of data analysis (pp. 277-286). London, England: Chapman Hall.
30.
MahalanobisP. C. (1936). On the generalized distance in statistics. Proceedings of the National Institute of Sciences of India, 2, 49-55.
31.
ManiaciM. R.RoggeR. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61-83. doi:10.1016/j.jrp.2013.09.008
32.
McGrathR. E.MitchellM.KimB. H.HoughL. (2010). Evidence for response bias as a source of error variance in applied assessment. Psychological Bulletin, 136, 450-470. doi:10.1037/a0019216
33.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437-455. doi:10.1037/a0028085
34.
NezlekJ. B.Schröder-AbéM.SchützA. (2006). Mehrebenenanalysen in der psychologischen Forschung [Multi level analyses in psychological research]. Psychologische Rundschau, 57, 213-223. doi:10.1026/0033-3042.57.4.213
35.
PaulhusD. L. (1981). Control of social desirability in personality inventories: Principal factor deletion. Journal of Research in Personality, 15, 383-388. doi:10.1016/0092-6566(81)90035-0
36.
PaulhusD. L. (1991). Measurement and control of response biases. In RobinsonJ.ShaverP.WrightsmanL. (Eds.), Measures of personality and social psychological attitudes (Vol. 1, pp. 17-59). San Diego, CA: Academic Press.
37.
PengK.NisbettR. E. (1999). Culture, dialectics, and reasoning about contradiction. American Psychologist, 54, 741-754. doi:10.1037/0003-066X.54.9.741
38.
PiedmontR. L.McCraeR. R.RiemannR.AngleitnerA. (2000). On the invalidity of validity scales: Evidence from self-reports and observer ratings in volunteer samples. Journal of Personality and Social Psychology, 78, 582-593. doi:10.1037//0022-3514.78.3.582
39.
PiferiR. L.JobeR. L. (2003). Identifying careless response in questionnaire research: A serendipitous finding. Psychological Reports, 93, 320-322. doi:10.2466/pr0.2003.93.1.320
40.
RaudenbushS. W.BrykA. S.CheongY. F.CongdonR.du ToitM. (2011). HLM7: Hierarchical linear and nonlinear modeling. Lincolnwood, IL: Scientific Software International.
41.
ShulmanA. (1973). A comparison of two scales on extremity response bias. Public Opinion Quarterly, 37, 407-412.
42.
SmithP. B. (2004). Acquiescent response bias as an aspect of cultural communication style. Journal of Cross-Cultural Psychology, 35, 50-61. doi:10.1177/0022022103260380
VonkovaH.PapajoanuO.StipekJ. (2018). Enhancing the cross-cultural comparability of self-reports using the overclaiming technique: An analysis of accuracy and exaggeration in 64 cultures. Journal of Cross-Cultural Psychology, 49, 1247-1268. doi:10.1177/0022022118787042
45.
WoodsC. (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28, 189-194. doi:10.1007/s10862-005-9004-7
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
