Handling the Inconsistency between Self-Report and the Actual Behavior: Validity of Excluding Survey Participants with Insufficient Effort Responding

Abstract

In this study, we aimed to understand and reduce the difference between self-report in a survey and the actual behavior. Thus, we investigated whether such a difference was caused by participants who engaged in insufficient effort responding (IER), which has been receiving increasing research attention. We collected and analyzed data of actual and self-reported smartphone game usage from behavior logs and survey responses including the items associated with the IER scale, respectively. The results confirmed a strong tendency of overreporting and low correlations between the behavior log and survey responses for IER participants. Although the distributions of survey responses differed between IER participants and others, those of the behavior log did not. In conclusion, when IER participants are excluded, the difference between the behavior log and survey responses reduces, but the distribution of the actual behavior remains the same without selection bias.

Keywords

behavior log survey response insufficient effort responding

Get full access to this article

View all access options for this article.

References

AlQuraan

(2019). The effect of insufficient effort responding on the validity of student evaluation of teaching. Journal of Applied Research in Higher Education, 11(3), 604–615. https://doi.org/10.1108/JARHE-03-2018-0034.

American Psychiatric Association . (2013). Diagnostic and statistical manual of mental disorders (5th ed.).

Araujo

Wonneberger

Neijens

de Vreese

(2017). How much time do you spend online? Understanding and improving the accuracy of self-reported measures of internet use. Communication Methods and Measures, 11(3), 173–190. https://doi.org/10.1080/19312458.2017.1317337.

Baumgartner

Steenkamp

J. -B. E. M.

(2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2), 143–156. https://doi.org/10.1509/jmkr.38.2.143.18840.

Baumgartner

Weijters

(2019). Measurement in marketing. Foundations and Trends® in Marketing, 12(4), 278–400. https://doi.org/10.1561/1700000058.

Beck

J. T.

Rahinel

Bleier

(2020). Company worth keeping: Personal control and preferences for brand leaders. Journal of Consumer Research, 46(5), 871–886. https://doi.org/10.1093/jcr/ucz040.

Boase

Ling

(2013). Measuring mobile phone use: Self-report versus log data. Journal of Computer-Mediated Communication, 18(4), 508–519. https://doi.org/10.1111/jcc4.12021.

Bond

S. D.

S. X.

Wen

(2019). Speaking for “free”: Word of mouth in free- and paid-product settings. Journal of Marketing Research, 56(2), 276–290. https://doi.org/10.1177/0022243718821957.

Bowling

N.A.

Huang

J. L.

Bragg

C. B.

Khazon

Liu

Blackmore

C. E.

(2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111(2), 218–229. https://doi.org/10.1037/pspp0000085.

10.

Brosnan

Babakhani

Dolnicar

(2019). “I know what you’re going to ask me” why respondents don’t read survey questions. International Journal of Market Research, 61(4), 366–379. https://doi.org/10.1177/1470785318821025.

11.

Chen

R. S.

C. H.

(2015). Investigating the relationship between thinking style and personal electronic device use and its implications for academic performance. Computers in Human Behavior, 52, 177–183. https://doi.org/10.1016/j.chb.2015.05.042.

12.

Chou

Condron

Belland

J. C.

(2005). A review of the research on internet addiction. Educational Psychology Review, 17(4), 363–388. https://doi.org/10.1007/s10648-005-8138-1.

13.

Cole

J. S.

Gonyea

R. M.

(2010). Accuracy of Self-reported SAT and ACT test scores: Implications for research. Research in Higher Education, 51(4), 305–319. https://doi.org/10.1007/s11162-009-9160-9.

14.

Collopy

(1996). Biases in retrospective self-reports of time use: An empirical study of computer users. Management Science, 42(5), 758–767. https://doi.org/10.1287/mnsc.42.5.758.

15.

Consiglio

van Osselaer

S. M. J.

(2019). The devil you know: Self-esteem and switching responses to poor service. Journal of Consumer Research, 46(3), 590–605. https://doi.org/10.1093/jcr/ucz001.

16.

Corral-Verdugo

(1997). Dual “realities” of conservation behavior: Self-reports vs observations of re-use and recycling behavior. Journal of Environmental Psychology, 17(2), 135–145. https://doi.org/10.1006/jevp.1997.0048.

17.

Corral-Verdugo

Bechtel

R. B.

Fraijo-Sing

(2003). Environmental beliefs and water conservation: An empirical study. Journal of Environmental Psychology, 23(3), 247–257. https://doi.org/10.1016/S0272-4944(02)00086-5.

18.

Corral-Verdugo

Figueredo

A. J.

(1999). Convergent and divergent validity of three measures of conservation behavior: The multitrait-multimethod approach. Environment and Behavior, 31(6), 805–820. https://doi.org/10.1177/00139169921972353.

19.

Croteau

A. M.

Dyer

Miguel

(2010). Employee reactions to paper and electronic surveys: An experimental comparison. IEEE Transactions on Professional Communication, 53(3), 249–259. https://doi.org/10.1109/TPC.2010.2052852.

20.

Curran

P. G.

(2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19. https://doi.org/10.1016/j.jesp.2015.07.006.

21.

Delley

Brunner

T. A.

(2018). Household food waste quantification: Comparison of two methods. British Food Journal, 120(7), 1504–1515. https://doi.org/10.1108/BFJ-09-2017-0486.

22.

Deng

Kanthawala

Meng

Peng

Kononova

Hao

Zhang

David

(2019). Measuring smartphone usage and task switching with log tracking and self-reports. Mobile Media & Communication, 7(1), 3–23. https://doi.org/10.1177/2050157918761491.

23.

De Reuver

Bouwman

(2015). Dealing with self-report bias in mobile Internet acceptance and usage studies. Information Management, 52(3), 287–294. https://doi.org/10.1016/j.im.2014.12.002.

24.

DeSimone

J. A.

Harms

P. D.

(2018). Dirty data: The effects of screening respondents who provide low-quality data in survey research. Journal of Business and Psychology, 33(5), 559–577. https://doi.org/10.1007/s10869-017-9514-9.

25.

Dunn

A. M.

Heggestad

E. D.

Shanock

L. R.

Theilgard

(2018). Intra-individual response variability as an indicator of insufficient effort responding: Comparison to other indicators and relationships with individual differences. Journal of Business and Psychology, 33(1), 105–121. https://doi.org/10.1007/s10869-016-9479-0.

26.

Dunnette

M. D.

(1952). Accuracy of students’ reported honor point averages. Journal of Applied Psychology, 36(1), 20–22. https://doi.org/10.1037/h0054387.

27.

Elimelech

Ert

Ayalon

(2019). Exploring the drivers behind self-reported and measured food wastage. Sustainability, 11(20), 5677. https://doi.org/10.3390/su11205677.

28.

Fernandes

Puntoni

van Osselaer

S. M. J.

Cowley

(2016). When and why we forget to buy. Journal of Consumer Psychology, 26(3), 363–380. https://doi.org/10.1016/j.jcps.2015.06.012.

29.

Goodman

J. K.

Paolacci

(2017). Crowdsourcing consumer research. Journal of Consumer Research, 44(1), 196–210. https://doi.org/10.1093/jcr/ucx047.

30.

Grau

Ebbeler

Banse

(2019). Cultural differences in careless responding. Journal of Cross-Cultural Psychology, 50(3), 336–357. https://doi.org/10.1177/0022022119827379.

31.

Groves

R. M.

(2011). Three eras of survey research. Public Opinion Quarterly, 75(5), 861–871. https://doi.org/10.1093/poq/nfr057.

32.

Herzog

A.R.

Bachman

J.G.

(1981). Effects of questionnaire length on response quality. Public Opinion Quarterly, 45(4), 549–559. https://doi.org/10.1086/268687.

33.

Huang

J. L.

Bowling

N. A.

Liu

(2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30(2), 299–311. https://doi.org/10.1007/s10869-014-9357-6.

34.

Huang

J. L.

Curran

G. C.

Keeney

Poposki

E. M.

DeShon

R. P.

(2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99–114. https://doi.org/10.1007/s10869-011-9231-8.

35.

Jia

Konold

T. R.

Cornell

Huang

(2018). The impact of validity screening on associations between self-reports of bullying victimization and student outcomes. Educational and Psychological Measurement, 78(1), 80–102. https://doi.org/10.1177/0013164416671767.

36.

Johnson

J. A.

(2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39(1), 103–129. https://doi.org/10.1016/j.jrp.2004.09.009.

37.

Junco

(2013). Comparing actual and self-reported measures of Facebook use. Computers in Human Behavior, 29(3), 626–631. https://doi.org/10.1016/j.chb.2012.11.007.

38.

Jung

Peck

Palmeira

Kim

(2022). An unintended consequence of product upgrades: How upgrades can make current consumers feel left behind. Journal of Marketing Research, 59(5), 1019–1039. https://doi.org/10.1177/00222437221078551.

39.

Kees

Berry

Burton

Sheehan

(2017). An analysis of data quality: Professional panels, student subject pools, and amazon’s mechanical turk. Journal of Advertising, 46(1), 141–155. https://doi.org/10.1080/00913367.2016.1269304.

40.

Kirk

B. A.

Sereda

(1969). Accuracy of self-reported college grade averages and characteristics of non and discrepant reporters. Educational and Psychological Measurement, 29(1), 147–155. https://doi.org/10.1177/001316446902900110.

41.

Klein

Völckner

Bruno

H. A.

Sattler

Bruno

(2019). Brand positioning based on brand image–country image fit. Marketing Science, 38(3), 516–538. https://doi.org/10.1287/mksc.2019.1151.

42.

Kormos

Gifford

(2014). The validity of self-report measures of proenvironmental behavior: A meta-analytic review. Journal of Environmental Psychology, 40, 359–371. https://doi.org/10.1016/j.jenvp.2014.09.003.

43.

Kosinski

Stillwell

Graepel

(2013). Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences, 110(15), 5802–5805. https://doi.org/10.1073/pnas.1218772110.

44.

Kreuter

Haas

G. C.

Keusch

Bähr

Trappmann

(2020). Collecting survey and smartphone sensor data with an app: Opportunities and challenges around privacy and informed consent. Social Science Computer Review, 38(5), 533–549. https://doi.org/10.1177/0894439318816389.

45.

Krosnick

J. A.

(1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. https://doi.org/10.1002/acp.2350050305.

46.

Kuncel

N. R.

Credé

Thomas

L. L.

(2005). The validity of self-reported grade point averages, class ranks, and test scores: A meta-analysis and review of the literature. Review of Educational Research, 75(1), 63–82. https://doi.org/10.3102/00346543075001063.

47.

Laran

Janiszewski

Salerno

(2019). Nonconscious nudges: Encouraging sustained goal pursuit. Journal of Consumer Research, 46(2), 307–329. https://doi.org/10.1093/jcr/ucy071.

48.

Lee

M. Y.

Toh

R. S.

(2000). Are consumer survey results distorted? Systematic impact of behavioral frequency and duration on survey response errors. Journal of Marketing Research, 37(1), 125–133. https://doi.org/10.1509/jmkr.37.1.125.18724.

49.

Lin

Y. H.

Chang

L. R.

Lee

Y. H.

Tseng

H. W.

Kuo

T. B.

Chen

S. H.

(2014). Development and validation of the smartphone addiction inventory (SPAI). PLoS ONE, 9(6), e98312. https://doi.org/10.1371/journal.pone.0098312.

50.

Lin

Y. H.

Lin

Y. C.

Lee

Y. H.

Lin

P. H.

Lin

S. H.

Chang

L. R.

Tseng

H. W.

Yen

L. Y.

Yang

C. C. H.

Kuo

T. B.

(2015). Time distortion associated with smartphone addiction: Identifying smartphone addiction via a mobile application (App). Journal of Psychiatric Research, 65, 139–145. https://doi.org/10.1016/j.jpsychires.2015.04.003.

51.

Liu

Wronski

(2018). Trap questions in online surveys: Results from three web survey experiments. International Journal of Market Research, 60(1), 32–49. https://doi.org/10.1177/1470785317744856.

52.

Wang

Lin

Wang

Zhou

(2019). Inequalities in the health survey using validation question to filter insufficient effort responding: Reducing overestimated effects or creating selection bias? International Journal for Equity in Health, 18, 131. https://doi.org/10.1186/s12939-019-1030-2.

53.

Malhotra

(2008). Completion time and response order effects in web surveys. Public Opinion Quarterly, 72(5), 914–934. https://doi.org/10.1093/poq/nfn050.

54.

Malhotra

N. K.

(2009). Marketing research: An applied orientation (6th ed.). Pearson Education.

55.

Maniaci

M. R.

Rogge

R. D.

(2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83. https://doi.org/10.1016/j.jrp.2013.09.008.

56.

Mayer

R. E.

Stull

A. T.

Campbell

Almeroth

Bimber

Chun

Knight

(2007). Overestimation bias in self-reported SAT scores. Educational Psychology Review, 19(4), 443–454. https://doi.org/10.1007/s10648-006-9034-z.

57.

McKay

A. S.

Garcia

D. M.

Clapper

J. P.

Shultz

K. S.

(2018). The attentive and the careless: Examining the relationship between benevolent and malevolent personality traits with careless responding in online surveys. Computers in Human Behavior, 84, 295–303. https://doi.org/10.1016/j.chb.2018.03.007.

58.

Meade

A. W.

Craig

S. B.

(2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://psycnet.apa.org/doi/10.1037/a0028085.

59.

Merckelbach

Boskovic

Pesy

Dalsklev

Lynn

S. J.

(2017). Symptom overreporting and dissociative experiences: A qualitative review. Consciousness and Cognition, 49, 132–144. https://doi.org/10.1016/j.concog.2017.01.007.

60.

Meyer

J. F.

Faust

K. A.

Faust

Baker

A. M.

Cook

N. E.

(2013). Careless and random responding on clinical and research measures in the addictions: A concerning problem and investigation of their detection. International Journal of Mental Health and Addiction, 11(3), 292–306. https://doi.org/10.1007/s11469-012-9410-5.

61.

Meyvis

van Osselaer

S. M. J.

(2018). Increasing the power of your study by increasing the effect size. Journal of Consumer Research, 44(5), 1157–1173. https://doi.org/10.1093/jcr/ucx110.

62.

Moore

H. E.

Rutherfurd

I. D.

(2020). Researching agricultural environmental behaviour: Improving the reliability of self-reporting. Journal of Rural Studies, 76, 296–304. https://doi.org/10.1016/j.jrurstud.2020.04.012.

63.

Nakano

Kondo

F. N.

(2018). Customer segmentation with purchase channels and media touchpoints using single source panel data. Journal of Retailing and Consumer Services, 41, 142–152. https://doi.org/10.1016/j.jretconser.2017.11.012.

64.

Nenycz-Thiel

Beal

Ludwichowska

Romaniuk

(2013). Investigating the accuracy of self-reports of brand usage behavior. Journal of Business Research, 66(2), 224–232. https://doi.org/10.1016/j.jbusres.2012.07.016.

65.

Oppenheimer

D. M.

Meyvis

Davidenko

(2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872. https://doi.org/10.1016/j.jesp.2009.03.009.

66.

Paas

L. J.

Dolnicar

Karlsson

(2018). Instructional manipulation checks: A longitudinal analysis with implications for mturk. International Journal of Research in Marketing, 35(2), 258–269. https://doi.org/10.1016/j.ijresmar.2018.01.003.

67.

Perry

J. D.

(1940). The reliability of high school averages computed from students’ estimates of their high school grades. School and Society, 52, 63–64.

68.

Prior

(2009). The immensely inflated news audience: Assessing bias in self-reported news exposure. Public Opinion Quarterly, 73(1), 130–143. https://doi.org/10.1093/poq/nfp002.

69.

Revilla

Ochoa

(2015). What are the links in a web survey among response time, quality, and auto-evaluation of the efforts done? Social Science Computer Review, 33(1), 97–114. https://doi.org/10.1177/0894439314531214.

70.

Scharkow

(2016). The accuracy of self-reported internet use: A validation study using client log data. Communication Methods and Measures, 10(1), 13–27. https://doi.org/10.1080/19312458.2015.1118446.

71.

Silber

Danner

Rammstedt

(2019). The impact of respondent attentiveness on reliability and validity. International Journal of Social Research Methodology, 22(2), 153–164. https://doi.org/10.1080/13645579.2018.1507378.

72.

Simon

H.A.

(1956). Rational choice and the structure of the environment. Psychological Review, 63(2), 129–138. https://doi.org/10.1037/h0042769.

73.

Steedle

J. T.

Hong

Cheng

(2019). The effects of inattentive responding on construct validity evidence when measuring social–emotional learning competencies. Educational Measurement: Issues and Practice, 38(2), 101–111. https://doi.org/10.1111/emip.12256.

74.

Stier

Breuer

Siegers

Thorson

(2020). Integrating survey data and digital trace data: Key issues in developing an emerging field. Social Science Computer Review, 38(5), 503–516. https://doi.org/10.1177/0894439319843669.

75.

Stopczynski

Sekara

Sapiezynski

Cuttone

Madsen

M. M.

Larsen

J. E.

Lehmann

(2014). Measuring large-scale social networks with high resolution. PLoS One, 9(4), e95978. https://doi.org/10.1371/journal.pone.0095978.

76.

Vanden Abeele

Beullens

Roe

(2013). Measuring mobile phone use: Gender, age and real usage level in relation to the accuracy and validity of self-reported mobile phone use. Mobile Media and Communication, 1(2), 213–236. https://doi.org/10.1177/2050157913477095.

77.

Ward

M.K.

Meade

A.W.

Allred

C. M.

Pappalardo

Stoughton

J.W.

(2017). Careless response and attrition as sources of bias in online survey assessments of personality traits and performance. Computers in Human Behavior, 76, 417–430. https://doi.org/10.1016/j.chb.2017.06.032.

78.

Ward

M.K.

Pond

S.B.

(2015). Using virtual presence and survey instructions to minimize careless responding on internet-based surveys. Computers in Human Behavior, 48, 554–568. https://doi.org/10.1016/j.chb.2015.01.070.

79.

Weijters

Baumgartner

(2012). Misresponse to reversed and negated items in surveys: A review. Journal of Marketing Research, 49(5), 737–747. https://doi.org/10.1509/jmr.11.0368.

80.

WHO. (2021). 6C51 gaming disorder. International Classification of Diseases, Eleventh Revision (ICD-11). World Health Organization (WHO) . https://icd.who.int/browse11/l-m/en#/http%3a%2f%2fid.who.int%2ficd%2fentity%2f1448597234 (Accessed 19 Apr 2023).

81.

Wise

S. L.

Kong

(2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. https://doi.org/10.1207/s15324818ame1802_2.

82.

Yang

L. C.

Toubia

de Jong

M. G.

(2018). Attention, information processing, and choice in incentive-aligned choice experiments. Journal of Marketing Research, 55(6), 783–800. https://doi.org/10.1177/0022243718817004.