The Effects of Sampling Frequency and Questionnaire Length on Perceived Burden,Compliance,and Careless Responding in Experience Sampling Data in a Student Population

Abstract

Currently, little is known about the association between assessment intensity, burden, data quantity, and data quality in experience sampling method (ESM) studies. Researchers therefore have insufficient information to make informed decisions about the design of their ESM study. Our aim was to investigate the effects of different sampling frequencies and questionnaire lengths on burden, compliance, and careless responding. Students (n = 163) received either a 30- or 60-item questionnaire three, six, or nine times per day for 14 days. Preregistered multilevel regression analyses and analyses of variance were used to analyze the effect of design condition on momentary outcomes, changes in those outcomes over time, and retrospective outcomes. Our findings offer support for increased burden and compromised data quantity and quality with longer questionnaires, but not with increased sampling frequency. We therefore advise against the use of long ESM questionnaires, while high-sampling frequencies do not seem to be associated with negative consequences.

Keywords

experience sampling ecological momentary assessment sampling frequency questionnaire length data quality

Get full access to this article

View all access options for this article.

References

Arslan

R. C.

Reitz

A. K.

Driebe

J. C.

Gerlach

T. M.

Penke

(2019). Routinely randomize the display and order of items to estimate and adjust for biases in subjective reports. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000294

Bates

Maechler

Bolker

Walker

(2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. https://doi.org/10.18637/jss.v067.i01

Collins

L. M.

Graham

J. W.

(2002). The effect of the timing and spacing of observations in longitudinal studies of tobacco and other drug use: Temporal design considerations. Drug and Alcohol Dependence, 68(Suppl. 1), 85-96. https://doi.org/10.1016/S0376-8716(02)00217-X

Conner

T. S.

Reid

K. A.

(2012). Effects of intensive mobile happiness reporting in daily life. Social Psychological and Personality Science, 3(3), 315-323. https://doi.org/10.1177/1948550611419677

Courvoisier

D. S.

Eid

Lischetzke

(2012). Compliance to a cell phone-based ecological momentary assessment study: The effect of time and personality characteristics. Psychological Assessment, 24(3), 713-720. https://doi.org/10.1037/a0026733

Dowle

Srinivasan

(2019). data.table: Extension of ‘data.frame’ (1.12.2) [Computer software]. https://CRAN.R-project.org/package=data.table

Ebner-Priemer

U. W.

Sawitzki

(2007). Ambulatory assessment of affective instability in borderline personality disorder. European Journal of Psychological Assessment, 23(4), 238-247. https://doi.org/10.1027/1015-5759.23.4.238

Edwards

C. J.

Cella

Tarrier

Wykes

(2016). The optimisation of experience sampling protocols in people with schizophrenia. Psychiatry Research, 244(October), 289-293. https://doi.org/10.1016/j.psychres.2016.07.048

Forkmann

Spangenberg

Hallensleben

Hegerl

Kersting

(2018). Assessing suicidality in real time: A psychometric evaluation of self-report items for the assessment of suicidal ideation and its proximal risk factors using ecological momentary assessments. Journal of Abnormal Psychology, 127(8), 758-769. https://doi.org/10.1037/abn0000381

10.

Fox

Weisberg

(2019). An R companion to applied regression (3rd ed.). Sage. https://socialsciences.mcmaster.ca/jfox/Books/Companion/

11.

Fuller-Tyszkiewicz

Skouteris

Richardson

Blore

Holmes

Mills

(2013). Does the burden of the experience sampling method undermine data quality in state body image research? Body Image, 10(4), 607-613. https://doi.org/10.1016/j.bodyim.2013.06.003

12.

Galesic

Bosnjak

(2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73(2), 349-360. https://doi.org/10.1093/poq/nfp031

13.

Himmelstein

P. H.

Woods

W. C.

Wright

A. G. C.

(2019). A comparison of signal- and event-contingent ambulatory assessment of interpersonal behavior and affect in social situations. Psychological Assessment, 31(7), 952-960. http://dx.doi.org/10.1037/pas0000718

14.

Huang

J. L.

Curran

P. G.

Keeney

Poposki

E. M.

DeShon

R. P.

(2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99-114. https://doi.org/10.1007/s10869-011-9231-8

15.

Intille

S. S.

Haynes

Maniar

Ponnada

Manjourides

(2016). µEMA: Microinteraction-based ecological momentary assessment (EMA) using a smartwatch. In UbiComp ’16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 1124-1135). ACM. https://doi.org/10.1145/2971648.2971717

16.

Janssens

K. A. M.

Bos

E. H.

Rosmalen

J. G. M.

Wichers

Riese

(2018). A qualitative approach to guide choices for diary design. BMC Medical Research Methodology, 18(1), Article 140. https://doi.org/10.1186/s12874-018-0579-6

17.

Jones

Remmerswaal

Verveer

Robinson

Franken

I. H. A.

Fred Wen

C. K.

Field

(2018). Compliance with ecological momentary assessment protocols in substance users: A meta-analysis. Addiction, 114(4), 609-619. https://doi.org/10.1111/add.14503

18.

Kim

D. S.

McCabe

C. J.

Yamasaki

B. L.

Louie

K. A.

King

K. M.

(2018). Detecting random responders with infrequency scales using an error-balancing threshold. Behavior Research Methods, 50(5), 1960-1970. https://doi.org/10.3758/s13428-017-0964-9

19.

Larson

Csikszentmihalyi

(1983). The experience sampling method. In Reis

H. T.

(Ed.), New directions for methodology of social and behavioral science (pp. 41-56). Jossey-Bass.

20.

May

Junghaenel

D. U.

Ono

Stone

A. A.

(2018). Ecological momentary assessment methodology in chronic pain research: A systematic review. Journal of Pain, 19(7), 699-716. https://doi.org/10.1016/j.jpain.2018.01.006

21.

McCabe

K. O.

Mack

Fleeson

(2012). A guide to data cleaning in experience-sampling studies. In Mehl

M. R.

Conner

T. S.

(Eds.), Handbook of Research methods for studying daily life (pp. 321-338). Guilford Press.

22.

McCarthy

D. E.

Minami

Yeh

V. M.

Bold

(2015). An experimental investigation of reactivity to ecological momentary assessment frequency among adults trying to quit smoking. Addiction, 110(10), 1549-1560. https://doi.org/10.1111/add.12996

23.

Meade

A. W.

Craig

S. B.

(2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437-455. https://doi.org/10.1037/a0028085

24.

Meers

Dejonckheere

Kalokerinos

E. K.

Rummens

Kuppens

(2020). mobileQ: A free user-friendly application for collecting experience sampling data. Behavior Research Methods, 52, 1510-1515. https://doi.org/10.3758/s13428-019-01330-1

25.

Morren

van Dulmen

Ouwerkerk

Bensing

(2009). Compliance with momentary pain measurement using electronic diaries: A systematic review. European Journal of Pain, 13(4), 354-365. https://doi.org/10.1016/j.ejpain.2008.05.010

26.

Myin-Germeys

Kasanova

Vaessen

Vachon

Kirtley

Viechtbauer

Reininghaus

(2018). Experience sampling methodology in mental health research: New insights and technical developments. World Psychiatry, 17(2), 123-132. https://doi.org/10.1002/wps.20513

27.

Ono

Schneider

Junghaenel

D. U.

Stone

A. A.

(2019). What affects the completion of ecological momentary assessments in chronic pain research? An individual patient data meta-analysis. Journal of Medical Internet Research, 21(2), Article e11398. https://doi.org/10.2196/11398

28.

Perrin

Anderson

. (2019, April 10). Share of U.S. adults using social media, including Facebook, is mostly unchanged since 2018. https://www.pewresearch.org/fact-tank/2019/04/10/share-of-u-s-adults-using-social-media-including-facebook-is-mostly-unchanged-since-2018/

29.

Piasecki

T. M.

Hufford

M. R.

Solhan

Trull

T. J.

(2007). Assessing clients in their natural environments with electronic diaries: Rationale, benefits, limitations, and barriers. Psychological Assessment, 19(1), 25-43. https://doi.org/10.1037/1040-3590.19.1.25

30.

R Core Team. (2019). R: A language and environment for statistical computing [Computer software]. https://www.r-project.org/

31.

Revelle

(2018). Psych: Procedures for personality and psychological research [Computer software]. Northwestern University. https://CRAN.R-project.org/package=psych

32.

Reynolds

B. M.

Robles

T. F.

Repetti

R. L.

(2016). Measurement reactivity and fatigue effects in daily diary research with families. Developmental Psychology, 52(3), 442-456. https://doi.org/10.1037/dev0000081

33.

Rintala

Wampers

Lafit

Myin-Germeys

Viechtbauer

(2020). Perceived disturbance of the diary signal and predictors thereof in studies using the experience sampling method [Manuscript in preparation].

34.

Rintala

Wampers

Myin-Germeys

Viechtbauer

(2018). Response compliance and predictors thereof in studies using the experience sampling method. Psychological Assessment, 31(2), 226-235. https://doi.org/10.1037/pas0000662

35.

Rolstad

Adler

Rydén

(2011). Response burden and questionnaire length: Is shorter better? A review and meta-analysis. Value in Health, 14(8), 1101-1108. https://doi.org/10.1016/j.jval.2011.06.003

36.

Rosenkranz

Takano

Watkins

E. R.

Ehring

(2020). Assessing repetitive negative thinking in daily life: Development of an ecological momentary assessment paradigm. PLOS ONE, 15(4), Article e0231783. https://doi.org/10.1371/journal.pone.0231783

37.

Santangelo

P. S.

Ebner-Priemer

U. W.

Trull

T. J.

(2013). Experience sampling methods in clinical psychology. In Comer

Kendall

(Eds.), The Oxford handbook of research strategies for clinical psychology (pp. 188-210). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199793549.013.0011

38.

Shiffman

Stone

A. A.

Hufford

M. R.

(2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1-32. https://doi.org/10.1146/annurev.clinpsy.3.022806.091415

39.

Shrout

Lane

(2012). Psychometrics. In Mehl

Conner

(Ed.), Handbook of research methods for studying daily life (pp. 302-320). Guilford Press.

40.

Silvia

P. J.

Kwapil

T. R.

Eddington

K. M.

Brown

L. H.

(2013). Missed beeps and missing data: Dispositional and situational predictors of nonresponse in experience sampling research. Social Science Computer Review, 31(4), 471-481. https://doi.org/10.1177/0894439313479902

41.

Sokolovsky

A. W.

Mermelstein

R. J.

Hedeker

(2014). Factors predicting compliance to ecological momentary assessment among adolescent smokers. Nicotine & Tobacco Research, 16(3), 351-358. https://doi.org/10.1093/ntr/ntt154

42.

Soyster

P. D.

Bosley

H. G.

Reeves

J. W.

Altman

A. D.

Fisher

A. J.

(2019). Evidence for the feasibility of person-specific ecological momentary assessment across diverse populations and study designs. Journal for Person-Oriented Research, 5(2), 53-64. https://doi.org/10.17505/jpor.2019.06

43.

Spook

J. E.

Paulussen

Kok

Van Empelen

(2013). Monitoring dietary intake and physical activity electronically: Feasibility, usability, and ecological validity of a mobile-based Ecological Momentary Assessment tool. Journal of Medical Internet Research, 15(9), e214. https://doi.org/10.2196/jmir.2617

44.

Stone

A. A.

Broderick

J. E.

Schwartz

J. E.

Shiffman

Litcher-Kelly

Calvanese

(2003). Intensive momentary reporting of pain with an electronic diary: Reactivity, compliance, and patient satisfaction. Pain, 104(1-2), 343-351. https://doi.org/10.1016/S0304-3959(03)00040-X

45.

Trull

Ebner-Priemer

(2020). Ambulatory assessment in psychopathology research: A review of recommended reporting guidelines and current practices. Journal of Abnormal Psychology, 129(1), 56-63. https://doi.org/10.31234/osf.io/eakyj

46.

Vachon

Bourbousson

Deschamps

Doron

Bulteau

Sauvaget

Thomas-Ollivier

(2016). Repeated self-evaluations may involve familiarization: An exploratory study related to ecological momentary assessment designs in patients with major depressive disorder. Psychiatry Research, 245(November), 99-104. https://doi.org/10.1016/j.psychres.2016.08.034

47.

Vachon

Viechtbauer

Rintala

Myin-Germeys

(2019). Compliance and retention with the Experience Sampling Method over the continuum of severe mental disorders: A systematic review and meta-analysis. Journal of Medical Internet Research, 21(12), Article e14475. https://doi.org/10.2196/14475

48.

van Berkel

Goncalves

Koval

Hosio

Dingler

Ferreira

Kostakos

. (2019). Context-informed scheduling and analysis: Improving accuracy of mobile self- reports. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-51). ACM.

49.

van Berkel

Goncalves

Lovén

Ferreira

Hosio

Kostakos

. (2018). Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports. International Journal of Human-Computer Studies, 125(May), 118-128. https://doi.org/10.1016/j.ijhcs.2018.12.002

50.

Walsh

Brinker

J. K.

(2016). Temporal considerations for self-report research using short message service. Journal of Media Psychology, 28(4), 200-206. https://doi.org/10.1027/1864-1105/a000161

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.33 MB