Restricted accessResearch articleFirst published online 2022-3
The Effects of Sampling Frequency and Questionnaire Length on Perceived Burden,Compliance,and Careless Responding in Experience Sampling Data in a Student Population
Currently, little is known about the association between assessment intensity, burden, data quantity, and data quality in experience sampling method (ESM) studies. Researchers therefore have insufficient information to make informed decisions about the design of their ESM study. Our aim was to investigate the effects of different sampling frequencies and questionnaire lengths on burden, compliance, and careless responding. Students (n = 163) received either a 30- or 60-item questionnaire three, six, or nine times per day for 14 days. Preregistered multilevel regression analyses and analyses of variance were used to analyze the effect of design condition on momentary outcomes, changes in those outcomes over time, and retrospective outcomes. Our findings offer support for increased burden and compromised data quantity and quality with longer questionnaires, but not with increased sampling frequency. We therefore advise against the use of long ESM questionnaires, while high-sampling frequencies do not seem to be associated with negative consequences.
ArslanR. C.ReitzA. K.DriebeJ. C.GerlachT. M.PenkeL. (2019). Routinely randomize the display and order of items to estimate and adjust for biases in subjective reports. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000294
2.
BatesD.MaechlerM.BolkerB.WalkerS. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. https://doi.org/10.18637/jss.v067.i01
3.
CollinsL. M.GrahamJ. W. (2002). The effect of the timing and spacing of observations in longitudinal studies of tobacco and other drug use: Temporal design considerations. Drug and Alcohol Dependence, 68(Suppl. 1), 85-96. https://doi.org/10.1016/S0376-8716(02)00217-X
4.
ConnerT. S.ReidK. A. (2012). Effects of intensive mobile happiness reporting in daily life. Social Psychological and Personality Science, 3(3), 315-323. https://doi.org/10.1177/1948550611419677
5.
CourvoisierD. S.EidM.LischetzkeT. (2012). Compliance to a cell phone-based ecological momentary assessment study: The effect of time and personality characteristics. Psychological Assessment, 24(3), 713-720. https://doi.org/10.1037/a0026733
Ebner-PriemerU. W.SawitzkiG. (2007). Ambulatory assessment of affective instability in borderline personality disorder. European Journal of Psychological Assessment, 23(4), 238-247. https://doi.org/10.1027/1015-5759.23.4.238
8.
EdwardsC. J.CellaM.TarrierN.WykesT. (2016). The optimisation of experience sampling protocols in people with schizophrenia. Psychiatry Research, 244(October), 289-293. https://doi.org/10.1016/j.psychres.2016.07.048
9.
ForkmannT.SpangenbergL.HallenslebenN.HegerlU.KerstingA. (2018). Assessing suicidality in real time: A psychometric evaluation of self-report items for the assessment of suicidal ideation and its proximal risk factors using ecological momentary assessments. Journal of Abnormal Psychology, 127(8), 758-769. https://doi.org/10.1037/abn0000381
Fuller-TyszkiewiczM.SkouterisH.RichardsonB.BloreJ.HolmesM.MillsJ. (2013). Does the burden of the experience sampling method undermine data quality in state body image research?Body Image, 10(4), 607-613. https://doi.org/10.1016/j.bodyim.2013.06.003
12.
GalesicM.BosnjakM. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73(2), 349-360. https://doi.org/10.1093/poq/nfp031
13.
HimmelsteinP. H.WoodsW. C.WrightA. G. C. (2019). A comparison of signal- and event-contingent ambulatory assessment of interpersonal behavior and affect in social situations. Psychological Assessment, 31(7), 952-960. http://dx.doi.org/10.1037/pas0000718
14.
HuangJ. L.CurranP. G.KeeneyJ.PoposkiE. M.DeShonR. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99-114. https://doi.org/10.1007/s10869-011-9231-8
15.
IntilleS. S.HaynesC.ManiarD.PonnadaA.ManjouridesJ. (2016). µEMA: Microinteraction-based ecological momentary assessment (EMA) using a smartwatch. In UbiComp ’16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 1124-1135). ACM. https://doi.org/10.1145/2971648.2971717
16.
JanssensK. A. M.BosE. H.RosmalenJ. G. M.WichersM.RieseH. (2018). A qualitative approach to guide choices for diary design. BMC Medical Research Methodology, 18(1), Article 140. https://doi.org/10.1186/s12874-018-0579-6
17.
JonesA.RemmerswaalD.VerveerI.RobinsonE.FrankenI. H. A.Fred WenC. K.FieldM. (2018). Compliance with ecological momentary assessment protocols in substance users: A meta-analysis. Addiction, 114(4), 609-619. https://doi.org/10.1111/add.14503
18.
KimD. S.McCabeC. J.YamasakiB. L.LouieK. A.KingK. M. (2018). Detecting random responders with infrequency scales using an error-balancing threshold. Behavior Research Methods, 50(5), 1960-1970. https://doi.org/10.3758/s13428-017-0964-9
19.
LarsonR.CsikszentmihalyiM. (1983). The experience sampling method. In ReisH. T. (Ed.), New directions for methodology of social and behavioral science (pp. 41-56). Jossey-Bass.
20.
MayM.JunghaenelD. U.OnoM.StoneA. A. (2018). Ecological momentary assessment methodology in chronic pain research: A systematic review. Journal of Pain, 19(7), 699-716. https://doi.org/10.1016/j.jpain.2018.01.006
21.
McCabeK. O.MackL.FleesonW. (2012). A guide to data cleaning in experience-sampling studies. In MehlM. R.ConnerT. S. (Eds.), Handbook of Research methods for studying daily life (pp. 321-338). Guilford Press.
22.
McCarthyD. E.MinamiH.YehV. M.BoldK. (2015). An experimental investigation of reactivity to ecological momentary assessment frequency among adults trying to quit smoking. Addiction, 110(10), 1549-1560. https://doi.org/10.1111/add.12996
23.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437-455. https://doi.org/10.1037/a0028085
24.
MeersK.DejonckheereE.KalokerinosE. K.RummensK.KuppensP. (2020). mobileQ: A free user-friendly application for collecting experience sampling data. Behavior Research Methods, 52, 1510-1515. https://doi.org/10.3758/s13428-019-01330-1
25.
MorrenM.van DulmenS.OuwerkerkJ.BensingJ. (2009). Compliance with momentary pain measurement using electronic diaries: A systematic review. European Journal of Pain, 13(4), 354-365. https://doi.org/10.1016/j.ejpain.2008.05.010
26.
Myin-GermeysI.KasanovaZ.VaessenT.VachonH.KirtleyO.ViechtbauerW.ReininghausU. (2018). Experience sampling methodology in mental health research: New insights and technical developments. World Psychiatry, 17(2), 123-132. https://doi.org/10.1002/wps.20513
27.
OnoM.SchneiderS.JunghaenelD. U.StoneA. A. (2019). What affects the completion of ecological momentary assessments in chronic pain research? An individual patient data meta-analysis. Journal of Medical Internet Research, 21(2), Article e11398. https://doi.org/10.2196/11398
PiaseckiT. M.HuffordM. R.SolhanM.TrullT. J. (2007). Assessing clients in their natural environments with electronic diaries: Rationale, benefits, limitations, and barriers. Psychological Assessment, 19(1), 25-43. https://doi.org/10.1037/1040-3590.19.1.25
30.
R Core Team. (2019). R: A language and environment for statistical computing [Computer software]. https://www.r-project.org/
31.
RevelleW. (2018). Psych: Procedures for personality and psychological research [Computer software]. Northwestern University. https://CRAN.R-project.org/package=psych
32.
ReynoldsB. M.RoblesT. F.RepettiR. L. (2016). Measurement reactivity and fatigue effects in daily diary research with families. Developmental Psychology, 52(3), 442-456. https://doi.org/10.1037/dev0000081
33.
RintalaA.WampersM.LafitG.Myin-GermeysI.ViechtbauerW. (2020). Perceived disturbance of the diary signal and predictors thereof in studies using the experience sampling method [Manuscript in preparation].
34.
RintalaA.WampersM.Myin-GermeysI.ViechtbauerW. (2018). Response compliance and predictors thereof in studies using the experience sampling method. Psychological Assessment, 31(2), 226-235. https://doi.org/10.1037/pas0000662
35.
RolstadS.AdlerJ.RydénA. (2011). Response burden and questionnaire length: Is shorter better? A review and meta-analysis. Value in Health, 14(8), 1101-1108. https://doi.org/10.1016/j.jval.2011.06.003
36.
RosenkranzT.TakanoK.WatkinsE. R.EhringT. (2020). Assessing repetitive negative thinking in daily life: Development of an ecological momentary assessment paradigm. PLOS ONE, 15(4), Article e0231783. https://doi.org/10.1371/journal.pone.0231783
37.
SantangeloP. S.Ebner-PriemerU. W.TrullT. J. (2013). Experience sampling methods in clinical psychology. In ComerJ.KendallP. (Eds.), The Oxford handbook of research strategies for clinical psychology (pp. 188-210). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199793549.013.0011
ShroutP.LaneS. (2012). Psychometrics. In MehlM.ConnerT. (Ed.), Handbook of research methods for studying daily life (pp. 302-320). Guilford Press.
40.
SilviaP. J.KwapilT. R.EddingtonK. M.BrownL. H. (2013). Missed beeps and missing data: Dispositional and situational predictors of nonresponse in experience sampling research. Social Science Computer Review, 31(4), 471-481. https://doi.org/10.1177/0894439313479902
SoysterP. D.BosleyH. G.ReevesJ. W.AltmanA. D.FisherA. J. (2019). Evidence for the feasibility of person-specific ecological momentary assessment across diverse populations and study designs. Journal for Person-Oriented Research, 5(2), 53-64. https://doi.org/10.17505/jpor.2019.06
43.
SpookJ. E.PaulussenT.KokG.Van EmpelenP. (2013). Monitoring dietary intake and physical activity electronically: Feasibility, usability, and ecological validity of a mobile-based Ecological Momentary Assessment tool. Journal of Medical Internet Research, 15(9), e214. https://doi.org/10.2196/jmir.2617
44.
StoneA. A.BroderickJ. E.SchwartzJ. E.ShiffmanS.Litcher-KellyL.CalvaneseP. (2003). Intensive momentary reporting of pain with an electronic diary: Reactivity, compliance, and patient satisfaction. Pain, 104(1-2), 343-351. https://doi.org/10.1016/S0304-3959(03)00040-X
45.
TrullT.Ebner-PriemerU. (2020). Ambulatory assessment in psychopathology research: A review of recommended reporting guidelines and current practices. Journal of Abnormal Psychology, 129(1), 56-63. https://doi.org/10.31234/osf.io/eakyj
46.
VachonH.BourboussonM.DeschampsT.DoronJ.BulteauS.SauvagetA.Thomas-OllivierV. (2016). Repeated self-evaluations may involve familiarization: An exploratory study related to ecological momentary assessment designs in patients with major depressive disorder. Psychiatry Research, 245(November), 99-104. https://doi.org/10.1016/j.psychres.2016.08.034
47.
VachonH.ViechtbauerW.RintalaA.Myin-GermeysI. (2019). Compliance and retention with the Experience Sampling Method over the continuum of severe mental disorders: A systematic review and meta-analysis. Journal of Medical Internet Research, 21(12), Article e14475. https://doi.org/10.2196/14475
48.
van BerkelN.GoncalvesJ.KovalP.HosioS.DinglerT.FerreiraD.KostakosV. (2019). Context-informed scheduling and analysis: Improving accuracy of mobile self- reports. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-51). ACM.
49.
van BerkelN.GoncalvesJ.LovénL.FerreiraD.HosioS.KostakosV. (2018). Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports. International Journal of Human-Computer Studies, 125(May), 118-128. https://doi.org/10.1016/j.ijhcs.2018.12.002
50.
WalshE.BrinkerJ. K. (2016). Temporal considerations for self-report research using short message service. Journal of Media Psychology, 28(4), 200-206. https://doi.org/10.1027/1864-1105/a000161
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
