The experience sampling method (ESM) has become a popular tool in psychology. However, the intensive nature of ESM raises concerns about careless responding, where participants respond without paying sufficient attention. This study investigated the temporal dynamics of carelessness across three common sample types (community, student, clinical). We leveraged four careless responding indicators—response time, within-beep standard deviation, an inconsistency index, and occasion-person correlation—and used univariate and multivariate multilevel models to examine their temporal trajectories. Our results demonstrate that careless responding is not a stable phenomenon but changes over time, with evidence for increases across days and non-stationarity within days across the different samples. The presence of few and small associations among the indicators implies either that they flag distinct kinds of carelessness or that some of them do not capture carelessness at all. Overall, our findings underscore the importance of considering the temporal dynamics of carelessness in ESM studies.
AriasV. B.GarridoL. E.JenaroC.Martínez-MolinaA.AriasB. (2020). A little garbage in, lots of garbage out: Assessing the impact of careless responding in personality survey data. Behavior Research Methods, 52(6), 2489–2505. https://doi.org/10.3758/s13428-020-01401-8
2.
ArslanR. C.ReitzA. K.DriebeJ. C.GerlachT. M.PenkeL. (2021). Routinely randomize potential sources of measurement reactivity to estimate and adjust for biases in subjective reports. Psychological Methods, 26(2), 175–185. https://doi.org/10.31234/osf.io/va8bx
3.
BassettJ.ClevelandA.AcornD.NixM.SnyderT. (2015). Are they paying attention? Students’ lack of motivation and attention potentially threaten the utility of course evaluations. Assessment & Evaluation in Higher Education, 42, 1–12. https://doi.org/10.1080/02602938.2015.1119801
4.
BatesD.MächlerM.BolkerB.WalkerS. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48. https://doi.org/10.18637/jss.v067.i01
5.
BenningS. D.BachrachR. L.SmithE. A.FreemanA. J.WrightA. G. C. (2019). The registration continuum in clinical science: A guide toward transparent practices. Journal of Abnormal Psychology, 128(6), 528–540. https://doi.org/10.1037/abn0000451
6.
BrühlmannF.PetralitoS.AeschbachL. F.OpwisK. (2020). The quality of data collected online: An investigation of careless responding in a crowdsourced sample. Methods in Psychology, 2, 100022. https://doi.org/10.1016/j.metip.2020.100022
7.
CsikszentmihalyiM.LarsonR. (1987). Validity and reliability of the experience-sampling method. The Journal of Nervous and Mental Disease, 175(9), 526–536. https://doi.org/10.1097/00005053-198709000-00004
8.
CsikszentmihalyiM.LarsonR. (2014). Validity and reliability of the experience-sampling method. In CsikszentmihalyiM. (Ed.), Flow and the foundations of positive psychology: The collected works of Mihaly Csikszentmihalyi (pp. 35–54). Springer Netherlands. https://doi.org/10.1007/978-94-017-9088-8_3
9.
CurranP.HauserK. (2019). I’m paid biweekly, just not by leprechauns: Evaluating valid-but-incorrect response rates to attention check items. Journal of Research in Personality, 82, 103849. https://doi.org/10.1016/j.jrp.2019.103849
10.
DenisonA. (2022). Prevalence and predictors of careless responding in experience sampling research [Master’s thesis, USF Tampa Graduate Theses and Dissertations]. https://digitalcommons.usf.edu/etd/9341
11.
DeSimoneJ. A.DeSimoneA. J.HarmsP. D.WoodD. (2018). The differential impacts of two forms of insufficient effort responding. Applied Psychology, 67(2), 309–338. https://doi.org/10.1111/apps.12117
12.
EiseleG.VachonH.LafitG.KuppensP.HoubenM.Myin-GermeysI.ViechtbauerW. (2022). The effects of sampling frequency and questionnaire length on perceived burden, compliance, and careless responding in experience sampling data in a student population. Assessment, 29(2), 136–151. https://doi.org/10.1177/1073191120957102
13.
EiseleG.VachonH.LafitG.TuyaertsD.HoubenM.KuppensP.Myin-GermeysI.ViechtbauerW. (2023). A mixed-method investigation into measurement reactivity to the experience sampling method: The role of sampling protocol and individual characteristics. Psychological Assessment, 35(1), 68–81. https://doi.org/10.1037/pas0001177
14.
FritzJ.StochlJ.FriedE. I.GoodyerI. M.BorkuloC. D.van WilkinsonP. O.HarmelenA.-L. V. (2019). Unravelling the complex nature of resilience factors and their changes between early and later adolescence. OSF Preprints. https://doi.org/10.31219/osf.io/8xgkb
15.
GoldammerP.AnnenH.StöckliP. L.JonasK. (2020). Careless responding in questionnaire measures: Detection, impact, and remedies. The Leadership Quarterly, 31(4), 101384. https://doi.org/10.1016/j.leaqua.2020.101384
16.
GrommischG.KovalP.HintonJ. D. X.GleesonJ.HollensteinT.KuppensP.LischetzkeT. (2020). Modeling individual differences in emotion regulation repertoire in daily life with multilevel latent profile analysis. Emotion, 20(8), 1462–1474. https://doi.org/10.1037/emo0000669
17.
HarrellF. E. (2001). Regression modeling strategies: With applications to linear models, logistic regression, and survival analysis. Springer. https://doi.org/10.1007/978-1-4757-3462-1
HaslbeckJ.MartínezA. J.RoefsA.FriedE. I.LemmensL. H. J. M.GrootE.EdelsbrunnerP. (2024). Comparing Likert and visual analogue scales in ecological momentary assessment. OSF. https://doi.org/10.31234/osf.io/yt8xw
21.
HasselhornK.OttensteinC.LischetzkeT. (2023). Modeling careless responding in ambulatory assessment studies using multilevel latent class analysis: Factors influencing careless responding. Psychological Methods, 30(2), 374–392. https://doi.org/10.1037/met0000580
22.
HernandezR.SchneiderS.PinkhamA. E.DeppC. A.AckermanR.PyatakE. A.BadalV. D.MooreR. C.HarveyP. D.FunschK.StoneA. A. (2024). Comparisons of self-report with objective measurements suggest faster responding but little change in response quality over time in ecological momentary assessment studies. Assessment, 32(3), 335–355. https://doi.org/10.1177/10731911241245793
23.
HoubenM.VansteelandtK.ClaesL.SienaertP.BerensA.SleuwaegenE.KuppensP. (2016). Emotional switching in borderline personality disorder: A daily life study. Personality Disorders: Theory, Research, and Treatment, 7(1), 50–60. https://doi.org/10.1037/per0000126
24.
HuangJ. L.BowlingN. A.LiuM.LiY. (2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30(2), 299–311.
25.
HuangJ. L.CurranP. G.KeeneyJ.PoposkiE. M.DeShonR. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99–114. https://doi.org/10.1007/s10869-011-9231-8
26.
HuangJ. L.LiuM.BowlingN. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100(3), 828–845. https://doi.org/10.1037/a0038510
27.
HussongA. M.HicksR. E.LevyS. A.CurranP. J. (2001). Specifying the relations between affect and heavy alcohol use among young adults. Journal of Abnormal Psychology, 110(3), 449–461. https://doi.org/10.1037/0021-843X.110.3.449
28.
JasoB. A.KrausN. I.HellerA. S. (2022). Identification of careless responding in ecological momentary assessment research: From posthoc analyses to real-time data monitoring. Psychological Methods, 27(6), 958–981. https://doi.org/10.1037/met0000312
29.
JonesA.EarnestJ.AdamM.ClarkeR.YatesJ.PenningtonC. R. (2022). Careless responding in crowdsourced alcohol research: A systematic review and meta-analysis of practices and prevalence. Experimental and Clinical Psychopharmacology, 30(4), 381–399. https://doi.org/10.1037/pha0000546
30.
KalokerinosE. K.Russo-BatterhamD.GrewalK. K.MoeckE. K.WilsonE.SmithI.GreenwayK. H.GarrettP.ShresthaK. M.KuppensP.KovalP. (in preparation). The EMOTE Database: An open searchable database of experience sampling data mapping everyday life.
KirtleyO. J.LafitG.AchterhofR.HiekkarantaA. P.Myin-GermeysI. (2021). Making the black box transparent: A template and tutorial for registration of studies using experience-sampling methods. Advances in Methods and Practices in Psychological Science, 4(1), 1–16. https://doi.org/10.1177/2515245920924686
33.
KovalP.PeM. L.MeersK.KuppensP. (2013). Affect dynamics in relation to depressive symptoms: Variable, unstable or inert?Emotion, 13(6), 1132–1141. https://doi.org/10.1037/a0033579
34.
KuppensP.OraveczZ.TuerlinckxF. (2010). Feelings change: Accounting for individual differences in the temporal dynamics of affect. Journal of Personality and Social Psychology, 99(6), 1042–1060. https://doi.org/10.1037/a0020962
35.
LafitG.RevolJ.CloosL.KuppensP.CeulemansE. (2023). The effect of different construct operationalizations, study duration, and preprocessing choices on power-based sample size recommendations in intensive longitudinal research. OSF. https://doi.org/10.31234/osf.io/7msh6
36.
MarjanovicZ.HoldenR.StruthersW.CribbieR.GreenglassE. (2015). The inter-item standard deviation (ISD): An index that discriminates between conscientious and random responders. Personality and Individual Differences, 84, 79–83. https://doi.org/10.1016/j.paid.2014.08.021
37.
MayM.JunghaenelD. U.OnoM.StoneA. A.SchneiderS. (2018). Ecological momentary assessment methodology in chronic pain research: A systematic review. The Journal of Pain, 19(7), 699–716. https://doi.org/10.1016/j.jpain.2018.01.006
38.
McKayA. S.GarciaD. M.ClapperJ. P.ShultzK. S. (2018). The attentive and the careless: Examining the relationship between benevolent and malevolent personality traits with careless responding in online surveys. Computers in Human Behavior, 84, 295–303. https://doi.org/10.1016/j.chb.2018.03.007
39.
McKibbenW. B.SilviaP. J. (2017). Evaluating the distorting effects of inattentive responding and social desirability on self-report scales in creativity and the arts. The Journal of Creative Behavior, 51(1), 57–69. https://doi.org/10.1002/jocb.86
40.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085
41.
Myin-GermeysI.KasanovaZ.VaessenT.VachonH.KirtleyO.ViechtbauerW.ReininghausU. (2018). Experience sampling methodology in mental health research: New insights and technical developments. World Psychiatry: Official Journal of the World Psychiatric Association (WPA), 17(2), 123–132. https://doi.org/10.1002/wps.20513
42.
Myin-GermeysI.KuppensP. (2021). The open handbook of experience sampling methodology: A step-by-step guide to designing, conducting, and analyzing ESM studies. Center for Research on Experience Sampling and Ambulatory Methods Leuven (REAL).
43.
PinheiroJ.BatesD.DebRoyS.SarkarD., EISPACK authors, HeisterkampS.Van WilligenB.RankeJ., & R Core Team. (2024). nlme: Linear and Nonlinear Mixed Effects Models (Version 3.1-166) [Computer software]. https://cran.r-project.org/web/packages/nlme/index.html
44.
R Core Team. (2021). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/
45.
RightsJ. D.SterbaS. K. (2019). Quantifying explained variance in multilevel models: An integrative framework for defining R-squared measures. Psychological Methods, 24(3), 309–338. https://doi.org/10.1037/met0000184
46.
SchneiderS.MayM.StoneA. A. (2018). Careless responding in internet-based quality of life assessments. Quality of Life Research, 27(4), 1077–1088. https://doi.org/10.1007/s11136-017-1767-2
47.
SerreF.FatseasM.DebrabantR.AlexandreJ.-M.AuriacombeM.SwendsenJ. (2012). Ecological momentary assessment in alcohol, tobacco, cannabis and opiate dependence: A comparison of feasibility and validity. Drug and Alcohol Dependence, 126(1–2), 118–123. https://doi.org/10.1016/j.drugalcdep.2012.04.025
48.
SilviaP. J.KwapilT. R.EddingtonK. M.BrownL. H. (2013). Missed beeps and missing data: Dispositional and situational predictors of nonresponse in experience sampling research. Social Science Computer Review, 31(4), 471–481. https://doi.org/10.1177/0894439313479902
49.
TateA. D.FertigA. R.de BritoJ. N.EllisÉ. M.CarrC. P.TrofholzA.BergeJ. M. (2024). Momentary factors and study characteristics associated with participant burden and protocol adherence: Ecological momentary assessment. JMIR Formative Research, 8, e49512. https://doi.org/10.2196/49512
50.
TelfordC.McCarthy-JonesS.CorcoranR.RowseG. (2012). Experience sampling methodology studies of depression: The state of the art. Psychological Medicine, 42(6), 1119–1129. https://doi.org/10.1017/S0033291711002200
51.
TwiskJ. W. R. (2006). Applied multilevel analysis: A practical guide for medical researchers. Cambridge University Press. https://doi.org/10.1017/CBO9780511610806
52.
UlitzschE.DomingueB. W.KapoorR.KanopkaK.RiosJ. A. (2023). A probabilistic filtering approach to non-effortful responding. Educational Measurement: Issues and Practice, 42(3), 50–64. https://doi.org/10.1111/emip.12567
53.
UlitzschE.NestlerS.LüdtkeO.NagyG. (2024). A screen-time-based mixture model for identifying and monitoring careless and insufficient effort responding in ecological momentary assessment data. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000636
54.
UlitzschE.ViechtbauerW.LüdtkeO.Myin-GermeysI.NagyG.NestlerS.EiseleG. (2024). Investigating the effect of experience sampling study design on careless and insufficient effort responding identified with a screen-time-based mixture model. OSF. https://doi.org/10.31234/osf.io/2s38a
55.
VachonH.ViechtbauerW.RintalaA.Myin-GermeysI. (2019). Compliance and retention with the experience sampling method over the continuum of severe mental disorders: Meta-analysis and recommendations. Journal of Medical Internet Research, 21(12), e14475. https://doi.org/10.2196/14475
56.
VogelsmeierL. V. D. E.CloosL.KuppensP.CeulemansE. (2023). Evaluating dynamics in affect structure with latent Markov factor analysis. Emotion, 24(3), 782–794. https://doi.org/10.1037/emo0001307
57.
VogelsmeierL. V. D. E.JongerlingJ.UlitzschE. (2025). Accounting for measurement invariance violations in careless responding detection in intensive longitudinal data: Exploratory vs. partially constrained latent Markov factor analysis. OSF. https://doi.org/10.31234/osf.io/rab7u_v2
58.
VogelsmeierL. V. D. E.UglanovaI.ReinM. T.UlitzschE. (2024). Investigating contextual correlates of inattentive responding in ecological momentary assessment data with a confirmatory mixture IRT model. https://osf.io/p9cfm
59.
WangS. D.HatzingerL.MoralesJ.HewusM.IntilleS.DuntonG. F. (2024). Burden and inattentive responding in a 12-month intensive longitudinal study: Interview study among young adults. JMIR Formative Research, 8, e52165. https://doi.org/10.2196/52165
60.
WardM. K.MeadeA. W. (2018). Applying social psychology to prevent careless responding during online surveys. Applied Psychology, 67(2), 231–263. https://doi.org/10.1111/apps.12118
61.
WardM. K.MeadeA. W. (2023). Dealing with careless responding in survey data: Prevention, identification, and recommended best practices. Annual Review of Psychology, 74(1), 577–596. https://doi.org/10.1146/annurev-psych-040422-045007
62.
WillrothE. C.AthertonO. E. (2024). Best laid plans: A guide to reporting preregistration deviations. Advances in Methods and Practices in Psychological Science, 7(1), 1–14. https://doi.org/10.1177/25152459231213802
63.
WoodsC. (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28, 186–191. https://doi.org/10.1007/s10862-005-9004-7
64.
WrzusC.NeubauerA. B. (2023). Ecological momentary assessment: A meta-analysis on designs, samples, and compliance across research fields. Assessment, 30(3), 825–846. https://doi.org/10.1177/10731911211067538
